Nov 03 17:16:24.196650 ip-10-0-143-117 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 17:16:24.555404 ip-10-0-143-117 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:16:24.555404 ip-10-0-143-117 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 17:16:24.555404 ip-10-0-143-117 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:16:24.555404 ip-10-0-143-117 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:16:24.558738 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558661    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:16:24.558738 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558736    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:16:24.558738 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558740    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:16:24.558822 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558744    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:16:24.558822 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558748    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:16:24.558822 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558750    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:16:24.558822 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558753    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:16:24.558822 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558756    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:16:24.558822 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558758    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:16:24.558822 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558761    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:16:24.558822 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558764    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:16:24.558822 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558766    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:16:24.558822 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558769    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:16:24.558822 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558771    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:16:24.558822 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558775    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:16:24.558822 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558777    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:16:24.558822 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558779    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:16:24.558822 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558782    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:16:24.558822 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558785    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:16:24.558822 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558787    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:16:24.558822 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558793    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:16:24.558822 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558796    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:16:24.558822 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558799    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:16:24.559291 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558801    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:16:24.559291 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558804    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:16:24.559291 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558806    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:16:24.559291 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558809    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:16:24.559291 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558812    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:16:24.559291 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558816    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:16:24.559291 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558819    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:16:24.559291 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558821    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:16:24.559291 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558824    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:16:24.559291 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558827    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:16:24.559291 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558829    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:16:24.559291 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558832    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:16:24.559291 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558835    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:16:24.559291 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558838    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:16:24.559291 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558841    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:16:24.559291 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558843    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:16:24.559291 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558846    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:16:24.559291 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558849    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:16:24.559291 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558851    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:16:24.559738 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558853    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:16:24.559738 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558856    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:16:24.559738 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558859    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:16:24.559738 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558861    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:16:24.559738 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558864    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:16:24.559738 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558866    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:16:24.559738 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558869    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:16:24.559738 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558871    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:16:24.559738 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558875    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:16:24.559738 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558879    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:16:24.559738 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558883    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:16:24.559738 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558886    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:16:24.559738 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558888    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:16:24.559738 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558891    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:16:24.559738 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558894    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:16:24.559738 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558897    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:16:24.559738 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558899    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:16:24.559738 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558902    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:16:24.559738 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558905    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:16:24.560178 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558908    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:16:24.560178 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558910    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:16:24.560178 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558913    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:16:24.560178 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558915    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:16:24.560178 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558918    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:16:24.560178 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558920    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:16:24.560178 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558923    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:16:24.560178 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558925    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:16:24.560178 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558928    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:16:24.560178 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558930    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:16:24.560178 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558933    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:16:24.560178 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558941    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:16:24.560178 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558944    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:16:24.560178 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558947    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:16:24.560178 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558949    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:16:24.560178 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558952    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:16:24.560178 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558954    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:16:24.560178 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558957    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:16:24.560178 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558960    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:16:24.560637 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558962    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:16:24.560637 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558964    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:16:24.560637 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558967    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:16:24.560637 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558969    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:16:24.560637 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558972    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:16:24.560637 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558974    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:16:24.560637 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558977    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:16:24.560637 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558980    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:16:24.560637 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558982    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:16:24.560637 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558985    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:16:24.560637 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558988    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:16:24.560637 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558991    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:16:24.560637 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558994    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:16:24.560637 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558996    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:16:24.560637 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.558999    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:16:24.560637 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.559001    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:16:24.560637 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.559004    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:16:24.560637 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.559006    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:16:24.560637 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.559009    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:16:24.561079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.559013    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:16:24.561079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.559017    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:16:24.561079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.559019    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:16:24.561079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.559022    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:16:24.561079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.559025    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:16:24.561079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.559027    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:16:24.561079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.559032    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:16:24.561321 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561308    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:16:24.561321 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561320    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:16:24.561391 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561323    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:16:24.561391 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561327    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:16:24.561391 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561330    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:16:24.561391 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561334    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:16:24.561391 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561338    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:16:24.561391 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561340    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:16:24.561391 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561343    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:16:24.561391 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561346    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:16:24.561391 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561349    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:16:24.561391 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561351    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:16:24.561391 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561368    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:16:24.561391 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561371    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:16:24.561391 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561374    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:16:24.561391 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561376    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:16:24.561391 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561379    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:16:24.561391 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561382    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:16:24.561391 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561384    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:16:24.561391 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561387    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:16:24.561391 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561389    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:16:24.561391 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561392    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:16:24.561971 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561395    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:16:24.561971 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561398    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:16:24.561971 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561400    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:16:24.561971 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561403    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:16:24.561971 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561406    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:16:24.561971 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561409    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:16:24.561971 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561411    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:16:24.561971 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561414    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:16:24.561971 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561416    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:16:24.561971 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561419    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:16:24.561971 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561422    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:16:24.561971 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561425    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:16:24.561971 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561427    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:16:24.561971 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561430    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:16:24.561971 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561434    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:16:24.561971 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561437    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:16:24.561971 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561440    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:16:24.561971 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561443    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:16:24.561971 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561445    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:16:24.561971 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561448    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:16:24.562460 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561450    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:16:24.562460 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561453    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:16:24.562460 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561455    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:16:24.562460 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561458    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:16:24.562460 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561460    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:16:24.562460 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561463    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:16:24.562460 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561465    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:16:24.562460 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561468    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:16:24.562460 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561470    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:16:24.562460 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561473    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:16:24.562460 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561475    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:16:24.562460 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561478    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:16:24.562460 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561480    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:16:24.562460 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561484    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:16:24.562460 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561486    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:16:24.562460 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561489    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:16:24.562460 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561492    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:16:24.562460 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561495    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:16:24.562460 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561497    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:16:24.562460 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561500    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:16:24.562935 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561502    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:16:24.562935 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561504    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:16:24.562935 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561507    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:16:24.562935 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561510    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:16:24.562935 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561512    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:16:24.562935 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561516    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:16:24.562935 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561519    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:16:24.562935 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561523    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:16:24.562935 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561526    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:16:24.562935 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561530    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:16:24.562935 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561533    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:16:24.562935 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561536    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:16:24.562935 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561538    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:16:24.562935 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561540    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:16:24.562935 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561543    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:16:24.562935 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561545    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:16:24.562935 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561548    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:16:24.562935 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561551    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:16:24.562935 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561554    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:16:24.562935 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561556    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:16:24.563425 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561559    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:16:24.563425 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561562    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:16:24.563425 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561564    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:16:24.563425 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561567    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:16:24.563425 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561569    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:16:24.563425 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561573    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:16:24.563425 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561575    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:16:24.563425 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561578    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:16:24.563425 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561580    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:16:24.563425 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561583    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:16:24.563425 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561586    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:16:24.563425 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561589    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:16:24.563425 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561592    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:16:24.563425 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561595    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:16:24.563425 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561597    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:16:24.563425 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561600    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:16:24.563425 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561603    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:16:24.563425 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561605    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:16:24.563425 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561608    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:16:24.563878 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561610    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:16:24.563878 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561613    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:16:24.563878 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561615    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:16:24.563878 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561618    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:16:24.563878 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.561621    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:16:24.563878 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562102    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:16:24.563878 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562107    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:16:24.563878 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562109    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:16:24.563878 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562113    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:16:24.563878 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562116    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:16:24.563878 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562119    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:16:24.563878 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562121    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:16:24.563878 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562124    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:16:24.563878 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562127    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:16:24.563878 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562130    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:16:24.563878 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562132    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:16:24.563878 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562135    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:16:24.563878 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562138    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:16:24.563878 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562140    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:16:24.564326 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562143    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:16:24.564326 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562146    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:16:24.564326 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562148    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:16:24.564326 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562151    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:16:24.564326 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562153    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:16:24.564326 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562156    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:16:24.564326 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562159    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:16:24.564326 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562161    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:16:24.564326 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562164    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:16:24.564326 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562166    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:16:24.564326 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562174    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:16:24.564326 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562177    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:16:24.564326 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562180    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:16:24.564326 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562182    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:16:24.564326 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562185    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:16:24.564326 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562187    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:16:24.564326 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562190    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:16:24.564326 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562192    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:16:24.564326 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562195    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:16:24.564326 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562197    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:16:24.564805 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562200    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:16:24.564805 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562202    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:16:24.564805 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562205    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:16:24.564805 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562207    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:16:24.564805 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562210    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:16:24.564805 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562212    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:16:24.564805 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562215    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:16:24.564805 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562218    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:16:24.564805 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562221    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:16:24.564805 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562223    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:16:24.564805 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562226    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:16:24.564805 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562229    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:16:24.564805 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562231    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:16:24.564805 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562234    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:16:24.564805 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562236    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:16:24.564805 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562238    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:16:24.564805 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562241    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:16:24.564805 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562243    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:16:24.564805 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562246    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:16:24.564805 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562248    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:16:24.565294 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562251    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:16:24.565294 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562253    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:16:24.565294 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562256    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:16:24.565294 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562261    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:16:24.565294 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562264    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:16:24.565294 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562266    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:16:24.565294 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562269    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:16:24.565294 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562271    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:16:24.565294 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562274    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:16:24.565294 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562277    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:16:24.565294 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562279    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:16:24.565294 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562281    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:16:24.565294 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562284    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:16:24.565294 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562287    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:16:24.565294 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562291    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:16:24.565294 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562294    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:16:24.565294 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562296    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:16:24.565294 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562299    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:16:24.565294 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562301    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:16:24.565294 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562304    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:16:24.565788 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562306    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:16:24.565788 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562309    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:16:24.565788 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562311    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:16:24.565788 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562314    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:16:24.565788 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562316    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:16:24.565788 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562318    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:16:24.565788 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562321    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:16:24.565788 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562325    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:16:24.565788 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562328    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:16:24.565788 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562331    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:16:24.565788 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562333    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:16:24.565788 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562336    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:16:24.565788 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562338    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:16:24.565788 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562341    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:16:24.565788 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562343    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:16:24.565788 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562346    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:16:24.565788 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562349    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:16:24.565788 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562351    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:16:24.565788 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562367    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:16:24.566267 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562370    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:16:24.566267 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562373    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:16:24.566267 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562378    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:16:24.566267 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562382    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:16:24.566267 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562386    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:16:24.566267 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562391    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:16:24.566267 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562394    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:16:24.566267 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562403    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:16:24.566267 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562406    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:16:24.566267 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562409    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:16:24.566267 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562412    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:16:24.566267 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562414    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:16:24.566267 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.562417    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:16:24.566267 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563255    2578 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 17:16:24.566267 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563266    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 17:16:24.566267 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563276    2578 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 17:16:24.566267 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563280    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 17:16:24.566267 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563285    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 17:16:24.566267 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563288    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 17:16:24.566267 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563292    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563296    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563299    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563302    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563306    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563309    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563312    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563315    2578 flags.go:64] FLAG: --cgroup-root=""
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563318    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563321    2578 flags.go:64] FLAG: --client-ca-file=""
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563324    2578 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563333    2578 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563340    2578 flags.go:64] FLAG: --cluster-domain=""
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563343    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563346    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563349    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563365    2578 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563369    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563373    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563376    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563379    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563382    2578 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563385    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563388    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 17:16:24.566743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563392    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563394    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563399    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563402    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563405    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563407    2578 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563410    2578 flags.go:64] FLAG: --enable-server="true"
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563413    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563420    2578 flags.go:64] FLAG: --event-burst="100"
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563423    2578 flags.go:64] FLAG: --event-qps="50"
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563426    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563430    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563433    2578 flags.go:64] FLAG: --eviction-hard=""
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563437    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563440    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563444    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563447    2578 flags.go:64] FLAG: --eviction-soft=""
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563450    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563453    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563456    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563464    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563467    2578 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563470    2578 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563473    2578 flags.go:64] FLAG: --feature-gates=""
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563477    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 17:16:24.567352 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563480    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563483    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563486    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563489    2578 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563492    2578 flags.go:64] FLAG: --help="false"
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563495    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563498    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563501    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563504    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563507    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563510    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563513    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563516    2578 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563519    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563521    2578 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563524    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563527    2578 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563531    2578 flags.go:64] FLAG: --kube-reserved=""
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563533    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563536    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563540    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563542    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563545    2578 flags.go:64] FLAG: --lock-file=""
Nov 03 17:16:24.567952 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563548    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563551    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563554    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563559    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563562    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563570    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563574    2578 flags.go:64] FLAG: --logging-format="text"
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563576    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563580    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563583    2578 flags.go:64] FLAG: --manifest-url=""
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563586    2578 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563590    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563594    2578 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563598    2578 flags.go:64] FLAG: --max-pods="110"
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563601    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563604    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563606    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563609    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563612    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563615    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563618    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563624    2578 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563627    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563630    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563633    2578 flags.go:64] FLAG: --pod-cidr=""
Nov 03 17:16:24.568508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563636    2578 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563639    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563642    2578 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563645    2578 flags.go:64] FLAG: --port="10250"
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563648    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563651    2578 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-00fb8ca6b0119f7b9"
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563654    2578 flags.go:64] FLAG: --qos-reserved=""
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563657    2578 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563660    2578 flags.go:64] FLAG: --register-node="true"
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563662    2578 flags.go:64] FLAG: --register-with-taints=""
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563666    2578 flags.go:64] FLAG: --registry-burst="10"
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563669    2578 flags.go:64] FLAG: --registry-qps="5"
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563672    2578 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563675    2578 flags.go:64] FLAG: --reserved-memory=""
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563679    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563682    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563685    2578 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563688    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563691    2578 flags.go:64] FLAG: --runonce="false"
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563694    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563697    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563700    2578 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563703    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563706    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563709    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563712    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 17:16:24.569102 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563715    2578 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563718    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563721    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563723    2578 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563726    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563729    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563732    2578 flags.go:64] FLAG: --system-cgroups=""
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563735    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563742    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563747    2578 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563750    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563754    2578 flags.go:64] FLAG: --tls-min-version=""
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563757    2578 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563760    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563763    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563766    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563769    2578 flags.go:64] FLAG: --v="2"
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563773    2578 flags.go:64] FLAG: --version="false"
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563777    2578 flags.go:64] FLAG: --vmodule=""
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563781    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.563785    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.563954    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.563958    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.563961    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:16:24.569793 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.563964    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:16:24.570352 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.563967    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:16:24.570352 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.563970    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:16:24.570352 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.563972    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:16:24.570352 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.563975    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:16:24.570352 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.563978    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:16:24.570352 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.563980    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:16:24.570352 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.563983    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:16:24.570352 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.563985    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:16:24.570352 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.563988    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:16:24.570352 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.563990    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:16:24.570352 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.563993    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:16:24.570352 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.563996    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:16:24.570352 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.563998    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:16:24.570352 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564001    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:16:24.570352 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564004    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:16:24.570352 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564008    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:16:24.570352 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564012    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:16:24.570352 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564015    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:16:24.570352 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564018    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:16:24.570352 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564020    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:16:24.570842 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564023    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:16:24.570842 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564026    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:16:24.570842 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564028    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:16:24.570842 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564031    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:16:24.570842 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564033    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:16:24.570842 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564036    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:16:24.570842 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564038    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:16:24.570842 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564041    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:16:24.570842 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564049    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:16:24.570842 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564053    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:16:24.570842 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564056    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:16:24.570842 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564060    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:16:24.570842 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564063    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:16:24.570842 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564065    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:16:24.570842 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564068    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:16:24.570842 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564071    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:16:24.570842 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564074    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:16:24.570842 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564077    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:16:24.570842 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564080    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:16:24.571328 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564082    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:16:24.571328 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564085    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:16:24.571328 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564087    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:16:24.571328 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564090    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:16:24.571328 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564092    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:16:24.571328 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564095    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:16:24.571328 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564097    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:16:24.571328 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564100    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:16:24.571328 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564104    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:16:24.571328 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564109    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:16:24.571328 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564111    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:16:24.571328 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564114    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:16:24.571328 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564116    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:16:24.571328 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564119    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:16:24.571328 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564122    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:16:24.571328 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564124    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:16:24.571328 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564127    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:16:24.571328 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564129    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:16:24.571328 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564132    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:16:24.571803 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564135    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:16:24.571803 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564138    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:16:24.571803 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564140    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:16:24.571803 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564152    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:16:24.571803 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564155    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:16:24.571803 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564158    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:16:24.571803 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564160    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:16:24.571803 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564163    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:16:24.571803 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564166    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:16:24.571803 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564168    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:16:24.571803 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564171    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:16:24.571803 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564174    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:16:24.571803 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564176    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:16:24.571803 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564179    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:16:24.571803 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564181    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:16:24.571803 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564184    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:16:24.571803 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564186    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:16:24.571803 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564188    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:16:24.571803 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564191    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:16:24.571803 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564193    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:16:24.572280 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564196    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:16:24.572280 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564199    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:16:24.572280 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564204    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:16:24.572280 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564206    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:16:24.572280 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564209    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:16:24.572280 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564211    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:16:24.572280 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564214    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:16:24.572280 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564217    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:16:24.572280 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564219    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:16:24.572280 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564222    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:16:24.572280 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564224    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:16:24.572280 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564227    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:16:24.572280 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564229    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:16:24.572280 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564232    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:16:24.572280 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564234    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:16:24.572280 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564237    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:16:24.572280 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564245    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:16:24.572280 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564248    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:16:24.572280 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564252    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:16:24.572280 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564256    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:16:24.572757 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564259    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:16:24.572757 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564262    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:16:24.572757 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564266    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:16:24.572757 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564269    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:16:24.572757 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.564295    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:16:24.572757 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564343    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:16:24.572757 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564348    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:16:24.572757 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564351    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:16:24.572757 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564365    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:16:24.572757 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564368    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:16:24.572757 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564371    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:16:24.572757 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564373    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:16:24.572757 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564376    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:16:24.572757 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564379    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:16:24.572757 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564384    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:16:24.573111 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564386    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:16:24.573111 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564389    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:16:24.573111 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564391    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:16:24.573111 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564394    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:16:24.573111 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564397    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:16:24.573111 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564399    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:16:24.573111 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564402    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:16:24.573111 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564404    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:16:24.573111 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564407    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:16:24.573111 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564410    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:16:24.573111 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564412    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:16:24.573111 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564415    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:16:24.573111 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564417    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:16:24.573111 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564420    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:16:24.573111 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564428    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:16:24.573111 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564431    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:16:24.573111 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564434    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:16:24.573111 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564436    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:16:24.573111 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564439    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:16:24.573111 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564441    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:16:24.573635 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564444    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:16:24.573635 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564446    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:16:24.573635 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564449    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:16:24.573635 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564451    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:16:24.573635 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564459    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:16:24.573635 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564463    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:16:24.573635 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564467    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:16:24.573635 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564470    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:16:24.573635 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564473    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:16:24.573635 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564476    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:16:24.573635 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564479    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:16:24.573635 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564484    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:16:24.573635 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564486    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:16:24.573635 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564489    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:16:24.573635 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564491    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:16:24.573635 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564494    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:16:24.573635 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564497    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:16:24.573635 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564499    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:16:24.573635 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564502    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:16:24.574075 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564504    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:16:24.574075 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564507    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:16:24.574075 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564509    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:16:24.574075 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564512    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:16:24.574075 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564515    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:16:24.574075 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564517    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:16:24.574075 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564520    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:16:24.574075 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564522    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:16:24.574075 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564531    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:16:24.574075 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564535    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:16:24.574075 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564537    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:16:24.574075 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564539    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:16:24.574075 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564542    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:16:24.574075 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564544    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:16:24.574075 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564548    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:16:24.574075 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564550    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:16:24.574075 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564553    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:16:24.574075 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564555    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:16:24.574075 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564557    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:16:24.574075 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564560    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:16:24.574554 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564563    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:16:24.574554 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564565    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:16:24.574554 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564567    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:16:24.574554 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564570    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:16:24.574554 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564575    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:16:24.574554 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564578    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:16:24.574554 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564580    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:16:24.574554 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564583    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:16:24.574554 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564585    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:16:24.574554 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564588    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:16:24.574554 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564590    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:16:24.574554 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564593    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:16:24.574554 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564595    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:16:24.574554 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564598    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:16:24.574554 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564600    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:16:24.574554 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564603    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:16:24.574554 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564606    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:16:24.574554 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564609    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:16:24.574554 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564611    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:16:24.574554 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564613    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:16:24.575016 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564616    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:16:24.575016 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564624    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:16:24.575016 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564627    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:16:24.575016 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564630    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:16:24.575016 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564633    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:16:24.575016 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564635    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:16:24.575016 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564637    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:16:24.575016 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564640    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:16:24.575016 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564642    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:16:24.575016 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564645    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:16:24.575016 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564647    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:16:24.575016 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564650    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:16:24.575016 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564653    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:16:24.575016 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564657    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:16:24.575016 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564660    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:16:24.575016 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564663    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:16:24.575016 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.564668    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:16:24.575016 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.571841    2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 17:16:24.575016 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.571856    2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 17:16:24.575016 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.571872    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 17:16:24.575512 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.571876    2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 17:16:24.575512 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.571952    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:16:24.575512 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.571958    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:16:24.575512 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.571961    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:16:24.575512 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.571964    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:16:24.575512 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.571967    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:16:24.575512 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.571969    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:16:24.575512 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.571972    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:16:24.575512 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.571975    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:16:24.575512 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.571977    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:16:24.575512 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.571980    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:16:24.575512 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.571982    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:16:24.575512 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.571986    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:16:24.575512 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.571996    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:16:24.575512 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.571999    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:16:24.575512 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572002    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:16:24.575512 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572005    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:16:24.575512 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572008    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:16:24.575922 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572011    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:16:24.575922 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572013    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:16:24.575922 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572016    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:16:24.575922 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572019    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:16:24.575922 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572021    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:16:24.575922 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572024    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:16:24.575922 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572028    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:16:24.575922 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572030    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:16:24.575922 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572033    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:16:24.575922 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572035    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:16:24.575922 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572038    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:16:24.575922 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572041    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:16:24.575922 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572044    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:16:24.575922 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572047    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:16:24.575922 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572049    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:16:24.575922 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572052    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:16:24.575922 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572054    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:16:24.575922 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572057    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:16:24.575922 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572061    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:16:24.575922 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572064    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:16:24.576429 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572067    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:16:24.576429 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572070    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:16:24.576429 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572073    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:16:24.576429 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572076    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:16:24.576429 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572078    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:16:24.576429 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572081    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:16:24.576429 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572083    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:16:24.576429 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572085    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:16:24.576429 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572093    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:16:24.576429 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572096    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:16:24.576429 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572099    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:16:24.576429 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572101    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:16:24.576429 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572104    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:16:24.576429 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572107    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:16:24.576429 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572109    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:16:24.576429 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572112    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:16:24.576429 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572114    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:16:24.576429 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572117    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:16:24.576429 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572119    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:16:24.576866 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572122    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:16:24.576866 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572124    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:16:24.576866 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572127    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:16:24.576866 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572129    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:16:24.576866 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572132    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:16:24.576866 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572135    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:16:24.576866 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572137    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:16:24.576866 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572140    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:16:24.576866 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572142    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:16:24.576866 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572145    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:16:24.576866 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572148    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:16:24.576866 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572150    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:16:24.576866 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572153    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:16:24.576866 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572155    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:16:24.576866 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572158    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:16:24.576866 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572160    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:16:24.576866 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572163    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:16:24.576866 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572165    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:16:24.576866 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572168    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:16:24.576866 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572170    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:16:24.577331 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572172    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:16:24.577331 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572175    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:16:24.577331 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572183    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:16:24.577331 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572186    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:16:24.577331 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572189    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:16:24.577331 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572191    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:16:24.577331 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572194    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:16:24.577331 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572196    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:16:24.577331 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572199    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:16:24.577331 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572201    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:16:24.577331 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572204    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:16:24.577331 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572206    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:16:24.577331 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572209    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:16:24.577331 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572211    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:16:24.577331 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572214    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:16:24.577331 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572216    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:16:24.577331 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572220    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:16:24.577331 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572222    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:16:24.577331 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572224    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:16:24.577800 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572227    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:16:24.577800 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572229    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:16:24.577800 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572232    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:16:24.577800 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572235    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:16:24.577800 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572237    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:16:24.577800 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572239    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:16:24.577800 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572242    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:16:24.577800 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572244    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:16:24.577800 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572247    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:16:24.577800 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572249    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:16:24.577800 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572252    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:16:24.577800 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.572279    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:16:24.577800 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572505    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:16:24.577800 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572511    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:16:24.577800 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572514    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:16:24.577800 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572517    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:16:24.578175 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572525    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:16:24.578175 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572528    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:16:24.578175 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572531    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:16:24.578175 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572534    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:16:24.578175 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572536    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:16:24.578175 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572539    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:16:24.578175 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572541    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:16:24.578175 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572544    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:16:24.578175 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572546    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:16:24.578175 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572549    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:16:24.578175 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572551    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:16:24.578175 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572554    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:16:24.578175 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572556    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:16:24.578175 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572559    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:16:24.578175 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572562    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:16:24.578175 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572564    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:16:24.578175 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572567    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:16:24.578175 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572569    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:16:24.578175 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572572    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:16:24.578640 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572574    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:16:24.578640 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572577    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:16:24.578640 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572579    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:16:24.578640 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572582    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:16:24.578640 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572584    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:16:24.578640 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572587    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:16:24.578640 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572589    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:16:24.578640 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572592    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:16:24.578640 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572594    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:16:24.578640 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572597    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:16:24.578640 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572599    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:16:24.578640 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572601    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:16:24.578640 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572604    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:16:24.578640 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572607    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:16:24.578640 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572615    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:16:24.578640 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572618    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:16:24.578640 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572620    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:16:24.578640 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572623    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:16:24.578640 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572625    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:16:24.579079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572628    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:16:24.579079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572631    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:16:24.579079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572633    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:16:24.579079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572636    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:16:24.579079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572638    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:16:24.579079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572641    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:16:24.579079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572643    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:16:24.579079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572646    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:16:24.579079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572648    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:16:24.579079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572650    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:16:24.579079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572653    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:16:24.579079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572655    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:16:24.579079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572658    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:16:24.579079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572660    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:16:24.579079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572663    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:16:24.579079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572665    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:16:24.579079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572668    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:16:24.579079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572671    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:16:24.579079 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572673    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:16:24.579595 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572675    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:16:24.579595 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572678    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:16:24.579595 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572680    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:16:24.579595 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572683    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:16:24.579595 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572685    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:16:24.579595 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572688    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:16:24.579595 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572690    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:16:24.579595 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572693    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:16:24.579595 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572695    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:16:24.579595 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572703    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:16:24.579595 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572706    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:16:24.579595 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572708    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:16:24.579595 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572711    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:16:24.579595 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572714    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:16:24.579595 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572716    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:16:24.579595 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572719    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:16:24.579595 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572721    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:16:24.579595 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572724    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:16:24.579595 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572728    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:16:24.579595 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572731    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:16:24.580261 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572734    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:16:24.580261 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572736    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:16:24.580261 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572739    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:16:24.580261 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572741    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:16:24.580261 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572744    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:16:24.580261 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572746    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:16:24.580261 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572749    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:16:24.580261 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572751    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:16:24.580261 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572754    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:16:24.580261 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572756    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:16:24.580261 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572759    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:16:24.580261 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572761    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:16:24.580261 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572764    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:16:24.580261 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572767    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:16:24.580261 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572770    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:16:24.580261 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572774    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:16:24.580261 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572777    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:16:24.580261 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572780    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:16:24.580261 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572783    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:16:24.581059 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572785    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:16:24.581059 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572788    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:16:24.581059 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572790    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:16:24.581059 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572799    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:16:24.581059 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572802    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:16:24.581059 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572805    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:16:24.581059 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.572829    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:16:24.581059 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572880    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:16:24.581059 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572883    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:16:24.581059 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572886    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:16:24.581059 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572889    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:16:24.581059 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572892    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:16:24.581059 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572894    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:16:24.581059 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572897    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:16:24.581059 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572899    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:16:24.581059 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572902    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:16:24.581538 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572907    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:16:24.581538 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572910    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:16:24.581538 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572913    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:16:24.581538 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572915    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:16:24.581538 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572918    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:16:24.581538 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572920    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:16:24.581538 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572923    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:16:24.581538 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572925    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:16:24.581538 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572928    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:16:24.581538 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572931    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:16:24.581538 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572933    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:16:24.581538 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572936    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:16:24.581538 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572938    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:16:24.581538 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572941    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:16:24.581538 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572944    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:16:24.581538 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572946    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:16:24.581538 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572948    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:16:24.581538 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572951    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:16:24.581538 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572953    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:16:24.581538 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572956    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:16:24.582033 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572964    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:16:24.582033 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572967    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:16:24.582033 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572970    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:16:24.582033 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572972    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:16:24.582033 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572974    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:16:24.582033 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572977    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:16:24.582033 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572979    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:16:24.582033 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572982    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:16:24.582033 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572984    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:16:24.582033 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572987    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:16:24.582033 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572990    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:16:24.582033 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572992    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:16:24.582033 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.572997    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:16:24.582033 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573000    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:16:24.582033 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573004    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:16:24.582033 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573007    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:16:24.582033 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573010    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:16:24.582033 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573012    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:16:24.582033 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573015    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:16:24.582033 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573018    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:16:24.582541 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573021    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:16:24.582541 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573024    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:16:24.582541 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573026    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:16:24.582541 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573028    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:16:24.582541 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573031    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:16:24.582541 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573033    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:16:24.582541 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573036    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:16:24.582541 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573038    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:16:24.582541 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573042    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:16:24.582541 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573046    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:16:24.582541 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573050    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:16:24.582541 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573052    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:16:24.582541 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573055    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:16:24.582541 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573063    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:16:24.582541 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573066    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:16:24.582541 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573069    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:16:24.582541 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573072    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:16:24.582541 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573074    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:16:24.582541 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573077    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:16:24.582997 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573080    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:16:24.582997 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573082    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:16:24.582997 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573084    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:16:24.582997 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573087    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:16:24.582997 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573090    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:16:24.582997 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573093    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:16:24.582997 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573096    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:16:24.582997 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573099    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:16:24.582997 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573101    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:16:24.582997 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573104    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:16:24.582997 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573107    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:16:24.582997 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573113    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:16:24.582997 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573116    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:16:24.582997 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573118    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:16:24.582997 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573121    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:16:24.582997 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573124    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:16:24.582997 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573126    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:16:24.582997 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573129    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:16:24.582997 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573131    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:16:24.583620 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573134    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:16:24.583620 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573136    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:16:24.583620 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573139    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:16:24.583620 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573141    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:16:24.583620 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573144    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:16:24.583620 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573146    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:16:24.583620 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573149    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:16:24.583620 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573151    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:16:24.583620 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573159    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:16:24.583620 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573162    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:16:24.583620 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573164    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:16:24.583620 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573167    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:16:24.583620 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573169    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:16:24.583620 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573172    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:16:24.583620 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573174    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:16:24.583620 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573177    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:16:24.583620 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573179    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:16:24.583620 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573182    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:16:24.583620 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:24.573186    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:16:24.583620 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.573232    2578 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 17:16:24.584144 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.576824    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 17:16:24.584144 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.577706    2578 server.go:1014] "Starting client certificate rotation"
Nov 03 17:16:24.584144 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.577825    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 17:16:24.584144 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.577858    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 17:16:24.584144 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.579719    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 17:16:24.584144 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.583033    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 17:16:24.593263 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.593247    2578 log.go:25] "Validated CRI v1 runtime API"
Nov 03 17:16:24.596789 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.596774    2578 log.go:25] "Validated CRI v1 image API"
Nov 03 17:16:24.597669 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.597634    2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 17:16:24.602527 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.602502    2578 fs.go:135] Filesystem UUIDs: map[279dd0c6-92a3-441b-989b-0d0f46d02635:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d99b5073-8e62-430a-b123-093e8eb6f858:/dev/nvme0n1p3]
Nov 03 17:16:24.602593 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.602525    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 17:16:24.607533 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.607435    2578 manager.go:217] Machine: {Timestamp:2026-11-03 17:16:24.606100185 +0000 UTC m=+0.306638657 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100547 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ccaacf930ee484cd321e8f630becd SystemUUID:ec2ccaac-f930-ee48-4cd3-21e8f630becd BootID:a7c7c148-c3c6-40c2-bac8-029ad736844b Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d9:0c:5c:37 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d9:0c:5c:37 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:e7:26:dd:08:6d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 17:16:24.607533 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.607533    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 17:16:24.607625 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.607603    2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 17:16:24.610369 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.610334    2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 17:16:24.610504 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.610371    2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-117.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 17:16:24.610559 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.610507    2578 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 17:16:24.610559 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.610515    2578 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 17:16:24.610559 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.610549    2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 17:16:24.610634 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.610567    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 17:16:24.611495 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.611484    2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 17:16:24.615522 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.615510    2578 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 17:16:24.615629 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.615620    2578 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 17:16:24.616255 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.616247    2578 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 17:16:24.616288 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.616258    2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 17:16:24.616288 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.616269    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 17:16:24.616288 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.616278    2578 kubelet.go:405] "Adding apiserver pod source"
Nov 03 17:16:24.616288 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.616286    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 17:16:24.616711 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.616693    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 17:16:24.618782 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.618767    2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 17:16:24.624902 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.624884    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 17:16:24.625700 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.625687    2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 17:16:24.625748 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.625724    2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 17:16:24.625870 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.625859    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 17:16:24.625903 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.625876    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 17:16:24.625903 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.625882    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 17:16:24.625903 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.625888    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 17:16:24.625903 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.625893    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 17:16:24.625903 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.625898    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 17:16:24.626420 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.626410    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 17:16:24.626450 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.626420    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 17:16:24.626450 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.626428    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 17:16:24.626450 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.626434    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 17:16:24.626450 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.626442    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 17:16:24.626450 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.626451    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 17:16:24.627045 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.627035    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 17:16:24.627045 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.627045    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 17:16:24.630635 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.630621    2578 server.go:1263] "Started kubelet"
Nov 03 17:16:24.630775 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.630707    2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 17:16:24.630819 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.630758    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 17:16:24.630848 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.630824    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 17:16:24.631472 ip-10-0-143-117 systemd[1]: Started Kubernetes Kubelet.
Nov 03 17:16:24.631999 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.631984    2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 17:16:24.634709 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.634697    2578 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 17:16:24.638002 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.637985    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 17:16:24.638478 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:24.638458    2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 17:16:24.639066 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.639054    2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 17:16:24.639780 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:24.639748    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-117.us-west-2.compute.internal\" not found"
Nov 03 17:16:24.639853 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.639802    2578 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 17:16:24.639853 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.639817    2578 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 17:16:24.639954 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.639860    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 17:16:24.640091 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.639994    2578 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 17:16:24.640091 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.640002    2578 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 17:16:24.640786 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.640769    2578 factory.go:55] Registering systemd factory
Nov 03 17:16:24.640786 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.640786    2578 factory.go:223] Registration of the systemd container factory successfully
Nov 03 17:16:24.641018 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.641003    2578 factory.go:153] Registering CRI-O factory
Nov 03 17:16:24.641018 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.641018    2578 factory.go:223] Registration of the crio container factory successfully
Nov 03 17:16:24.641130 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.641069    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 17:16:24.641130 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.641094    2578 factory.go:103] Registering Raw factory
Nov 03 17:16:24.641130 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.641105    2578 manager.go:1199] Started watching for new ooms in manager
Nov 03 17:16:24.642037 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.642012    2578 manager.go:319] Starting recovery of all containers
Nov 03 17:16:24.656983 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.655228    2578 manager.go:324] Recovery completed
Nov 03 17:16:24.661072 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.661059    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:16:24.663002 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.662977    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-117.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:16:24.663104 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.663013    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-117.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:16:24.663104 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.663024    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-117.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:16:24.663634 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.663620    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:16:24.663722 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.663625    2578 cpu_manager.go:226] "Starting" policy="none"
Nov 03 17:16:24.663722 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.663662    2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 17:16:24.663722 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.663681    2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 17:16:24.665163 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.665137    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-117.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:16:24.665261 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.665180    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-117.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:16:24.665261 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.665197    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-117.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:16:24.740946 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:24.740915    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-117.us-west-2.compute.internal\" not found"
Nov 03 17:16:24.789931 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.789898    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-117.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 17:16:24.790110 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:24.789905    2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-117.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 17:16:24.790877 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:24.789905    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-117.us-west-2.compute.internal.18e498761b392646  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-117.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-117.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 17:16:24.630593094 +0000 UTC m=+0.331131567,LastTimestamp:2026-11-03 17:16:24.630593094 +0000 UTC m=+0.331131567,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-117.us-west-2.compute.internal,}"
Nov 03 17:16:24.810808 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.810755    2578 policy_none.go:50] "Start"
Nov 03 17:16:24.810808 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.810779    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 17:16:24.810808 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.810794    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 17:16:24.812917 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.812904    2578 policy_none.go:44] "Start"
Nov 03 17:16:24.841046 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:24.841023    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-117.us-west-2.compute.internal\" not found"
Nov 03 17:16:24.843620 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.843586    2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 17:16:24.845868 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.845850    2578 manager.go:341] "Starting Device Plugin manager"
Nov 03 17:16:24.845954 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:24.845945    2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 17:16:24.846014 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.845957    2578 server.go:87] "Starting device plugin registration server"
Nov 03 17:16:24.846266 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.846253    2578 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 17:16:24.846310 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.846267    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 17:16:24.846406 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.846387    2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 17:16:24.846482 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.846470    2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 17:16:24.846482 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.846480    2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 17:16:24.847013 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:24.846993    2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 17:16:24.847087 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:24.847046    2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-117.us-west-2.compute.internal\" not found"
Nov 03 17:16:24.862553 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.862523    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 17:16:24.862622 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.862572    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tw88f"
Nov 03 17:16:24.927082 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.927048    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 17:16:24.929050 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.928298    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 17:16:24.929050 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.928323    2578 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 17:16:24.929050 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.928345    2578 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 17:16:24.929050 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:24.928396    2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 17:16:24.938806 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.938787    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:16:24.940406 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.940392    2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:16:24.947257 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.947234    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:16:24.949382 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.949369    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-117.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:16:24.949436 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.949396    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-117.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:16:24.949436 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.949407    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-117.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:16:24.949436 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:24.949430    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:16:25.029446 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.029411    2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal"]
Nov 03 17:16:25.029571 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.029483    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:16:25.030430 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.030412    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:16:25.031531 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.031506    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-117.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:16:25.031628 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.031544    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-117.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:16:25.031628 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.031555    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-117.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:16:25.032150 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.032130    2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:16:25.032236 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:25.032156    2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-117.us-west-2.compute.internal\": node \"ip-10-0-143-117.us-west-2.compute.internal\" not found"
Nov 03 17:16:25.032508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.032493    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:16:25.033960 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.033943    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:16:25.035885 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.035870    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:16:25.039726 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.039707    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:16:25.041647 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.041626    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/971b27eeac85072c20cb928b063f959b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal\" (UID: \"971b27eeac85072c20cb928b063f959b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:16:25.041719 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.041672    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/971b27eeac85072c20cb928b063f959b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal\" (UID: \"971b27eeac85072c20cb928b063f959b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:16:25.041719 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.041702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01359929955a1f3cff637d7cc239a2bc-config\") pod \"kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal\" (UID: \"01359929955a1f3cff637d7cc239a2bc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:16:25.063954 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.063937    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:16:25.066293 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.066280    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:16:25.098017 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.098000    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:16:25.098099 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.098086    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:16:25.121579 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.121560    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:16:25.127514 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.127499    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:16:25.127566 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:25.127537    2578 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:16:25.142323 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.142304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/971b27eeac85072c20cb928b063f959b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal\" (UID: \"971b27eeac85072c20cb928b063f959b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:16:25.142401 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.142328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01359929955a1f3cff637d7cc239a2bc-config\") pod \"kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal\" (UID: \"01359929955a1f3cff637d7cc239a2bc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:16:25.142401 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.142344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/971b27eeac85072c20cb928b063f959b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal\" (UID: \"971b27eeac85072c20cb928b063f959b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:16:25.142497 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.142398    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/971b27eeac85072c20cb928b063f959b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal\" (UID: \"971b27eeac85072c20cb928b063f959b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:16:25.142497 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.142409    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01359929955a1f3cff637d7cc239a2bc-config\") pod \"kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal\" (UID: \"01359929955a1f3cff637d7cc239a2bc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:16:25.142497 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.142415    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/971b27eeac85072c20cb928b063f959b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal\" (UID: \"971b27eeac85072c20cb928b063f959b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:16:25.156564 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.156538    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:16:25.156657 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:25.156578    2578 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:16:25.427331 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.427246    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:16:25.432596 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.432184    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:16:25.577567 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.577537    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 17:16:25.578050 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.577652    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 17:16:25.578050 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.577707    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 17:16:25.578050 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.577708    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 17:16:25.578050 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:25.577770    2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ae4f21c5985754fdab24e2f2e3e09060-2a5fa75794b3f366.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-143-117.us-west-2.compute.internal.18e498761d27a25d\": read tcp 10.0.143.117:47116->18.116.56.47:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-143-117.us-west-2.compute.internal.18e498761d27a25d  default   5172 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-117.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-117.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-117.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 17:16:24 +0000 UTC,LastTimestamp:2026-11-03 17:16:25.031526669 +0000 UTC m=+0.732065148,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-117.us-west-2.compute.internal,}"
Nov 03 17:16:25.618132 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.618105    2578 apiserver.go:52] "Watching apiserver"
Nov 03 17:16:25.638896 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.638873    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 17:16:25.867453 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.867346    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 17:11:25 +0000 UTC" deadline="2028-04-16 16:57:57.807616432 +0000 UTC"
Nov 03 17:16:25.867453 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.867392    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12719h41m31.940230314s"
Nov 03 17:16:25.921062 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.921028    2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 17:16:25.921450 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.921423    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal","openshift-network-operator/iptables-alerter-z5v9n","openshift-ovn-kubernetes/ovnkube-node-c5n7n","kube-system/konnectivity-agent-dzlp4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2","openshift-cluster-node-tuning-operator/tuned-4dtnl","openshift-multus/multus-9cgng","kube-system/global-pull-secret-syncer-rt4lh","kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal","openshift-dns/node-resolver-6l5k8","openshift-multus/multus-additional-cni-plugins-b99b8","openshift-network-diagnostics/network-check-target-d252q","openshift-image-registry/node-ca-bkmvr","openshift-multus/network-metrics-daemon-484l6"]
Nov 03 17:16:25.924033 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.924016    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z5v9n"
Nov 03 17:16:25.926308 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.926283    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:25.928504 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.928488    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dzlp4"
Nov 03 17:16:25.930911 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.930893    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:25.933842 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.933819    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:25.936215 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.936199    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9cgng"
Nov 03 17:16:25.938601 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.938583    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:25.938680 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:25.938657    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt4lh" podUID="902e0389-e70e-4d23-a607-d8f4b9da868d"
Nov 03 17:16:25.941110 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.941092    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6l5k8"
Nov 03 17:16:25.943709 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.943690    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:25.946058 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.946038    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:25.946149 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:25.946120    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d252q" podUID="37080dcf-15b0-4d4e-85fa-ff669783f356"
Nov 03 17:16:25.948065 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-kubernetes\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:25.948139 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948071    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-run\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:25.948139 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948087    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-sys\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:25.948139 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hljmh\" (UniqueName: \"kubernetes.io/projected/00710f2d-31bb-42da-9ae8-135ddc117df5-kube-api-access-hljmh\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:25.948258 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948205    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/88482077-204d-4634-8045-c741d2c10d80-iptables-alerter-script\") pod \"iptables-alerter-z5v9n\" (UID: \"88482077-204d-4634-8045-c741d2c10d80\") " pod="openshift-network-operator/iptables-alerter-z5v9n"
Nov 03 17:16:25.948258 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88482077-204d-4634-8045-c741d2c10d80-host-slash\") pod \"iptables-alerter-z5v9n\" (UID: \"88482077-204d-4634-8045-c741d2c10d80\") " pod="openshift-network-operator/iptables-alerter-z5v9n"
Nov 03 17:16:25.948345 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-slash\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:25.948345 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948324    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmd9x\" (UniqueName: \"kubernetes.io/projected/c1e30c92-229f-4030-82eb-9295f5328da4-kube-api-access-bmd9x\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:25.948422 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl2pt\" (UniqueName: \"kubernetes.io/projected/88482077-204d-4634-8045-c741d2c10d80-kube-api-access-hl2pt\") pod \"iptables-alerter-z5v9n\" (UID: \"88482077-204d-4634-8045-c741d2c10d80\") " pod="openshift-network-operator/iptables-alerter-z5v9n"
Nov 03 17:16:25.948472 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-kubelet\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:25.948472 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpc58\" (UniqueName: \"kubernetes.io/projected/8906c21b-3233-4214-8ce1-f9189b79ccd3-kube-api-access-jpc58\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:25.948553 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-tuned\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:25.948553 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a607eca-e6ad-4e5d-aa57-663768133ec0-tmp\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:25.948553 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948520    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:25.948684 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948564    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5923113e-7941-4bf8-bfde-28622a520e86-agent-certs\") pod \"konnectivity-agent-dzlp4\" (UID: \"5923113e-7941-4bf8-bfde-28622a520e86\") " pod="kube-system/konnectivity-agent-dzlp4"
Nov 03 17:16:25.948684 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948609    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bkmvr"
Nov 03 17:16:25.948787 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5923113e-7941-4bf8-bfde-28622a520e86-konnectivity-ca\") pod \"konnectivity-agent-dzlp4\" (UID: \"5923113e-7941-4bf8-bfde-28622a520e86\") " pod="kube-system/konnectivity-agent-dzlp4"
Nov 03 17:16:25.948787 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kx94\" (UniqueName: \"kubernetes.io/projected/e59c4c3e-b582-4b16-a6e3-4aadf0612a32-kube-api-access-2kx94\") pod \"node-resolver-6l5k8\" (UID: \"e59c4c3e-b582-4b16-a6e3-4aadf0612a32\") " pod="openshift-dns/node-resolver-6l5k8"
Nov 03 17:16:25.948868 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-node-log\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:25.948868 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948829    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-host\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:25.948959 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948894    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz4kk\" (UniqueName: \"kubernetes.io/projected/8a607eca-e6ad-4e5d-aa57-663768133ec0-kube-api-access-tz4kk\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:25.948959 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948915    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e59c4c3e-b582-4b16-a6e3-4aadf0612a32-tmp-dir\") pod \"node-resolver-6l5k8\" (UID: \"e59c4c3e-b582-4b16-a6e3-4aadf0612a32\") " pod="openshift-dns/node-resolver-6l5k8"
Nov 03 17:16:25.948959 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-systemd-units\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:25.949063 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948973    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8906c21b-3233-4214-8ce1-f9189b79ccd3-ovnkube-config\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:25.949063 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.948992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-run-openvswitch\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:25.949063 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-modprobe-d\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:25.949158 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949072    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-sysconfig\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:25.949158 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949090    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-run-systemd\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:25.949158 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949105    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-run-ovn-kubernetes\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:25.949158 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-sysctl-d\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:25.949158 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-sysctl-conf\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:25.949364 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949206    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-var-lib-kubelet\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:25.949364 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949251    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-var-lib-kubelet\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:25.949364 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949282    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-cni-bin\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:25.949364 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949305    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8906c21b-3233-4214-8ce1-f9189b79ccd3-env-overrides\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:25.949596 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949371    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-registration-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:25.949596 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-lib-modules\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:25.949596 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-var-lib-openvswitch\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:25.949596 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-cni-netd\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:25.949596 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-device-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:25.949596 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949534    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-etc-selinux\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:25.949596 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949563    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-systemd\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:25.949596 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949588    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-cnibin\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:25.949900 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949624    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-daemon-config\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:25.949900 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-etc-kubernetes\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:25.949900 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949682    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:25.949900 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949733    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-run-k8s-cni-cncf-io\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:25.949900 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949770    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/902e0389-e70e-4d23-a607-d8f4b9da868d-dbus\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:25.949900 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-run-netns\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:25.949900 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949825    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-run-netns\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:25.949900 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949850    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-var-lib-cni-multus\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:25.949900 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949874    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-hostroot\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:25.949900 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-conf-dir\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:25.950268 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-run-ovn\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:25.950268 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949936    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-log-socket\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:25.950268 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949967    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8906c21b-3233-4214-8ce1-f9189b79ccd3-ovn-node-metrics-cert\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:25.950268 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.949998    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-socket-dir-parent\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:25.950268 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.950022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-run-multus-certs\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:25.950268 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.950050    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:25.950268 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.950067    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-var-lib-cni-bin\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:25.950268 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.950121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/902e0389-e70e-4d23-a607-d8f4b9da868d-kubelet-config\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:25.950268 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.950143    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-socket-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:25.950268 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.950203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-system-cni-dir\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:25.950268 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.950257    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00710f2d-31bb-42da-9ae8-135ddc117df5-cni-binary-copy\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:25.950644 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.950274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e59c4c3e-b582-4b16-a6e3-4aadf0612a32-hosts-file\") pod \"node-resolver-6l5k8\" (UID: \"e59c4c3e-b582-4b16-a6e3-4aadf0612a32\") " pod="openshift-dns/node-resolver-6l5k8"
Nov 03 17:16:25.950644 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.950289    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-etc-openvswitch\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:25.950644 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.950305    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8906c21b-3233-4214-8ce1-f9189b79ccd3-ovnkube-script-lib\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:25.950644 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.950323    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-cni-dir\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:25.950644 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.950358    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-sys-fs\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:25.950928 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.950913    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:25.950980 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:25.950966    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484l6" podUID="bd2a75ff-95c3-4b08-b75b-30059af6bd37"
Nov 03 17:16:25.983459 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.983219    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:16:25.983459 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.983241    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 17:16:25.983459 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.983406    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 17:16:25.983459 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.983426    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-czdwp\""
Nov 03 17:16:25.984379 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.984354    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 17:16:25.984559 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.984544    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 17:16:25.984621 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.984602    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c6dxq\""
Nov 03 17:16:25.984659 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.984615    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 17:16:25.984813 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.984800    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 17:16:25.985260 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.985240    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 17:16:25.986599 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.986579    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 17:16:25.986599 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.986590    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 17:16:25.987100 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.987072    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9lkll\""
Nov 03 17:16:25.987100 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.987096    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 17:16:25.987888 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.987869    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 17:16:25.987975 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.987871    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 17:16:25.987975 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.987902    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 17:16:25.988085 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.988007    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2ptg9\""
Nov 03 17:16:25.990793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:25.990776    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:16:26.032292 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.032264    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 17:16:26.032292 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.032263    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 17:16:26.032538 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.032303    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 17:16:26.032538 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.032343    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fmcvz\""
Nov 03 17:16:26.032538 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.032351    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 17:16:26.032538 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.032374    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-whvtr\""
Nov 03 17:16:26.035922 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.035901    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h78gf\""
Nov 03 17:16:26.035922 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.035902    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 17:16:26.036080 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.035933    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 17:16:26.036080 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.035937    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 17:16:26.036080 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.035945    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 17:16:26.036080 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.035985    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 17:16:26.037308 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.037283    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 17:16:26.037397 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.037339    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h6tl4\""
Nov 03 17:16:26.037397 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.037349    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 17:16:26.037397 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.037351    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 17:16:26.037536 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.037404    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-krbpr"
Nov 03 17:16:26.037536 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.037424    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hwgrr\""
Nov 03 17:16:26.037536 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.037500    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 17:16:26.040659 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.040644    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 17:16:26.050672 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.050648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-var-lib-cni-multus\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.050753 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.050692    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-hostroot\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.050753 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.050726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-conf-dir\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.050855 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.050756    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-run-ovn\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.050855 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.050771    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-var-lib-cni-multus\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.050855 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.050776    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-hostroot\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.050855 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.050804    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-conf-dir\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.050855 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.050812    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-log-socket\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.050855 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.050782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-log-socket\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.051094 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.050818    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-run-ovn\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.051094 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.050896    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8906c21b-3233-4214-8ce1-f9189b79ccd3-ovn-node-metrics-cert\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.051094 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.050955    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-socket-dir-parent\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.051094 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.050974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-run-multus-certs\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.051094 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.050996    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.051094 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051040    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.051094 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051042    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-run-multus-certs\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.051094 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051065    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:26.051508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-var-lib-cni-bin\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.051508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051125    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-socket-dir-parent\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.051508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051149    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-var-lib-cni-bin\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.051508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/902e0389-e70e-4d23-a607-d8f4b9da868d-kubelet-config\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:26.051508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051205    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-socket-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:26.051508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051236    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:26.051508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051263    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/902e0389-e70e-4d23-a607-d8f4b9da868d-kubelet-config\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:26.051508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-system-cni-dir\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.051508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051318    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00710f2d-31bb-42da-9ae8-135ddc117df5-cni-binary-copy\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.051508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051300    2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 17:16:26.051508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e59c4c3e-b582-4b16-a6e3-4aadf0612a32-hosts-file\") pod \"node-resolver-6l5k8\" (UID: \"e59c4c3e-b582-4b16-a6e3-4aadf0612a32\") " pod="openshift-dns/node-resolver-6l5k8"
Nov 03 17:16:26.051508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051377    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-system-cni-dir\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.051508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051383    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-socket-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:26.051508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051454    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-etc-openvswitch\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.051508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-etc-openvswitch\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.051508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051456    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e59c4c3e-b582-4b16-a6e3-4aadf0612a32-hosts-file\") pod \"node-resolver-6l5k8\" (UID: \"e59c4c3e-b582-4b16-a6e3-4aadf0612a32\") " pod="openshift-dns/node-resolver-6l5k8"
Nov 03 17:16:26.051508 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051491    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8906c21b-3233-4214-8ce1-f9189b79ccd3-ovnkube-script-lib\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.052346 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051521    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs\") pod \"network-metrics-daemon-484l6\" (UID: \"bd2a75ff-95c3-4b08-b75b-30059af6bd37\") " pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:26.052346 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051549    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-cni-dir\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.052346 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-sys-fs\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:26.052346 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l8qd\" (UniqueName: \"kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd\") pod \"network-check-target-d252q\" (UID: \"37080dcf-15b0-4d4e-85fa-ff669783f356\") " pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:26.052346 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051673    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d9f8f17-bb8e-4f09-ade6-099a78bfe422-serviceca\") pod \"node-ca-bkmvr\" (UID: \"5d9f8f17-bb8e-4f09-ade6-099a78bfe422\") " pod="openshift-image-registry/node-ca-bkmvr"
Nov 03 17:16:26.052346 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051743    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-cni-dir\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.052346 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051755    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-sys-fs\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:26.052346 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051779    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-kubernetes\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.052346 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051819    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-run\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.052346 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-sys\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.052346 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hljmh\" (UniqueName: \"kubernetes.io/projected/00710f2d-31bb-42da-9ae8-135ddc117df5-kube-api-access-hljmh\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.052346 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051875    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/88482077-204d-4634-8045-c741d2c10d80-iptables-alerter-script\") pod \"iptables-alerter-z5v9n\" (UID: \"88482077-204d-4634-8045-c741d2c10d80\") " pod="openshift-network-operator/iptables-alerter-z5v9n"
Nov 03 17:16:26.052346 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88482077-204d-4634-8045-c741d2c10d80-host-slash\") pod \"iptables-alerter-z5v9n\" (UID: \"88482077-204d-4634-8045-c741d2c10d80\") " pod="openshift-network-operator/iptables-alerter-z5v9n"
Nov 03 17:16:26.052346 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-slash\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.052346 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051924    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmd9x\" (UniqueName: \"kubernetes.io/projected/c1e30c92-229f-4030-82eb-9295f5328da4-kube-api-access-bmd9x\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:26.052346 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051923    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00710f2d-31bb-42da-9ae8-135ddc117df5-cni-binary-copy\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.052346 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051941    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl2pt\" (UniqueName: \"kubernetes.io/projected/88482077-204d-4634-8045-c741d2c10d80-kube-api-access-hl2pt\") pod \"iptables-alerter-z5v9n\" (UID: \"88482077-204d-4634-8045-c741d2c10d80\") " pod="openshift-network-operator/iptables-alerter-z5v9n"
Nov 03 17:16:26.053191 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051963    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-kubelet\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.053191 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.051982    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpc58\" (UniqueName: \"kubernetes.io/projected/8906c21b-3233-4214-8ce1-f9189b79ccd3-kube-api-access-jpc58\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.053191 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052000    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d9f8f17-bb8e-4f09-ade6-099a78bfe422-host\") pod \"node-ca-bkmvr\" (UID: \"5d9f8f17-bb8e-4f09-ade6-099a78bfe422\") " pod="openshift-image-registry/node-ca-bkmvr"
Nov 03 17:16:26.053191 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:26.053191 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052035    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-tuned\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.053191 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a607eca-e6ad-4e5d-aa57-663768133ec0-tmp\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.053191 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:26.053191 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052101    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5923113e-7941-4bf8-bfde-28622a520e86-agent-certs\") pod \"konnectivity-agent-dzlp4\" (UID: \"5923113e-7941-4bf8-bfde-28622a520e86\") " pod="kube-system/konnectivity-agent-dzlp4"
Nov 03 17:16:26.053191 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052135    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5923113e-7941-4bf8-bfde-28622a520e86-konnectivity-ca\") pod \"konnectivity-agent-dzlp4\" (UID: \"5923113e-7941-4bf8-bfde-28622a520e86\") " pod="kube-system/konnectivity-agent-dzlp4"
Nov 03 17:16:26.053191 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kx94\" (UniqueName: \"kubernetes.io/projected/e59c4c3e-b582-4b16-a6e3-4aadf0612a32-kube-api-access-2kx94\") pod \"node-resolver-6l5k8\" (UID: \"e59c4c3e-b582-4b16-a6e3-4aadf0612a32\") " pod="openshift-dns/node-resolver-6l5k8"
Nov 03 17:16:26.053191 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052171    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-node-log\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.053191 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052188    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-system-cni-dir\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:26.053191 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052199    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8906c21b-3233-4214-8ce1-f9189b79ccd3-ovnkube-script-lib\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.053191 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-host\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.053191 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052232    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz4kk\" (UniqueName: \"kubernetes.io/projected/8a607eca-e6ad-4e5d-aa57-663768133ec0-kube-api-access-tz4kk\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.053191 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052250    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e59c4c3e-b582-4b16-a6e3-4aadf0612a32-tmp-dir\") pod \"node-resolver-6l5k8\" (UID: \"e59c4c3e-b582-4b16-a6e3-4aadf0612a32\") " pod="openshift-dns/node-resolver-6l5k8"
Nov 03 17:16:26.053191 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-systemd-units\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.054029 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052272    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-run\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.054029 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8906c21b-3233-4214-8ce1-f9189b79ccd3-ovnkube-config\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.054029 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-cni-binary-copy\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:26.054029 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-run-openvswitch\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.054029 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052366    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-kubelet\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.054029 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2wsj\" (UniqueName: \"kubernetes.io/projected/5d9f8f17-bb8e-4f09-ade6-099a78bfe422-kube-api-access-r2wsj\") pod \"node-ca-bkmvr\" (UID: \"5d9f8f17-bb8e-4f09-ade6-099a78bfe422\") " pod="openshift-image-registry/node-ca-bkmvr"
Nov 03 17:16:26.054029 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052410    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/88482077-204d-4634-8045-c741d2c10d80-iptables-alerter-script\") pod \"iptables-alerter-z5v9n\" (UID: \"88482077-204d-4634-8045-c741d2c10d80\") " pod="openshift-network-operator/iptables-alerter-z5v9n"
Nov 03 17:16:26.054029 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052414    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88482077-204d-4634-8045-c741d2c10d80-host-slash\") pod \"iptables-alerter-z5v9n\" (UID: \"88482077-204d-4634-8045-c741d2c10d80\") " pod="openshift-network-operator/iptables-alerter-z5v9n"
Nov 03 17:16:26.054029 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-cnibin\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:26.054029 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-modprobe-d\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.054029 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052501    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-sys\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.054029 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052505    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-sysconfig\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.054029 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-run-systemd\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.054029 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052568    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-run-ovn-kubernetes\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.054029 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052585    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-sysctl-d\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.054029 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052611    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-sysctl-conf\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.054029 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-var-lib-kubelet\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.055015 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052672    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-var-lib-kubelet\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.055015 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-cni-bin\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.055015 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052327    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-kubernetes\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.055015 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052812    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-slash\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.055015 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052879    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-var-lib-kubelet\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.055015 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052916    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-modprobe-d\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.055015 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:26.052937    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:16:26.055015 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.052964    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-host\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.055015 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.053006    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-node-log\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.055015 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:26.053026    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret podName:902e0389-e70e-4d23-a607-d8f4b9da868d nodeName:}" failed. No retries permitted until 2026-11-03 17:16:26.552984963 +0000 UTC m=+2.253523445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret") pod "global-pull-secret-syncer-rt4lh" (UID: "902e0389-e70e-4d23-a607-d8f4b9da868d") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:16:26.055015 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.053046    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8906c21b-3233-4214-8ce1-f9189b79ccd3-env-overrides\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.055015 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.053077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-registration-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:26.055015 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.053111    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-857nt\" (UniqueName: \"kubernetes.io/projected/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-kube-api-access-857nt\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:26.055015 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.053142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-lib-modules\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.055015 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.053365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-var-lib-openvswitch\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.055015 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.053398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-cni-netd\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.055015 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.053504    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5923113e-7941-4bf8-bfde-28622a520e86-konnectivity-ca\") pod \"konnectivity-agent-dzlp4\" (UID: \"5923113e-7941-4bf8-bfde-28622a520e86\") " pod="kube-system/konnectivity-agent-dzlp4"
Nov 03 17:16:26.055765 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.053518    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8906c21b-3233-4214-8ce1-f9189b79ccd3-ovnkube-config\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.055765 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.053545    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-systemd-units\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.055765 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.053627    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-sysctl-conf\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.055765 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.053652    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-sysconfig\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.055765 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.053684    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-device-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:26.055765 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.053723    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-lib-modules\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.055765 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.053805    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-registration-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:26.055765 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.053950    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-sysctl-d\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.055765 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054014    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-var-lib-kubelet\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.055765 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054034    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-cni-bin\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.055765 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054092    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8906c21b-3233-4214-8ce1-f9189b79ccd3-env-overrides\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.055765 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054099    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-run-systemd\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.055765 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054138    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-run-openvswitch\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.055765 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054168    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-run-ovn-kubernetes\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.055765 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-etc-selinux\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:26.055765 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5th7\" (UniqueName: \"kubernetes.io/projected/bd2a75ff-95c3-4b08-b75b-30059af6bd37-kube-api-access-z5th7\") pod \"network-metrics-daemon-484l6\" (UID: \"bd2a75ff-95c3-4b08-b75b-30059af6bd37\") " pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:26.055765 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054215    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-device-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:26.056477 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054247    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-var-lib-openvswitch\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.056477 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054280    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-systemd\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.056477 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054299    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-cnibin\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.056477 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-daemon-config\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.056477 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054353    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-etc-kubernetes\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.056477 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054366    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e59c4c3e-b582-4b16-a6e3-4aadf0612a32-tmp-dir\") pod \"node-resolver-6l5k8\" (UID: \"e59c4c3e-b582-4b16-a6e3-4aadf0612a32\") " pod="openshift-dns/node-resolver-6l5k8"
Nov 03 17:16:26.056477 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054378    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-cni-netd\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.056477 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054382    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:26.056477 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-run-k8s-cni-cncf-io\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.056477 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/902e0389-e70e-4d23-a607-d8f4b9da868d-dbus\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:26.056477 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054460    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-etc-selinux\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:26.056477 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054507    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-run-k8s-cni-cncf-io\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.056477 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054514    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-cnibin\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.056477 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054550    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-run-netns\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.056477 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054557    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-etc-kubernetes\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.056477 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-run-netns\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.056477 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054598    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:26.056477 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054514    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-systemd\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.057055 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054685    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-run-netns\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.057055 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054742    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/902e0389-e70e-4d23-a607-d8f4b9da868d-dbus\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:26.057055 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054788    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-run-netns\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.057055 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.054976    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-daemon-config\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.057055 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.055660    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a607eca-e6ad-4e5d-aa57-663768133ec0-tmp\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.057055 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.055885    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8906c21b-3233-4214-8ce1-f9189b79ccd3-ovn-node-metrics-cert\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.057055 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.056259    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-tuned\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.057575 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.057552    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5923113e-7941-4bf8-bfde-28622a520e86-agent-certs\") pod \"konnectivity-agent-dzlp4\" (UID: \"5923113e-7941-4bf8-bfde-28622a520e86\") " pod="kube-system/konnectivity-agent-dzlp4"
Nov 03 17:16:26.120348 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.120269    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl2pt\" (UniqueName: \"kubernetes.io/projected/88482077-204d-4634-8045-c741d2c10d80-kube-api-access-hl2pt\") pod \"iptables-alerter-z5v9n\" (UID: \"88482077-204d-4634-8045-c741d2c10d80\") " pod="openshift-network-operator/iptables-alerter-z5v9n"
Nov 03 17:16:26.120500 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.120364    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz4kk\" (UniqueName: \"kubernetes.io/projected/8a607eca-e6ad-4e5d-aa57-663768133ec0-kube-api-access-tz4kk\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.120500 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.120418    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpc58\" (UniqueName: \"kubernetes.io/projected/8906c21b-3233-4214-8ce1-f9189b79ccd3-kube-api-access-jpc58\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.120776 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.120757    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hljmh\" (UniqueName: \"kubernetes.io/projected/00710f2d-31bb-42da-9ae8-135ddc117df5-kube-api-access-hljmh\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.120839 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.120804    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kx94\" (UniqueName: \"kubernetes.io/projected/e59c4c3e-b582-4b16-a6e3-4aadf0612a32-kube-api-access-2kx94\") pod \"node-resolver-6l5k8\" (UID: \"e59c4c3e-b582-4b16-a6e3-4aadf0612a32\") " pod="openshift-dns/node-resolver-6l5k8"
Nov 03 17:16:26.121336 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.121315    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmd9x\" (UniqueName: \"kubernetes.io/projected/c1e30c92-229f-4030-82eb-9295f5328da4-kube-api-access-bmd9x\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:26.155817 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.155790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5th7\" (UniqueName: \"kubernetes.io/projected/bd2a75ff-95c3-4b08-b75b-30059af6bd37-kube-api-access-z5th7\") pod \"network-metrics-daemon-484l6\" (UID: \"bd2a75ff-95c3-4b08-b75b-30059af6bd37\") " pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:26.155931 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.155849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:26.155931 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.155881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:26.155931 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.155926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs\") pod \"network-metrics-daemon-484l6\" (UID: \"bd2a75ff-95c3-4b08-b75b-30059af6bd37\") " pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:26.156093 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.155953    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l8qd\" (UniqueName: \"kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd\") pod \"network-check-target-d252q\" (UID: \"37080dcf-15b0-4d4e-85fa-ff669783f356\") " pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:26.156093 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.155979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d9f8f17-bb8e-4f09-ade6-099a78bfe422-serviceca\") pod \"node-ca-bkmvr\" (UID: \"5d9f8f17-bb8e-4f09-ade6-099a78bfe422\") " pod="openshift-image-registry/node-ca-bkmvr"
Nov 03 17:16:26.156194 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.156125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d9f8f17-bb8e-4f09-ade6-099a78bfe422-host\") pod \"node-ca-bkmvr\" (UID: \"5d9f8f17-bb8e-4f09-ade6-099a78bfe422\") " pod="openshift-image-registry/node-ca-bkmvr"
Nov 03 17:16:26.156194 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.156163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:26.156300 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.156231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-system-cni-dir\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:26.156300 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.156269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-cni-binary-copy\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:26.156433 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.156301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2wsj\" (UniqueName: \"kubernetes.io/projected/5d9f8f17-bb8e-4f09-ade6-099a78bfe422-kube-api-access-r2wsj\") pod \"node-ca-bkmvr\" (UID: \"5d9f8f17-bb8e-4f09-ade6-099a78bfe422\") " pod="openshift-image-registry/node-ca-bkmvr"
Nov 03 17:16:26.156433 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.156329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-cnibin\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:26.156433 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.156370    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-857nt\" (UniqueName: \"kubernetes.io/projected/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-kube-api-access-857nt\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:26.156433 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:26.156377    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:16:26.156628 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:26.156473    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs podName:bd2a75ff-95c3-4b08-b75b-30059af6bd37 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:26.656452222 +0000 UTC m=+2.356990689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs") pod "network-metrics-daemon-484l6" (UID: "bd2a75ff-95c3-4b08-b75b-30059af6bd37") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:16:26.156628 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.156544    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d9f8f17-bb8e-4f09-ade6-099a78bfe422-host\") pod \"node-ca-bkmvr\" (UID: \"5d9f8f17-bb8e-4f09-ade6-099a78bfe422\") " pod="openshift-image-registry/node-ca-bkmvr"
Nov 03 17:16:26.156628 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.156554    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:26.156628 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.156548    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d9f8f17-bb8e-4f09-ade6-099a78bfe422-serviceca\") pod \"node-ca-bkmvr\" (UID: \"5d9f8f17-bb8e-4f09-ade6-099a78bfe422\") " pod="openshift-image-registry/node-ca-bkmvr"
Nov 03 17:16:26.156772 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.156623    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:26.156772 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.156643    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-cnibin\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:26.156772 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.156628    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-system-cni-dir\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:26.156772 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.156728    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:26.156887 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.156792    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-cni-binary-copy\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:26.226840 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:26.226800    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:16:26.226840 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:26.226827    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:16:26.226840 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:26.226840    2578 projected.go:196] Error preparing data for projected volume kube-api-access-8l8qd for pod openshift-network-diagnostics/network-check-target-d252q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:16:26.227093 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:26.226918    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd podName:37080dcf-15b0-4d4e-85fa-ff669783f356 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:26.726893736 +0000 UTC m=+2.427432219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8l8qd" (UniqueName: "kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd") pod "network-check-target-d252q" (UID: "37080dcf-15b0-4d4e-85fa-ff669783f356") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:16:26.229121 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.229088    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-857nt\" (UniqueName: \"kubernetes.io/projected/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-kube-api-access-857nt\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:26.229223 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.229144    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5th7\" (UniqueName: \"kubernetes.io/projected/bd2a75ff-95c3-4b08-b75b-30059af6bd37-kube-api-access-z5th7\") pod \"network-metrics-daemon-484l6\" (UID: \"bd2a75ff-95c3-4b08-b75b-30059af6bd37\") " pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:26.229223 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.229159    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2wsj\" (UniqueName: \"kubernetes.io/projected/5d9f8f17-bb8e-4f09-ade6-099a78bfe422-kube-api-access-r2wsj\") pod \"node-ca-bkmvr\" (UID: \"5d9f8f17-bb8e-4f09-ade6-099a78bfe422\") " pod="openshift-image-registry/node-ca-bkmvr"
Nov 03 17:16:26.262719 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.262683    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z5v9n"
Nov 03 17:16:26.294951 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.294920    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:26.297644 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.297620    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dzlp4"
Nov 03 17:16:26.304791 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.304756    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:16:26.318770 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.318743    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:16:26.349262 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.349214    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9cgng"
Nov 03 17:16:26.358744 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.358714    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6l5k8"
Nov 03 17:16:26.393539 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.393443    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bkmvr"
Nov 03 17:16:26.397450 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.397423    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:16:26.451189 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.451161    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:16:26.560674 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.560639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:26.560916 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:26.560776    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:16:26.560916 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:26.560836    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret podName:902e0389-e70e-4d23-a607-d8f4b9da868d nodeName:}" failed. No retries permitted until 2026-11-03 17:16:27.560822903 +0000 UTC m=+3.261361365 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret") pod "global-pull-secret-syncer-rt4lh" (UID: "902e0389-e70e-4d23-a607-d8f4b9da868d") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:16:26.632144 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:26.632115    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01359929955a1f3cff637d7cc239a2bc.slice/crio-74523671170fb1512036d6c47d3b5c4cd185b44d3dc90ff6412360d24a69690f WatchSource:0}: Error finding container 74523671170fb1512036d6c47d3b5c4cd185b44d3dc90ff6412360d24a69690f: Status 404 returned error can't find the container with id 74523671170fb1512036d6c47d3b5c4cd185b44d3dc90ff6412360d24a69690f
Nov 03 17:16:26.633737 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:26.633706    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971b27eeac85072c20cb928b063f959b.slice/crio-d575a9e899a6ee915a4ab04da5ad50997b35b07410a6755aa4fcf3c7a98e9d11 WatchSource:0}: Error finding container d575a9e899a6ee915a4ab04da5ad50997b35b07410a6755aa4fcf3c7a98e9d11: Status 404 returned error can't find the container with id d575a9e899a6ee915a4ab04da5ad50997b35b07410a6755aa4fcf3c7a98e9d11
Nov 03 17:16:26.638083 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.638069    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:16:26.661135 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.661082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs\") pod \"network-metrics-daemon-484l6\" (UID: \"bd2a75ff-95c3-4b08-b75b-30059af6bd37\") " pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:26.661216 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:26.661204    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:16:26.661254 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:26.661249    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs podName:bd2a75ff-95c3-4b08-b75b-30059af6bd37 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:27.661236556 +0000 UTC m=+3.361775018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs") pod "network-metrics-daemon-484l6" (UID: "bd2a75ff-95c3-4b08-b75b-30059af6bd37") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:16:26.762210 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.762159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l8qd\" (UniqueName: \"kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd\") pod \"network-check-target-d252q\" (UID: \"37080dcf-15b0-4d4e-85fa-ff669783f356\") " pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:26.762210 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:26.762207    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:16:26.762448 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:26.762228    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:16:26.762448 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:26.762238    2578 projected.go:196] Error preparing data for projected volume kube-api-access-8l8qd for pod openshift-network-diagnostics/network-check-target-d252q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:16:26.762448 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:26.762292    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd podName:37080dcf-15b0-4d4e-85fa-ff669783f356 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:27.762275723 +0000 UTC m=+3.462814188 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8l8qd" (UniqueName: "kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd") pod "network-check-target-d252q" (UID: "37080dcf-15b0-4d4e-85fa-ff669783f356") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:16:26.835554 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.835522    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:16:26.934840 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.934735    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal" event={"ID":"971b27eeac85072c20cb928b063f959b","Type":"ContainerStarted","Data":"d575a9e899a6ee915a4ab04da5ad50997b35b07410a6755aa4fcf3c7a98e9d11"}
Nov 03 17:16:26.935236 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:26.935213    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal" event={"ID":"01359929955a1f3cff637d7cc239a2bc","Type":"ContainerStarted","Data":"74523671170fb1512036d6c47d3b5c4cd185b44d3dc90ff6412360d24a69690f"}
Nov 03 17:16:27.040455 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:27.040388    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:11:26 +0000 UTC" deadline="2028-05-17 15:27:36.074381649 +0000 UTC"
Nov 03 17:16:27.040455 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:27.040445    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13462h11m9.033940831s"
Nov 03 17:16:27.171301 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:27.171274    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e30c92_229f_4030_82eb_9295f5328da4.slice/crio-dfe4d3f2f599f651419b79d4758312d379c6a971c82608c6aa290b494658d70c WatchSource:0}: Error finding container dfe4d3f2f599f651419b79d4758312d379c6a971c82608c6aa290b494658d70c: Status 404 returned error can't find the container with id dfe4d3f2f599f651419b79d4758312d379c6a971c82608c6aa290b494658d70c
Nov 03 17:16:27.192585 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:27.192560    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8906c21b_3233_4214_8ce1_f9189b79ccd3.slice/crio-38d59c19828a6bacf51ada0988f5e0ec2477f8d6fa87199490886da6f1bf6528 WatchSource:0}: Error finding container 38d59c19828a6bacf51ada0988f5e0ec2477f8d6fa87199490886da6f1bf6528: Status 404 returned error can't find the container with id 38d59c19828a6bacf51ada0988f5e0ec2477f8d6fa87199490886da6f1bf6528
Nov 03 17:16:27.205065 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:27.205046    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:16:27.228087 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:27.228063    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5923113e_7941_4bf8_bfde_28622a520e86.slice/crio-87f5d720e3b37e76574e95c4eae2cbc908611871a920c814d52e576eb107e365 WatchSource:0}: Error finding container 87f5d720e3b37e76574e95c4eae2cbc908611871a920c814d52e576eb107e365: Status 404 returned error can't find the container with id 87f5d720e3b37e76574e95c4eae2cbc908611871a920c814d52e576eb107e365
Nov 03 17:16:27.274584 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:27.274555    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a7781b_32fa_4f63_ab0e_663e1f2c6abb.slice/crio-1697bed59d554b35eb3bcdf037c62cf04db88502c215f7b4275cc6cbb4f0fcf7 WatchSource:0}: Error finding container 1697bed59d554b35eb3bcdf037c62cf04db88502c215f7b4275cc6cbb4f0fcf7: Status 404 returned error can't find the container with id 1697bed59d554b35eb3bcdf037c62cf04db88502c215f7b4275cc6cbb4f0fcf7
Nov 03 17:16:27.275173 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:27.275148    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59c4c3e_b582_4b16_a6e3_4aadf0612a32.slice/crio-57145a06bca89cdf364ce88792f350f4ffdcc3f29175c41550c42efd09a11b5f WatchSource:0}: Error finding container 57145a06bca89cdf364ce88792f350f4ffdcc3f29175c41550c42efd09a11b5f: Status 404 returned error can't find the container with id 57145a06bca89cdf364ce88792f350f4ffdcc3f29175c41550c42efd09a11b5f
Nov 03 17:16:27.330302 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:27.330274    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00710f2d_31bb_42da_9ae8_135ddc117df5.slice/crio-a309fe29a795ecf640b84d9617014ca6b16a5866db895234b68de3d37d75632e WatchSource:0}: Error finding container a309fe29a795ecf640b84d9617014ca6b16a5866db895234b68de3d37d75632e: Status 404 returned error can't find the container with id a309fe29a795ecf640b84d9617014ca6b16a5866db895234b68de3d37d75632e
Nov 03 17:16:27.352089 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:27.352064    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9f8f17_bb8e_4f09_ade6_099a78bfe422.slice/crio-5a509155bce896f34b6b640c82b27e12f25852c8d8b9e7a1b08f2fe8e6d257eb WatchSource:0}: Error finding container 5a509155bce896f34b6b640c82b27e12f25852c8d8b9e7a1b08f2fe8e6d257eb: Status 404 returned error can't find the container with id 5a509155bce896f34b6b640c82b27e12f25852c8d8b9e7a1b08f2fe8e6d257eb
Nov 03 17:16:27.537902 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:16:27.537869    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a607eca_e6ad_4e5d_aa57_663768133ec0.slice/crio-5c516c9d051b5b234e0203098e1a0879f942e8a35a0f19aee2a84979cc9ccdc7 WatchSource:0}: Error finding container 5c516c9d051b5b234e0203098e1a0879f942e8a35a0f19aee2a84979cc9ccdc7: Status 404 returned error can't find the container with id 5c516c9d051b5b234e0203098e1a0879f942e8a35a0f19aee2a84979cc9ccdc7
Nov 03 17:16:27.567520 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:27.567481    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:27.567684 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:27.567660    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:16:27.567742 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:27.567718    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret podName:902e0389-e70e-4d23-a607-d8f4b9da868d nodeName:}" failed. No retries permitted until 2026-11-03 17:16:29.567703165 +0000 UTC m=+5.268241627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret") pod "global-pull-secret-syncer-rt4lh" (UID: "902e0389-e70e-4d23-a607-d8f4b9da868d") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:16:27.668637 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:27.668597    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs\") pod \"network-metrics-daemon-484l6\" (UID: \"bd2a75ff-95c3-4b08-b75b-30059af6bd37\") " pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:27.669216 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:27.668774    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:16:27.669216 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:27.668860    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs podName:bd2a75ff-95c3-4b08-b75b-30059af6bd37 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:29.668838215 +0000 UTC m=+5.369376699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs") pod "network-metrics-daemon-484l6" (UID: "bd2a75ff-95c3-4b08-b75b-30059af6bd37") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:16:27.769963 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:27.769824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l8qd\" (UniqueName: \"kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd\") pod \"network-check-target-d252q\" (UID: \"37080dcf-15b0-4d4e-85fa-ff669783f356\") " pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:27.770131 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:27.770015    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:16:27.770131 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:27.770039    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:16:27.770131 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:27.770056    2578 projected.go:196] Error preparing data for projected volume kube-api-access-8l8qd for pod openshift-network-diagnostics/network-check-target-d252q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:16:27.770131 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:27.770118    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd podName:37080dcf-15b0-4d4e-85fa-ff669783f356 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:29.770103749 +0000 UTC m=+5.470642228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8l8qd" (UniqueName: "kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd") pod "network-check-target-d252q" (UID: "37080dcf-15b0-4d4e-85fa-ff669783f356") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:16:27.929490 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:27.929210    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:27.929490 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:27.929261    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:27.929490 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:27.929373    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484l6" podUID="bd2a75ff-95c3-4b08-b75b-30059af6bd37"
Nov 03 17:16:27.929734 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:27.929501    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt4lh" podUID="902e0389-e70e-4d23-a607-d8f4b9da868d"
Nov 03 17:16:27.929734 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:27.929571    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:27.929734 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:27.929643    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d252q" podUID="37080dcf-15b0-4d4e-85fa-ff669783f356"
Nov 03 17:16:27.939487 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:27.939453    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9cgng" event={"ID":"00710f2d-31bb-42da-9ae8-135ddc117df5","Type":"ContainerStarted","Data":"a309fe29a795ecf640b84d9617014ca6b16a5866db895234b68de3d37d75632e"}
Nov 03 17:16:27.941269 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:27.941233    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b99b8" event={"ID":"23a7781b-32fa-4f63-ab0e-663e1f2c6abb","Type":"ContainerStarted","Data":"1697bed59d554b35eb3bcdf037c62cf04db88502c215f7b4275cc6cbb4f0fcf7"}
Nov 03 17:16:27.943037 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:27.943009    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dzlp4" event={"ID":"5923113e-7941-4bf8-bfde-28622a520e86","Type":"ContainerStarted","Data":"87f5d720e3b37e76574e95c4eae2cbc908611871a920c814d52e576eb107e365"}
Nov 03 17:16:27.945670 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:27.945629    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z5v9n" event={"ID":"88482077-204d-4634-8045-c741d2c10d80","Type":"ContainerStarted","Data":"5509ce5deaaa98fcfbfc9bdcc053afcdf3e9d5a0b23a82b025612edf8312b66f"}
Nov 03 17:16:27.948385 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:27.948357    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bkmvr" event={"ID":"5d9f8f17-bb8e-4f09-ade6-099a78bfe422","Type":"ContainerStarted","Data":"5a509155bce896f34b6b640c82b27e12f25852c8d8b9e7a1b08f2fe8e6d257eb"}
Nov 03 17:16:27.950240 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:27.950216    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6l5k8" event={"ID":"e59c4c3e-b582-4b16-a6e3-4aadf0612a32","Type":"ContainerStarted","Data":"57145a06bca89cdf364ce88792f350f4ffdcc3f29175c41550c42efd09a11b5f"}
Nov 03 17:16:27.951867 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:27.951822    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n" event={"ID":"8906c21b-3233-4214-8ce1-f9189b79ccd3","Type":"ContainerStarted","Data":"38d59c19828a6bacf51ada0988f5e0ec2477f8d6fa87199490886da6f1bf6528"}
Nov 03 17:16:27.953577 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:27.953547    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2" event={"ID":"c1e30c92-229f-4030-82eb-9295f5328da4","Type":"ContainerStarted","Data":"dfe4d3f2f599f651419b79d4758312d379c6a971c82608c6aa290b494658d70c"}
Nov 03 17:16:27.956232 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:27.956207    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4dtnl" event={"ID":"8a607eca-e6ad-4e5d-aa57-663768133ec0","Type":"ContainerStarted","Data":"5c516c9d051b5b234e0203098e1a0879f942e8a35a0f19aee2a84979cc9ccdc7"}
Nov 03 17:16:28.041458 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:28.041377    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:11:26 +0000 UTC" deadline="2028-03-30 06:32:59.085599307 +0000 UTC"
Nov 03 17:16:28.041458 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:28.041450    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12301h16m31.044154294s"
Nov 03 17:16:29.591128 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:29.591090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:29.591736 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:29.591269    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:16:29.591736 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:29.591332    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret podName:902e0389-e70e-4d23-a607-d8f4b9da868d nodeName:}" failed. No retries permitted until 2026-11-03 17:16:33.591314114 +0000 UTC m=+9.291852593 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret") pod "global-pull-secret-syncer-rt4lh" (UID: "902e0389-e70e-4d23-a607-d8f4b9da868d") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:16:29.692249 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:29.692200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs\") pod \"network-metrics-daemon-484l6\" (UID: \"bd2a75ff-95c3-4b08-b75b-30059af6bd37\") " pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:29.692523 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:29.692414    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:16:29.692523 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:29.692476    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs podName:bd2a75ff-95c3-4b08-b75b-30059af6bd37 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:33.692455605 +0000 UTC m=+9.392994082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs") pod "network-metrics-daemon-484l6" (UID: "bd2a75ff-95c3-4b08-b75b-30059af6bd37") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:16:29.792839 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:29.792718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l8qd\" (UniqueName: \"kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd\") pod \"network-check-target-d252q\" (UID: \"37080dcf-15b0-4d4e-85fa-ff669783f356\") " pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:29.793003 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:29.792980    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:16:29.793114 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:29.793009    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:16:29.793114 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:29.793022    2578 projected.go:196] Error preparing data for projected volume kube-api-access-8l8qd for pod openshift-network-diagnostics/network-check-target-d252q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:16:29.793267 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:29.793135    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd podName:37080dcf-15b0-4d4e-85fa-ff669783f356 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:33.793061952 +0000 UTC m=+9.493600420 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8l8qd" (UniqueName: "kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd") pod "network-check-target-d252q" (UID: "37080dcf-15b0-4d4e-85fa-ff669783f356") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:16:29.928766 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:29.928716    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:29.928977 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:29.928858    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d252q" podUID="37080dcf-15b0-4d4e-85fa-ff669783f356"
Nov 03 17:16:29.928977 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:29.928933    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:29.929098 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:29.929028    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484l6" podUID="bd2a75ff-95c3-4b08-b75b-30059af6bd37"
Nov 03 17:16:29.929098 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:29.929075    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:29.929210 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:29.929140    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt4lh" podUID="902e0389-e70e-4d23-a607-d8f4b9da868d"
Nov 03 17:16:31.928843 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:31.928555    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:31.929274 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:31.928917    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt4lh" podUID="902e0389-e70e-4d23-a607-d8f4b9da868d"
Nov 03 17:16:31.929274 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:31.928960    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:31.929274 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:31.929050    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484l6" podUID="bd2a75ff-95c3-4b08-b75b-30059af6bd37"
Nov 03 17:16:31.929274 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:31.929100    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:31.929274 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:31.929173    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d252q" podUID="37080dcf-15b0-4d4e-85fa-ff669783f356"
Nov 03 17:16:31.989451 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:31.988644    2578 generic.go:356] "Generic (PLEG): container finished" podID="971b27eeac85072c20cb928b063f959b" containerID="9fe2e9daacd45c17ae4f9856b9be09a3afcb913896a511c25813f09043deb1c6" exitCode=0
Nov 03 17:16:31.989451 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:31.988732    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal" event={"ID":"971b27eeac85072c20cb928b063f959b","Type":"ContainerDied","Data":"9fe2e9daacd45c17ae4f9856b9be09a3afcb913896a511c25813f09043deb1c6"}
Nov 03 17:16:31.992612 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:31.992138    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal" event={"ID":"01359929955a1f3cff637d7cc239a2bc","Type":"ContainerStarted","Data":"95c2a442a3b634b410e3063b219ac4139120fce65246186bca68c30886d2c8eb"}
Nov 03 17:16:31.995961 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:31.995323    2578 generic.go:356] "Generic (PLEG): container finished" podID="23a7781b-32fa-4f63-ab0e-663e1f2c6abb" containerID="abb7f6775b662f2a5802b444939bd70ee54bcaeb07ef69e27c1e8e798b9daedd" exitCode=0
Nov 03 17:16:31.995961 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:31.995410    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b99b8" event={"ID":"23a7781b-32fa-4f63-ab0e-663e1f2c6abb","Type":"ContainerDied","Data":"abb7f6775b662f2a5802b444939bd70ee54bcaeb07ef69e27c1e8e798b9daedd"}
Nov 03 17:16:32.231614 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:32.231555    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal" podStartSLOduration=7.231537516 podStartE2EDuration="7.231537516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:32.230411168 +0000 UTC m=+7.930949651" watchObservedRunningTime="2026-11-03 17:16:32.231537516 +0000 UTC m=+7.932075999"
Nov 03 17:16:33.003050 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:33.002974    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal" event={"ID":"971b27eeac85072c20cb928b063f959b","Type":"ContainerStarted","Data":"a4fdfc3081da1d52d09832c26820b77ea8b2e84474e8e88a977d95bb2b7f1b16"}
Nov 03 17:16:33.006694 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:33.006650    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dzlp4" event={"ID":"5923113e-7941-4bf8-bfde-28622a520e86","Type":"ContainerStarted","Data":"b32944523384af42efeeeb6e37d531d4efbc11a7e67d0bebf7f21c50ccac5f4f"}
Nov 03 17:16:33.254479 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:33.254386    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal" podStartSLOduration=8.254365926 podStartE2EDuration="8.254365926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:33.12690816 +0000 UTC m=+8.827446643" watchObservedRunningTime="2026-11-03 17:16:33.254365926 +0000 UTC m=+8.954904401"
Nov 03 17:16:33.626289 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:33.626043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:33.626289 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:33.626202    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:16:33.626289 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:33.626263    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret podName:902e0389-e70e-4d23-a607-d8f4b9da868d nodeName:}" failed. No retries permitted until 2026-11-03 17:16:41.626244183 +0000 UTC m=+17.326782667 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret") pod "global-pull-secret-syncer-rt4lh" (UID: "902e0389-e70e-4d23-a607-d8f4b9da868d") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:16:33.730184 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:33.729538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs\") pod \"network-metrics-daemon-484l6\" (UID: \"bd2a75ff-95c3-4b08-b75b-30059af6bd37\") " pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:33.730184 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:33.729723    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:16:33.730184 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:33.729781    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs podName:bd2a75ff-95c3-4b08-b75b-30059af6bd37 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:41.729762586 +0000 UTC m=+17.430301060 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs") pod "network-metrics-daemon-484l6" (UID: "bd2a75ff-95c3-4b08-b75b-30059af6bd37") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:16:33.830921 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:33.830837    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l8qd\" (UniqueName: \"kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd\") pod \"network-check-target-d252q\" (UID: \"37080dcf-15b0-4d4e-85fa-ff669783f356\") " pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:33.831081 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:33.831049    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:16:33.831081 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:33.831070    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:16:33.831081 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:33.831080    2578 projected.go:196] Error preparing data for projected volume kube-api-access-8l8qd for pod openshift-network-diagnostics/network-check-target-d252q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:16:33.831239 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:33.831126    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd podName:37080dcf-15b0-4d4e-85fa-ff669783f356 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:41.83111411 +0000 UTC m=+17.531652571 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8l8qd" (UniqueName: "kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd") pod "network-check-target-d252q" (UID: "37080dcf-15b0-4d4e-85fa-ff669783f356") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:16:33.929481 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:33.929446    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:33.929665 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:33.929600    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484l6" podUID="bd2a75ff-95c3-4b08-b75b-30059af6bd37"
Nov 03 17:16:33.929743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:33.929682    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:33.929796 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:33.929757    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt4lh" podUID="902e0389-e70e-4d23-a607-d8f4b9da868d"
Nov 03 17:16:33.929852 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:33.929812    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:33.929904 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:33.929880    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d252q" podUID="37080dcf-15b0-4d4e-85fa-ff669783f356"
Nov 03 17:16:34.012899 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:34.012147    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bkmvr" event={"ID":"5d9f8f17-bb8e-4f09-ade6-099a78bfe422","Type":"ContainerStarted","Data":"2be1da6b1c5a9d8f7ebc20feaa6b9ca90d78467c5a8b3734556abef90a45b4d8"}
Nov 03 17:16:34.016462 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:34.016046    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2" event={"ID":"c1e30c92-229f-4030-82eb-9295f5328da4","Type":"ContainerStarted","Data":"af18903939b3474e552e8069c40226523bcc6fd66055d38c96087b134184cf22"}
Nov 03 17:16:34.128457 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:34.128340    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dzlp4" podStartSLOduration=4.302132179 podStartE2EDuration="9.128322833s" totalImagesPullingTime="4.826190654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:33.255572085 +0000 UTC m=+8.956110568" watchObservedRunningTime="2026-11-03 17:16:34.128322833 +0000 UTC m=+9.828861316"
Nov 03 17:16:35.798456 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:35.798335    2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 17:16:35.854926 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:35.852306    2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T17:16:35.798383545Z","UUID":"ad87ff93-cd00-4d47-b6c8-36c75d6b7adf","Handler":null,"Name":"","Endpoint":""}
Nov 03 17:16:35.854926 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:35.854902    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 17:16:35.856077 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:35.854984    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 17:16:35.929488 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:35.929002    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:35.929488 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:35.929132    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d252q" podUID="37080dcf-15b0-4d4e-85fa-ff669783f356"
Nov 03 17:16:35.929488 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:35.929217    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:35.929488 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:35.929307    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484l6" podUID="bd2a75ff-95c3-4b08-b75b-30059af6bd37"
Nov 03 17:16:35.929488 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:35.929382    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:35.929488 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:35.929453    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt4lh" podUID="902e0389-e70e-4d23-a607-d8f4b9da868d"
Nov 03 17:16:36.023814 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:36.023740    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2" event={"ID":"c1e30c92-229f-4030-82eb-9295f5328da4","Type":"ContainerStarted","Data":"22dfe648ad3dc4412d4352b1f3bd794b1a9191bb8e88711e481cb27206a1681c"}
Nov 03 17:16:36.027392 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:36.027337    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9cgng" event={"ID":"00710f2d-31bb-42da-9ae8-135ddc117df5","Type":"ContainerStarted","Data":"61f010f6d7d6393b86fa6fa435a88b6bea2a26589bfa185e7ab8a1fdc9fac4aa"}
Nov 03 17:16:36.151413 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:36.151292    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bkmvr" podStartSLOduration=5.312655901 podStartE2EDuration="11.151274513s" totalImagesPullingTime="5.838618612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:34.129209308 +0000 UTC m=+9.829747863" watchObservedRunningTime="2026-11-03 17:16:36.151274513 +0000 UTC m=+11.851812995"
Nov 03 17:16:36.152024 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:36.151981    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-9cgng" podStartSLOduration=2.665456023 podStartE2EDuration="11.151970362s" totalImagesPullingTime="8.486514339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:36.150826623 +0000 UTC m=+11.851365129" watchObservedRunningTime="2026-11-03 17:16:36.151970362 +0000 UTC m=+11.852508846"
Nov 03 17:16:36.719411 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:36.719193    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:36.947395 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:36.947292    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:37.033814 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:37.033780    2578 generic.go:356] "Generic (PLEG): container finished" podID="23a7781b-32fa-4f63-ab0e-663e1f2c6abb" containerID="92c2310c2040a1bb7d8976202fb78354aac3e1f9084cf6645f3f61cc4f32e630" exitCode=0
Nov 03 17:16:37.033979 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:37.033867    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b99b8" event={"ID":"23a7781b-32fa-4f63-ab0e-663e1f2c6abb","Type":"ContainerDied","Data":"92c2310c2040a1bb7d8976202fb78354aac3e1f9084cf6645f3f61cc4f32e630"}
Nov 03 17:16:37.038276 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:37.038246    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6l5k8" event={"ID":"e59c4c3e-b582-4b16-a6e3-4aadf0612a32","Type":"ContainerStarted","Data":"78190c14409ae9476a497296e35a771b9d1742c8e2d521bd42e9314a2b6a97d0"}
Nov 03 17:16:37.125487 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:37.125459    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:37.178892 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:37.178860    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dzlp4"
Nov 03 17:16:37.179483 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:37.179464    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dzlp4"
Nov 03 17:16:37.395213 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:37.395156    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-6l5k8" podStartSLOduration=3.600076009 podStartE2EDuration="12.395137456s" totalImagesPullingTime="8.795061447s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:37.27587762 +0000 UTC m=+12.976416102" watchObservedRunningTime="2026-11-03 17:16:37.395137456 +0000 UTC m=+13.095675941"
Nov 03 17:16:37.928690 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:37.928615    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:37.928690 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:37.928645    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:37.928690 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:37.928668    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:37.928925 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:37.928783    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d252q" podUID="37080dcf-15b0-4d4e-85fa-ff669783f356"
Nov 03 17:16:37.928925 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:37.928906    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484l6" podUID="bd2a75ff-95c3-4b08-b75b-30059af6bd37"
Nov 03 17:16:37.928995 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:37.928965    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt4lh" podUID="902e0389-e70e-4d23-a607-d8f4b9da868d"
Nov 03 17:16:38.042479 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:38.042157    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z5v9n" event={"ID":"88482077-204d-4634-8045-c741d2c10d80","Type":"ContainerStarted","Data":"937c0e8f230f2bb22d883bc2f5ffacbeb70038a0fd1cf9dce431cd3c63fb5308"}
Nov 03 17:16:38.044974 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:38.044943    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2" event={"ID":"c1e30c92-229f-4030-82eb-9295f5328da4","Type":"ContainerStarted","Data":"932b90fc16694c03c2e074369046d64df8b88e947f18d90f46d8e87bb9788e87"}
Nov 03 17:16:38.189002 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:38.188955    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-z5v9n" podStartSLOduration=4.4181415059999996 podStartE2EDuration="13.188942486s" totalImagesPullingTime="8.77080098s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:38.188822016 +0000 UTC m=+13.889360497" watchObservedRunningTime="2026-11-03 17:16:38.188942486 +0000 UTC m=+13.889480967"
Nov 03 17:16:38.239512 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:38.239487    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:38.314170 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:38.314105    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2" podStartSLOduration=3.437957793 podStartE2EDuration="13.314090432s" totalImagesPullingTime="9.876132639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:16:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:38.31369206 +0000 UTC m=+14.014230543" watchObservedRunningTime="2026-11-03 17:16:38.314090432 +0000 UTC m=+14.014628914"
Nov 03 17:16:38.458729 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:38.458655    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:38.743456 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:38.743379    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:39.048560 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:39.048467    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4dtnl" event={"ID":"8a607eca-e6ad-4e5d-aa57-663768133ec0","Type":"ContainerStarted","Data":"68ccefab22f028f42b89c670659ced7c5db96cfd1d767b8845d7852f85aaa344"}
Nov 03 17:16:39.050849 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:39.050818    2578 generic.go:356] "Generic (PLEG): container finished" podID="23a7781b-32fa-4f63-ab0e-663e1f2c6abb" containerID="b25a3b9416b77a869f89c508ea92eaaeb69ded4c5ccb7626f7b8328781bcdee3" exitCode=0
Nov 03 17:16:39.050969 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:39.050913    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b99b8" event={"ID":"23a7781b-32fa-4f63-ab0e-663e1f2c6abb","Type":"ContainerDied","Data":"b25a3b9416b77a869f89c508ea92eaaeb69ded4c5ccb7626f7b8328781bcdee3"}
Nov 03 17:16:39.169899 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:39.169842    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4dtnl" podStartSLOduration=3.641914128 podStartE2EDuration="14.169828016s" totalImagesPullingTime="10.527913888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:39.169341013 +0000 UTC m=+14.869879498" watchObservedRunningTime="2026-11-03 17:16:39.169828016 +0000 UTC m=+14.870366501"
Nov 03 17:16:39.230265 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:39.230236    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:39.929372 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:39.929324    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:39.929535 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:39.929384    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:39.929535 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:39.929324    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:39.929535 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:39.929487    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484l6" podUID="bd2a75ff-95c3-4b08-b75b-30059af6bd37"
Nov 03 17:16:39.929665 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:39.929573    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt4lh" podUID="902e0389-e70e-4d23-a607-d8f4b9da868d"
Nov 03 17:16:39.929665 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:39.929652    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d252q" podUID="37080dcf-15b0-4d4e-85fa-ff669783f356"
Nov 03 17:16:40.383024 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:40.382771    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:40.777517 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:40.777394    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:40.998026 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:40.997995    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:41.057623 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:41.057549    2578 generic.go:356] "Generic (PLEG): container finished" podID="23a7781b-32fa-4f63-ab0e-663e1f2c6abb" containerID="d099c0eeeb96a2e03e5f70e64eef6ddbeab21dbf1d5c725b0aaa68fa7dafc753" exitCode=0
Nov 03 17:16:41.057623 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:41.057612    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b99b8" event={"ID":"23a7781b-32fa-4f63-ab0e-663e1f2c6abb","Type":"ContainerDied","Data":"d099c0eeeb96a2e03e5f70e64eef6ddbeab21dbf1d5c725b0aaa68fa7dafc753"}
Nov 03 17:16:41.275101 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:41.275069    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:41.694106 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:41.694059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:41.694601 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:41.694216    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:16:41.694601 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:41.694298    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret podName:902e0389-e70e-4d23-a607-d8f4b9da868d nodeName:}" failed. No retries permitted until 2026-11-03 17:16:57.694275919 +0000 UTC m=+33.394814386 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret") pod "global-pull-secret-syncer-rt4lh" (UID: "902e0389-e70e-4d23-a607-d8f4b9da868d") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:16:41.737655 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:41.737621    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:41.794621 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:41.794591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs\") pod \"network-metrics-daemon-484l6\" (UID: \"bd2a75ff-95c3-4b08-b75b-30059af6bd37\") " pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:41.794770 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:41.794739    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:16:41.794814 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:41.794808    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs podName:bd2a75ff-95c3-4b08-b75b-30059af6bd37 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:57.794786375 +0000 UTC m=+33.495324852 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs") pod "network-metrics-daemon-484l6" (UID: "bd2a75ff-95c3-4b08-b75b-30059af6bd37") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:16:41.895990 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:41.895951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l8qd\" (UniqueName: \"kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd\") pod \"network-check-target-d252q\" (UID: \"37080dcf-15b0-4d4e-85fa-ff669783f356\") " pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:41.896165 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:41.896133    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:16:41.896165 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:41.896161    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:16:41.896274 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:41.896174    2578 projected.go:196] Error preparing data for projected volume kube-api-access-8l8qd for pod openshift-network-diagnostics/network-check-target-d252q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:16:41.896274 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:41.896240    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd podName:37080dcf-15b0-4d4e-85fa-ff669783f356 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:57.896221525 +0000 UTC m=+33.596759999 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8l8qd" (UniqueName: "kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd") pod "network-check-target-d252q" (UID: "37080dcf-15b0-4d4e-85fa-ff669783f356") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:16:41.929029 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:41.928996    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:41.929029 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:41.929026    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:41.929186 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:41.929029    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:41.929186 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:41.929128    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484l6" podUID="bd2a75ff-95c3-4b08-b75b-30059af6bd37"
Nov 03 17:16:41.929262 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:41.929179    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt4lh" podUID="902e0389-e70e-4d23-a607-d8f4b9da868d"
Nov 03 17:16:41.929262 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:41.929245    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d252q" podUID="37080dcf-15b0-4d4e-85fa-ff669783f356"
Nov 03 17:16:41.938653 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:41.938626    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:42.295270 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:42.295243    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:42.507266 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:42.507081    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:42.853006 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:42.852980    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6l5k8_e59c4c3e-b582-4b16-a6e3-4aadf0612a32/dns-node-resolver/0.log"
Nov 03 17:16:43.064274 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:43.064249    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:16:43.064598 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:43.064578    2578 generic.go:356] "Generic (PLEG): container finished" podID="8906c21b-3233-4214-8ce1-f9189b79ccd3" containerID="a45894f72c3f44cabca1dd1e1c377e0d970282e716a3c5aea600e6d3d1cf6f39" exitCode=1
Nov 03 17:16:43.064658 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:43.064612    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n" event={"ID":"8906c21b-3233-4214-8ce1-f9189b79ccd3","Type":"ContainerStarted","Data":"d66507910d3df299957f39a58c513d9b829d6fa331b3d11e8289ae9ef39c51ae"}
Nov 03 17:16:43.064693 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:43.064668    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n" event={"ID":"8906c21b-3233-4214-8ce1-f9189b79ccd3","Type":"ContainerStarted","Data":"33d53935f512b62360e25500f3ef9d26348e6cbcab9496dc49a86174f3130038"}
Nov 03 17:16:43.064722 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:43.064698    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n" event={"ID":"8906c21b-3233-4214-8ce1-f9189b79ccd3","Type":"ContainerStarted","Data":"b6fdbd6bb795ff7b90e57dc21dc88ee9191138432851c63c80933d259c811254"}
Nov 03 17:16:43.064752 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:43.064727    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n" event={"ID":"8906c21b-3233-4214-8ce1-f9189b79ccd3","Type":"ContainerStarted","Data":"f24894f035149c575191fb445b9ab21a438c2541529110cb79fc391f0793180b"}
Nov 03 17:16:43.064783 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:43.064755    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n" event={"ID":"8906c21b-3233-4214-8ce1-f9189b79ccd3","Type":"ContainerDied","Data":"a45894f72c3f44cabca1dd1e1c377e0d970282e716a3c5aea600e6d3d1cf6f39"}
Nov 03 17:16:43.064816 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:43.064783    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n" event={"ID":"8906c21b-3233-4214-8ce1-f9189b79ccd3","Type":"ContainerStarted","Data":"5eb819442b1c9c0d68484867c16f3c3ef6976ca208e2f598421f4f44334e2351"}
Nov 03 17:16:43.098308 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:43.098281    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:43.319071 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:43.319001    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:43.696132 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:43.696053    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:43.837155 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:43.837129    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bkmvr_5d9f8f17-bb8e-4f09-ade6-099a78bfe422/node-ca/0.log"
Nov 03 17:16:43.895300 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:43.895270    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:43.928781 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:43.928756    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:43.928887 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:43.928857    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:43.928887 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:43.928866    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:43.928887 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:43.928866    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484l6" podUID="bd2a75ff-95c3-4b08-b75b-30059af6bd37"
Nov 03 17:16:43.928997 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:43.928944    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt4lh" podUID="902e0389-e70e-4d23-a607-d8f4b9da868d"
Nov 03 17:16:43.929030 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:43.929003    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d252q" podUID="37080dcf-15b0-4d4e-85fa-ff669783f356"
Nov 03 17:16:44.069855 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:44.069776    2578 generic.go:356] "Generic (PLEG): container finished" podID="23a7781b-32fa-4f63-ab0e-663e1f2c6abb" containerID="c693989e9116ca8b52467d296d65554356e96f00eac19da8271143d564307219" exitCode=0
Nov 03 17:16:44.069855 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:44.069840    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b99b8" event={"ID":"23a7781b-32fa-4f63-ab0e-663e1f2c6abb","Type":"ContainerDied","Data":"c693989e9116ca8b52467d296d65554356e96f00eac19da8271143d564307219"}
Nov 03 17:16:44.330549 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:44.330466    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:44.506225 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:44.506195    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:45.074665 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:45.074571    2578 generic.go:356] "Generic (PLEG): container finished" podID="23a7781b-32fa-4f63-ab0e-663e1f2c6abb" containerID="9873e8f5a2fb31a0f8b82fd8aff2bfdbe9197254c7ce312c55a37a2267fa5679" exitCode=0
Nov 03 17:16:45.074665 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:45.074609    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b99b8" event={"ID":"23a7781b-32fa-4f63-ab0e-663e1f2c6abb","Type":"ContainerDied","Data":"9873e8f5a2fb31a0f8b82fd8aff2bfdbe9197254c7ce312c55a37a2267fa5679"}
Nov 03 17:16:45.077698 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:45.077620    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:16:45.077995 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:45.077973    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n" event={"ID":"8906c21b-3233-4214-8ce1-f9189b79ccd3","Type":"ContainerStarted","Data":"2faa5ffded3d4e95443a25022f8ab7d73359870fd959089af75fe230c1babebe"}
Nov 03 17:16:45.091088 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:45.091065    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:45.281733 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:45.281704    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:45.688204 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:45.688175    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:45.898328 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:45.898289    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:45.929056 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:45.929035    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:45.929188 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:45.929059    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:45.929188 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:45.929121    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484l6" podUID="bd2a75ff-95c3-4b08-b75b-30059af6bd37"
Nov 03 17:16:45.929188 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:45.929161    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:45.929296 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:45.929245    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d252q" podUID="37080dcf-15b0-4d4e-85fa-ff669783f356"
Nov 03 17:16:45.929344 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:45.929302    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt4lh" podUID="902e0389-e70e-4d23-a607-d8f4b9da868d"
Nov 03 17:16:46.083690 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:46.083620    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b99b8" event={"ID":"23a7781b-32fa-4f63-ab0e-663e1f2c6abb","Type":"ContainerStarted","Data":"44d9931f010198208a0f17035fef63770f831fa8805499ed72ca4662b0101a3c"}
Nov 03 17:16:46.210695 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:46.210649    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b99b8" podStartSLOduration=8.799294444000001 podStartE2EDuration="21.210636555s" totalImagesPullingTime="12.411342111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:16:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:46.209966393 +0000 UTC m=+21.910504875" watchObservedRunningTime="2026-11-03 17:16:46.210636555 +0000 UTC m=+21.911175036"
Nov 03 17:16:46.301900 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:46.301874    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:46.487494 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:46.487418    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:46.673458 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:46.673435    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:47.087485 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:47.087450    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:47.089220 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:47.089201    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:16:47.089605 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:47.089580    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n" event={"ID":"8906c21b-3233-4214-8ce1-f9189b79ccd3","Type":"ContainerStarted","Data":"db7aef04d79cd331093ff6bcd733f80d020a211d73f9424879f2edfcdd1b6f17"}
Nov 03 17:16:47.090199 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:47.090184    2578 scope.go:122] "RemoveContainer" containerID="a45894f72c3f44cabca1dd1e1c377e0d970282e716a3c5aea600e6d3d1cf6f39"
Nov 03 17:16:47.734166 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:47.734136    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:47.917630 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:47.917599    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:47.928661 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:47.928637    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:47.928783 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:47.928664    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:47.928835 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:47.928777    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484l6" podUID="bd2a75ff-95c3-4b08-b75b-30059af6bd37"
Nov 03 17:16:47.928835 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:47.928812    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d252q" podUID="37080dcf-15b0-4d4e-85fa-ff669783f356"
Nov 03 17:16:47.928932 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:47.928849    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:47.928932 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:47.928916    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt4lh" podUID="902e0389-e70e-4d23-a607-d8f4b9da868d"
Nov 03 17:16:48.095136 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:48.095058    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:16:48.095570 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:48.095470    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n" event={"ID":"8906c21b-3233-4214-8ce1-f9189b79ccd3","Type":"ContainerStarted","Data":"c4586fcdba2cb99de8f9e88e40b38bc232bdc7da767205a5c6a92b2da624a707"}
Nov 03 17:16:48.095824 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:48.095799    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:48.095953 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:48.095870    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:48.095953 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:48.095919    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:48.110512 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:48.110490    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:48.110631 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:48.110585    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:16:48.276677 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:48.276599    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n" podStartSLOduration=8.462154427 podStartE2EDuration="23.276583722s" totalImagesPullingTime="14.814429295s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:48.276447831 +0000 UTC m=+23.976986314" watchObservedRunningTime="2026-11-03 17:16:48.276583722 +0000 UTC m=+23.977122205"
Nov 03 17:16:48.289047 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:48.289018    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:48.484330 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:48.484284    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:48.547088 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:48.547054    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dzlp4"
Nov 03 17:16:48.547258 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:48.547233    2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 17:16:48.547643 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:48.547626    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dzlp4"
Nov 03 17:16:48.668257 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:48.668227    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:49.090975 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:49.090944    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:49.690295 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:49.690262    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:49.876997 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:49.876966    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:49.929048 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:49.929014    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:49.929204 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:49.929096    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:49.929204 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:49.929119    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:49.929330 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:49.929219    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d252q" podUID="37080dcf-15b0-4d4e-85fa-ff669783f356"
Nov 03 17:16:49.929330 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:49.929273    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt4lh" podUID="902e0389-e70e-4d23-a607-d8f4b9da868d"
Nov 03 17:16:49.929453 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:49.929352    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484l6" podUID="bd2a75ff-95c3-4b08-b75b-30059af6bd37"
Nov 03 17:16:50.065450 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:50.065412    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:50.299071 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:50.299038    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:50.519758 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:50.519730    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:50.710908 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:50.710879    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:50.918654 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:50.918573    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:51.118520 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:51.118483    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:51.329593 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:51.329561    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:51.929236 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:51.929200    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:51.929736 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:51.929243    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:51.929736 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:51.929328    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:51.929736 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:51.929331    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt4lh" podUID="902e0389-e70e-4d23-a607-d8f4b9da868d"
Nov 03 17:16:51.929736 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:51.929422    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d252q" podUID="37080dcf-15b0-4d4e-85fa-ff669783f356"
Nov 03 17:16:51.929736 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:51.929505    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484l6" podUID="bd2a75ff-95c3-4b08-b75b-30059af6bd37"
Nov 03 17:16:53.184699 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:53.184668    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:53.460451 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:53.460425    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:53.636509 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:53.636475    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:53.814984 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:53.814913    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:53.929327 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:53.929295    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:53.929491 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:53.929335    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:53.929491 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:53.929335    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:53.929491 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:53.929416    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484l6" podUID="bd2a75ff-95c3-4b08-b75b-30059af6bd37"
Nov 03 17:16:53.929593 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:53.929506    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt4lh" podUID="902e0389-e70e-4d23-a607-d8f4b9da868d"
Nov 03 17:16:53.929593 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:53.929566    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d252q" podUID="37080dcf-15b0-4d4e-85fa-ff669783f356"
Nov 03 17:16:54.036954 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:54.036917    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:54.280886 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:54.280855    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:54.469412 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:54.469386    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:54.662341 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:54.662262    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:55.906502 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:55.906471    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:55.929588 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:55.929563    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:55.929718 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:55.929622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:55.929718 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:55.929642    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:55.929820 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:55.929750    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d252q" podUID="37080dcf-15b0-4d4e-85fa-ff669783f356"
Nov 03 17:16:55.929820 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:55.929786    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt4lh" podUID="902e0389-e70e-4d23-a607-d8f4b9da868d"
Nov 03 17:16:55.929924 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:55.929853    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484l6" podUID="bd2a75ff-95c3-4b08-b75b-30059af6bd37"
Nov 03 17:16:57.552408 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:57.552374    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:16:57.715710 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:57.715672    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:57.715874 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:57.715779    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:16:57.715874 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:57.715827    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret podName:902e0389-e70e-4d23-a607-d8f4b9da868d nodeName:}" failed. No retries permitted until 2026-11-03 17:17:29.715814288 +0000 UTC m=+65.416352748 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret") pod "global-pull-secret-syncer-rt4lh" (UID: "902e0389-e70e-4d23-a607-d8f4b9da868d") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:16:57.816606 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:57.816529    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs\") pod \"network-metrics-daemon-484l6\" (UID: \"bd2a75ff-95c3-4b08-b75b-30059af6bd37\") " pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:57.816737 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:57.816649    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:16:57.816737 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:57.816698    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs podName:bd2a75ff-95c3-4b08-b75b-30059af6bd37 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:29.816686005 +0000 UTC m=+65.517224465 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs") pod "network-metrics-daemon-484l6" (UID: "bd2a75ff-95c3-4b08-b75b-30059af6bd37") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:16:57.917505 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:57.917467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l8qd\" (UniqueName: \"kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd\") pod \"network-check-target-d252q\" (UID: \"37080dcf-15b0-4d4e-85fa-ff669783f356\") " pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:57.917673 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:57.917612    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:16:57.917673 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:57.917642    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:16:57.917673 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:57.917654    2578 projected.go:196] Error preparing data for projected volume kube-api-access-8l8qd for pod openshift-network-diagnostics/network-check-target-d252q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:16:57.917787 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:57.917703    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd podName:37080dcf-15b0-4d4e-85fa-ff669783f356 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:29.917689536 +0000 UTC m=+65.618227997 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8l8qd" (UniqueName: "kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd") pod "network-check-target-d252q" (UID: "37080dcf-15b0-4d4e-85fa-ff669783f356") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:16:57.929514 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:57.929493    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:57.929589 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:57.929529    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:57.929589 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:57.929573    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d252q" podUID="37080dcf-15b0-4d4e-85fa-ff669783f356"
Nov 03 17:16:57.929657 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:57.929634    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484l6" podUID="bd2a75ff-95c3-4b08-b75b-30059af6bd37"
Nov 03 17:16:57.929689 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:57.929674    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:57.929736 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:57.929723    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt4lh" podUID="902e0389-e70e-4d23-a607-d8f4b9da868d"
Nov 03 17:16:59.928654 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:59.928604    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:16:59.928654 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:59.928651    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:16:59.929176 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:16:59.928626    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:16:59.929176 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:59.928733    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d252q" podUID="37080dcf-15b0-4d4e-85fa-ff669783f356"
Nov 03 17:16:59.929176 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:59.928798    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484l6" podUID="bd2a75ff-95c3-4b08-b75b-30059af6bd37"
Nov 03 17:16:59.929176 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:16:59.928870    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt4lh" podUID="902e0389-e70e-4d23-a607-d8f4b9da868d"
Nov 03 17:17:00.285000 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:00.284975    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:00.528933 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:00.528898    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:00.816845 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:00.816813    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:01.005542 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:01.005510    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:01.929406 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:01.929352    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:17:01.929576 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:01.929351    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:17:01.929576 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:17:01.929489    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d252q" podUID="37080dcf-15b0-4d4e-85fa-ff669783f356"
Nov 03 17:17:01.929576 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:17:01.929544    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt4lh" podUID="902e0389-e70e-4d23-a607-d8f4b9da868d"
Nov 03 17:17:01.929576 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:01.929352    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:17:01.929698 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:17:01.929633    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484l6" podUID="bd2a75ff-95c3-4b08-b75b-30059af6bd37"
Nov 03 17:17:02.428075 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:02.427840    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-484l6"]
Nov 03 17:17:02.428459 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:02.428212    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:17:02.428459 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:02.428329    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rt4lh"]
Nov 03 17:17:02.428459 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:17:02.428330    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484l6" podUID="bd2a75ff-95c3-4b08-b75b-30059af6bd37"
Nov 03 17:17:02.428459 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:02.428436    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:17:02.428654 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:17:02.428539    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt4lh" podUID="902e0389-e70e-4d23-a607-d8f4b9da868d"
Nov 03 17:17:02.428972 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:02.428954    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d252q"]
Nov 03 17:17:02.429030 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:02.429015    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:17:02.429108 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:17:02.429089    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d252q" podUID="37080dcf-15b0-4d4e-85fa-ff669783f356"
Nov 03 17:17:03.928749 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:03.928665    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:17:03.928749 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:03.928685    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:17:03.929158 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:03.928764    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:17:03.929158 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:17:03.928774    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d252q" podUID="37080dcf-15b0-4d4e-85fa-ff669783f356"
Nov 03 17:17:03.929158 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:17:03.928826    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt4lh" podUID="902e0389-e70e-4d23-a607-d8f4b9da868d"
Nov 03 17:17:03.929158 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:17:03.928907    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484l6" podUID="bd2a75ff-95c3-4b08-b75b-30059af6bd37"
Nov 03 17:17:04.571990 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:04.571957    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:04.757510 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:04.757481    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:04.933738 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:04.933666    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:05.108735 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:05.108705    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:05.928764 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:05.928735    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:17:05.928997 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:05.928731    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:17:05.928997 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:17:05.928827    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d252q" podUID="37080dcf-15b0-4d4e-85fa-ff669783f356"
Nov 03 17:17:05.928997 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:17:05.928929    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484l6" podUID="bd2a75ff-95c3-4b08-b75b-30059af6bd37"
Nov 03 17:17:05.928997 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:05.928749    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:17:05.929123 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:17:05.929011    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rt4lh" podUID="902e0389-e70e-4d23-a607-d8f4b9da868d"
Nov 03 17:17:07.622521 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.622489    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-117.us-west-2.compute.internal" event="NodeReady"
Nov 03 17:17:07.623090 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.622650    2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 17:17:07.779194 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.779099    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vpzpk"]
Nov 03 17:17:07.781052 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.781030    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dv7x7"]
Nov 03 17:17:07.781187 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.781166    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:07.782680 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.782665    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dv7x7"
Nov 03 17:17:07.811011 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.810982    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vpzpk"]
Nov 03 17:17:07.827714 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.827685    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5gv26"]
Nov 03 17:17:07.829704 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.829681    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:07.836050 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.836029    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 17:17:07.836322 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.836305    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 17:17:07.836421 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.836405    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qk7gm\""
Nov 03 17:17:07.837511 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.837496    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 17:17:07.837572 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.837545    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 17:17:07.837639 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.837621    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gnkq2\""
Nov 03 17:17:07.876540 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.876510    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dv7x7"]
Nov 03 17:17:07.888027 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.887996    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5gv26"]
Nov 03 17:17:07.891434 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.891407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5c5030e-696b-41dd-a96b-11c9eed11db2-metrics-tls\") pod \"dns-default-vpzpk\" (UID: \"e5c5030e-696b-41dd-a96b-11c9eed11db2\") " pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:07.891569 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.891442    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c5030e-696b-41dd-a96b-11c9eed11db2-tmp-dir\") pod \"dns-default-vpzpk\" (UID: \"e5c5030e-696b-41dd-a96b-11c9eed11db2\") " pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:07.891569 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.891517    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn94x\" (UniqueName: \"kubernetes.io/projected/9f6cf431-dc3b-478e-81cd-97127a917de3-kube-api-access-qn94x\") pod \"ingress-canary-dv7x7\" (UID: \"9f6cf431-dc3b-478e-81cd-97127a917de3\") " pod="openshift-ingress-canary/ingress-canary-dv7x7"
Nov 03 17:17:07.891569 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.891548    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7jmf\" (UniqueName: \"kubernetes.io/projected/e5c5030e-696b-41dd-a96b-11c9eed11db2-kube-api-access-n7jmf\") pod \"dns-default-vpzpk\" (UID: \"e5c5030e-696b-41dd-a96b-11c9eed11db2\") " pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:07.891706 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.891580    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c5030e-696b-41dd-a96b-11c9eed11db2-config-volume\") pod \"dns-default-vpzpk\" (UID: \"e5c5030e-696b-41dd-a96b-11c9eed11db2\") " pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:07.891706 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.891673    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f6cf431-dc3b-478e-81cd-97127a917de3-cert\") pod \"ingress-canary-dv7x7\" (UID: \"9f6cf431-dc3b-478e-81cd-97127a917de3\") " pod="openshift-ingress-canary/ingress-canary-dv7x7"
Nov 03 17:17:07.929434 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.929398    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:17:07.929599 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.929462    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:17:07.929599 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.929485    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:17:07.936504 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.936479    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 17:17:07.936689 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.936533    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jrssc\""
Nov 03 17:17:07.936689 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.936622    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 17:17:07.936689 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.936683    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 17:17:07.936868 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.936693    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 17:17:07.936922 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.936889    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 17:17:07.992520 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.992489    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5c5030e-696b-41dd-a96b-11c9eed11db2-metrics-tls\") pod \"dns-default-vpzpk\" (UID: \"e5c5030e-696b-41dd-a96b-11c9eed11db2\") " pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:07.992520 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.992521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c5030e-696b-41dd-a96b-11c9eed11db2-tmp-dir\") pod \"dns-default-vpzpk\" (UID: \"e5c5030e-696b-41dd-a96b-11c9eed11db2\") " pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:07.992784 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.992544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-crio-socket\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:07.992784 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.992562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn94x\" (UniqueName: \"kubernetes.io/projected/9f6cf431-dc3b-478e-81cd-97127a917de3-kube-api-access-qn94x\") pod \"ingress-canary-dv7x7\" (UID: \"9f6cf431-dc3b-478e-81cd-97127a917de3\") " pod="openshift-ingress-canary/ingress-canary-dv7x7"
Nov 03 17:17:07.992784 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.992585    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7jmf\" (UniqueName: \"kubernetes.io/projected/e5c5030e-696b-41dd-a96b-11c9eed11db2-kube-api-access-n7jmf\") pod \"dns-default-vpzpk\" (UID: \"e5c5030e-696b-41dd-a96b-11c9eed11db2\") " pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:07.992784 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.992700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-data-volume\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:07.992972 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.992810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c5030e-696b-41dd-a96b-11c9eed11db2-config-volume\") pod \"dns-default-vpzpk\" (UID: \"e5c5030e-696b-41dd-a96b-11c9eed11db2\") " pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:07.992972 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.992919    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c5030e-696b-41dd-a96b-11c9eed11db2-tmp-dir\") pod \"dns-default-vpzpk\" (UID: \"e5c5030e-696b-41dd-a96b-11c9eed11db2\") " pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:07.992972 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.992951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f6cf431-dc3b-478e-81cd-97127a917de3-cert\") pod \"ingress-canary-dv7x7\" (UID: \"9f6cf431-dc3b-478e-81cd-97127a917de3\") " pod="openshift-ingress-canary/ingress-canary-dv7x7"
Nov 03 17:17:07.993074 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.992994    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:07.993074 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.993028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:07.993172 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.993113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlwvb\" (UniqueName: \"kubernetes.io/projected/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-kube-api-access-wlwvb\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:07.993422 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.993403    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c5030e-696b-41dd-a96b-11c9eed11db2-config-volume\") pod \"dns-default-vpzpk\" (UID: \"e5c5030e-696b-41dd-a96b-11c9eed11db2\") " pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:07.996683 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.996653    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5c5030e-696b-41dd-a96b-11c9eed11db2-metrics-tls\") pod \"dns-default-vpzpk\" (UID: \"e5c5030e-696b-41dd-a96b-11c9eed11db2\") " pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:07.996780 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:07.996703    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f6cf431-dc3b-478e-81cd-97127a917de3-cert\") pod \"ingress-canary-dv7x7\" (UID: \"9f6cf431-dc3b-478e-81cd-97127a917de3\") " pod="openshift-ingress-canary/ingress-canary-dv7x7"
Nov 03 17:17:08.036612 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.036527    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 17:17:08.036808 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.036782    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mr6xr\""
Nov 03 17:17:08.036808 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.036792    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 17:17:08.036979 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.036819    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-d8tp8\""
Nov 03 17:17:08.036979 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.036847    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 17:17:08.036979 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.036865    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x9v6p\""
Nov 03 17:17:08.036979 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.036938    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 17:17:08.054307 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.054283    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7jmf\" (UniqueName: \"kubernetes.io/projected/e5c5030e-696b-41dd-a96b-11c9eed11db2-kube-api-access-n7jmf\") pod \"dns-default-vpzpk\" (UID: \"e5c5030e-696b-41dd-a96b-11c9eed11db2\") " pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:08.054553 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.054533    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn94x\" (UniqueName: \"kubernetes.io/projected/9f6cf431-dc3b-478e-81cd-97127a917de3-kube-api-access-qn94x\") pod \"ingress-canary-dv7x7\" (UID: \"9f6cf431-dc3b-478e-81cd-97127a917de3\") " pod="openshift-ingress-canary/ingress-canary-dv7x7"
Nov 03 17:17:08.093718 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.093683    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:08.093862 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.093820    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-crio-socket\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:08.093862 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.093851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-data-volume\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:08.093931 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.093920    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:08.093968 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.093939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:08.093968 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.093961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlwvb\" (UniqueName: \"kubernetes.io/projected/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-kube-api-access-wlwvb\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:08.094072 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.094049    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-crio-socket\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:08.094369 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.094340    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-data-volume\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:08.094624 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.094591    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:08.096449 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.096426    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:08.097906 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.097877    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dv7x7"
Nov 03 17:17:08.160712 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.157770    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlwvb\" (UniqueName: \"kubernetes.io/projected/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-kube-api-access-wlwvb\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:08.441321 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.441071    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:08.489451 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.489421    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dv7x7"]
Nov 03 17:17:08.536575 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.536551    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vpzpk"]
Nov 03 17:17:08.540257 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:17:08.540236    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c5030e_696b_41dd_a96b_11c9eed11db2.slice/crio-388325401b9403f65e71e7f80c658c2e479db62a04148a3bda6cf565cc0dee0a WatchSource:0}: Error finding container 388325401b9403f65e71e7f80c658c2e479db62a04148a3bda6cf565cc0dee0a: Status 404 returned error can't find the container with id 388325401b9403f65e71e7f80c658c2e479db62a04148a3bda6cf565cc0dee0a
Nov 03 17:17:08.813189 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:08.813159    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5gv26"]
Nov 03 17:17:08.816206 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:17:08.816179    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b4340c_7e77_4d6e_bdc3_2a6193620f58.slice/crio-a8474860be3e4da5daa7aa3ce533a8b0d1c67601203c301400fa1bcb32fdb559 WatchSource:0}: Error finding container a8474860be3e4da5daa7aa3ce533a8b0d1c67601203c301400fa1bcb32fdb559: Status 404 returned error can't find the container with id a8474860be3e4da5daa7aa3ce533a8b0d1c67601203c301400fa1bcb32fdb559
Nov 03 17:17:09.141707 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:09.141612    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5gv26" event={"ID":"b5b4340c-7e77-4d6e-bdc3-2a6193620f58","Type":"ContainerStarted","Data":"6e5b824fe2e484efe370729a337df1a3f12f37141aedaed6ef5e130c4f81452e"}
Nov 03 17:17:09.141707 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:09.141673    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5gv26" event={"ID":"b5b4340c-7e77-4d6e-bdc3-2a6193620f58","Type":"ContainerStarted","Data":"a8474860be3e4da5daa7aa3ce533a8b0d1c67601203c301400fa1bcb32fdb559"}
Nov 03 17:17:09.142819 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:09.142796    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dv7x7" event={"ID":"9f6cf431-dc3b-478e-81cd-97127a917de3","Type":"ContainerStarted","Data":"674240409aea1984b262957e2a7b4b0ee6e9691b2ae3eca66fa5c6c3f997ac7f"}
Nov 03 17:17:09.143907 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:09.143887    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vpzpk" event={"ID":"e5c5030e-696b-41dd-a96b-11c9eed11db2","Type":"ContainerStarted","Data":"388325401b9403f65e71e7f80c658c2e479db62a04148a3bda6cf565cc0dee0a"}
Nov 03 17:17:09.868864 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:09.868826    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:10.051275 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:10.051245    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:10.230458 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:10.230423    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:10.408445 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:10.408398    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:10.590751 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:10.590650    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:11.141075 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:11.140998    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:11.152235 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:11.152205    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vpzpk" event={"ID":"e5c5030e-696b-41dd-a96b-11c9eed11db2","Type":"ContainerStarted","Data":"bf6ed23317f754c3ce3df3943516015448cd8d606f33bb90e93e6eeda2715fdd"}
Nov 03 17:17:11.152395 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:11.152266    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vpzpk" event={"ID":"e5c5030e-696b-41dd-a96b-11c9eed11db2","Type":"ContainerStarted","Data":"9389e21bdb830338195b2393c52560e9fffd86f1e32012e1e1989e8275221953"}
Nov 03 17:17:11.152395 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:11.152309    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:11.154443 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:11.154415    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5gv26" event={"ID":"b5b4340c-7e77-4d6e-bdc3-2a6193620f58","Type":"ContainerStarted","Data":"938b8280f38e69037bed831a4dba8d1e6b9fa6ba7ac7c398c8f2e73d14576a31"}
Nov 03 17:17:11.343938 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:11.343871    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-vpzpk" podStartSLOduration=2.01811392 podStartE2EDuration="4.3438513s" totalImagesPullingTime="2.32573738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:11.304187494 +0000 UTC m=+47.004725978" watchObservedRunningTime="2026-11-03 17:17:11.3438513 +0000 UTC m=+47.044389780"
Nov 03 17:17:11.458094 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:11.458059    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:11.642385 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:11.642314    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:11.976876 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:11.976839    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:12.159034 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.158550    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dv7x7" event={"ID":"9f6cf431-dc3b-478e-81cd-97127a917de3","Type":"ContainerStarted","Data":"6219dea657144e59e31bb262b10fb97a15e82747fa4865bd5edd05d563a13fb6"}
Nov 03 17:17:12.192195 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.192141    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:12.290504 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.290455    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dv7x7" podStartSLOduration=1.9961771750000001 podStartE2EDuration="5.290440871s" totalImagesPullingTime="3.294263696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:12.2898804 +0000 UTC m=+47.990418883" watchObservedRunningTime="2026-11-03 17:17:12.290440871 +0000 UTC m=+47.990979402"
Nov 03 17:17:12.476742 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.476706    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:12.660769 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.660729    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:12.663163 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.663142    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"]
Nov 03 17:17:12.666219 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.666204    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"
Nov 03 17:17:12.724974 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.724935    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 17:17:12.727126 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.727062    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 17:17:12.727964 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.727941    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 17:17:12.728188 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.728169    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 17:17:12.731621 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.731598    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/117a0c38-48f5-430d-9efe-1a308f725309-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5srqg\" (UID: \"117a0c38-48f5-430d-9efe-1a308f725309\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"
Nov 03 17:17:12.731708 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.731657    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/117a0c38-48f5-430d-9efe-1a308f725309-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5srqg\" (UID: \"117a0c38-48f5-430d-9efe-1a308f725309\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"
Nov 03 17:17:12.731744 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.731733    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/117a0c38-48f5-430d-9efe-1a308f725309-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5srqg\" (UID: \"117a0c38-48f5-430d-9efe-1a308f725309\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"
Nov 03 17:17:12.731781 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.731762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l26gw\" (UniqueName: \"kubernetes.io/projected/117a0c38-48f5-430d-9efe-1a308f725309-kube-api-access-l26gw\") pod \"openshift-state-metrics-7bd7489957-5srqg\" (UID: \"117a0c38-48f5-430d-9efe-1a308f725309\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"
Nov 03 17:17:12.774812 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.774770    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xr7h8\""
Nov 03 17:17:12.774812 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.774770    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 17:17:12.775494 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.775473    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"]
Nov 03 17:17:12.776081 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.776064    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"]
Nov 03 17:17:12.779200 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.779183    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9gj97"]
Nov 03 17:17:12.779318 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.779302    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:12.782247 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.782221    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.832926 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.832898    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd5nd\" (UniqueName: \"kubernetes.io/projected/4af20ec7-7ea1-470f-b238-62e0de9ca527-kube-api-access-jd5nd\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.832926 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.832930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08770b22-f9ff-4919-af54-c7916eba9101-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:12.833162 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.832956    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46p7r\" (UniqueName: \"kubernetes.io/projected/08770b22-f9ff-4919-af54-c7916eba9101-kube-api-access-46p7r\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:12.833162 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.833025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/117a0c38-48f5-430d-9efe-1a308f725309-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5srqg\" (UID: \"117a0c38-48f5-430d-9efe-1a308f725309\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"
Nov 03 17:17:12.833162 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.833044    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/08770b22-f9ff-4919-af54-c7916eba9101-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:12.833162 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.833067    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4af20ec7-7ea1-470f-b238-62e0de9ca527-root\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.833162 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.833083    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4af20ec7-7ea1-470f-b238-62e0de9ca527-sys\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.833162 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.833150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l26gw\" (UniqueName: \"kubernetes.io/projected/117a0c38-48f5-430d-9efe-1a308f725309-kube-api-access-l26gw\") pod \"openshift-state-metrics-7bd7489957-5srqg\" (UID: \"117a0c38-48f5-430d-9efe-1a308f725309\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"
Nov 03 17:17:12.833503 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.833439    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/117a0c38-48f5-430d-9efe-1a308f725309-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5srqg\" (UID: \"117a0c38-48f5-430d-9efe-1a308f725309\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"
Nov 03 17:17:12.833503 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.833484    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-tls\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.835001 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.833751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/08770b22-f9ff-4919-af54-c7916eba9101-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:12.835001 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.833792    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4af20ec7-7ea1-470f-b238-62e0de9ca527-metrics-client-ca\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.835001 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.833821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08770b22-f9ff-4919-af54-c7916eba9101-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:12.835001 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.833902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/117a0c38-48f5-430d-9efe-1a308f725309-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5srqg\" (UID: \"117a0c38-48f5-430d-9efe-1a308f725309\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"
Nov 03 17:17:12.835001 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.833981    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-textfile\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.835001 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.834164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/08770b22-f9ff-4919-af54-c7916eba9101-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:12.835001 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.834217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-wtmp\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.835001 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.834269    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.835001 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.834373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-accelerators-collector-config\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.835001 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.834615    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/117a0c38-48f5-430d-9efe-1a308f725309-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5srqg\" (UID: \"117a0c38-48f5-430d-9efe-1a308f725309\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"
Nov 03 17:17:12.835374 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.835142    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"]
Nov 03 17:17:12.837289 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.837262    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/117a0c38-48f5-430d-9efe-1a308f725309-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5srqg\" (UID: \"117a0c38-48f5-430d-9efe-1a308f725309\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"
Nov 03 17:17:12.837521 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.837500    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/117a0c38-48f5-430d-9efe-1a308f725309-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5srqg\" (UID: \"117a0c38-48f5-430d-9efe-1a308f725309\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"
Nov 03 17:17:12.849923 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.849899    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:12.876384 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.876349    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 17:17:12.876384 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.876349    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 17:17:12.895458 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.895430    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l26gw\" (UniqueName: \"kubernetes.io/projected/117a0c38-48f5-430d-9efe-1a308f725309-kube-api-access-l26gw\") pod \"openshift-state-metrics-7bd7489957-5srqg\" (UID: \"117a0c38-48f5-430d-9efe-1a308f725309\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"
Nov 03 17:17:12.924643 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.924615    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 17:17:12.924771 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.924746    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nxfj8\""
Nov 03 17:17:12.924962 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.924935    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 17:17:12.925069 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.924986    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 17:17:12.925069 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.925003    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 17:17:12.925069 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.925046    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4vh8j\""
Nov 03 17:17:12.935323 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.935300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/08770b22-f9ff-4919-af54-c7916eba9101-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:12.935441 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.935336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-wtmp\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.935441 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.935371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.935441 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.935408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-accelerators-collector-config\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.935441 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.935427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd5nd\" (UniqueName: \"kubernetes.io/projected/4af20ec7-7ea1-470f-b238-62e0de9ca527-kube-api-access-jd5nd\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.935660 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.935447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08770b22-f9ff-4919-af54-c7916eba9101-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:12.935660 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.935475    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46p7r\" (UniqueName: \"kubernetes.io/projected/08770b22-f9ff-4919-af54-c7916eba9101-kube-api-access-46p7r\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:12.935660 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.935502    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-wtmp\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.935660 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.935504    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/08770b22-f9ff-4919-af54-c7916eba9101-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:12.935660 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.935567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4af20ec7-7ea1-470f-b238-62e0de9ca527-root\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.935660 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.935606    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4af20ec7-7ea1-470f-b238-62e0de9ca527-sys\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.935660 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.935640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-tls\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.935980 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.935681    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/08770b22-f9ff-4919-af54-c7916eba9101-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:12.935980 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.935718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4af20ec7-7ea1-470f-b238-62e0de9ca527-metrics-client-ca\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.935980 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.935725    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4af20ec7-7ea1-470f-b238-62e0de9ca527-sys\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.935980 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.935745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08770b22-f9ff-4919-af54-c7916eba9101-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:12.935980 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.935780    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-textfile\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.935980 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.935779    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4af20ec7-7ea1-470f-b238-62e0de9ca527-root\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.936245 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.936072    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/08770b22-f9ff-4919-af54-c7916eba9101-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:12.936245 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.936102    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-accelerators-collector-config\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.936245 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.936242    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4af20ec7-7ea1-470f-b238-62e0de9ca527-metrics-client-ca\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.936406 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.936350    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/08770b22-f9ff-4919-af54-c7916eba9101-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:12.936522 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.936501    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08770b22-f9ff-4919-af54-c7916eba9101-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:12.936577 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.936562    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-textfile\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.938039 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.938015    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/08770b22-f9ff-4919-af54-c7916eba9101-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:12.938147 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.938124    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08770b22-f9ff-4919-af54-c7916eba9101-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:12.938233 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.938196    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-tls\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.938390 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.938351    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.980907 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.980832    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"
Nov 03 17:17:12.996807 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.996779    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd5nd\" (UniqueName: \"kubernetes.io/projected/4af20ec7-7ea1-470f-b238-62e0de9ca527-kube-api-access-jd5nd\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:12.997000 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:12.996983    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46p7r\" (UniqueName: \"kubernetes.io/projected/08770b22-f9ff-4919-af54-c7916eba9101-kube-api-access-46p7r\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:13.034441 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:13.034415    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:13.096026 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:13.095989    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:13.102196 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:13.102171    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:13.112714 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:17:13.112685    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af20ec7_7ea1_470f_b238_62e0de9ca527.slice/crio-7f9f87517e2657f52914494ba534311726391a06b3ac603e16c528bf27104a3d WatchSource:0}: Error finding container 7f9f87517e2657f52914494ba534311726391a06b3ac603e16c528bf27104a3d: Status 404 returned error can't find the container with id 7f9f87517e2657f52914494ba534311726391a06b3ac603e16c528bf27104a3d
Nov 03 17:17:13.162881 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:13.162840    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5gv26" event={"ID":"b5b4340c-7e77-4d6e-bdc3-2a6193620f58","Type":"ContainerStarted","Data":"ae149d65dd24c4e3b89ef7db9b7ebda73ae5f77cc7cbe961bb7be3580d338c75"}
Nov 03 17:17:13.164925 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:13.164893    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9gj97" event={"ID":"4af20ec7-7ea1-470f-b238-62e0de9ca527","Type":"ContainerStarted","Data":"7f9f87517e2657f52914494ba534311726391a06b3ac603e16c528bf27104a3d"}
Nov 03 17:17:13.273473 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:13.273396    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:13.287371 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:13.287297    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5gv26" podStartSLOduration=2.922228538 podStartE2EDuration="6.287274515s" totalImagesPullingTime="3.365045977s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:17:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:13.284445928 +0000 UTC m=+48.984984410" watchObservedRunningTime="2026-11-03 17:17:13.287274515 +0000 UTC m=+48.987812998"
Nov 03 17:17:13.402320 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:13.402285    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"]
Nov 03 17:17:13.405469 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:17:13.405435    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117a0c38_48f5_430d_9efe_1a308f725309.slice/crio-0fb2a4c0997232dfde0191c1496a54041963ac981c8993a7d7b4c39f4904085f WatchSource:0}: Error finding container 0fb2a4c0997232dfde0191c1496a54041963ac981c8993a7d7b4c39f4904085f: Status 404 returned error can't find the container with id 0fb2a4c0997232dfde0191c1496a54041963ac981c8993a7d7b4c39f4904085f
Nov 03 17:17:13.447843 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:13.447808    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:13.469968 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:13.469937    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"]
Nov 03 17:17:13.473326 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:17:13.473298    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08770b22_f9ff_4919_af54_c7916eba9101.slice/crio-9b02d52421726ab2c0209a7de2746dae3311220b5a54e7b5d0dc202d5e063160 WatchSource:0}: Error finding container 9b02d52421726ab2c0209a7de2746dae3311220b5a54e7b5d0dc202d5e063160: Status 404 returned error can't find the container with id 9b02d52421726ab2c0209a7de2746dae3311220b5a54e7b5d0dc202d5e063160
Nov 03 17:17:14.146506 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:14.146472    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:14.170098 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:14.170059    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg" event={"ID":"117a0c38-48f5-430d-9efe-1a308f725309","Type":"ContainerStarted","Data":"a44698ef3792984745281be214730df425cad63488b7740ed5ca2fd5b5e2fd07"}
Nov 03 17:17:14.170580 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:14.170128    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg" event={"ID":"117a0c38-48f5-430d-9efe-1a308f725309","Type":"ContainerStarted","Data":"a489931b3c0b10ffa6c66eb355a18ce99d77b45215f88b5d47e6d7081f65fdbe"}
Nov 03 17:17:14.170580 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:14.170149    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg" event={"ID":"117a0c38-48f5-430d-9efe-1a308f725309","Type":"ContainerStarted","Data":"0fb2a4c0997232dfde0191c1496a54041963ac981c8993a7d7b4c39f4904085f"}
Nov 03 17:17:14.171328 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:14.171306    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc" event={"ID":"08770b22-f9ff-4919-af54-c7916eba9101","Type":"ContainerStarted","Data":"9b02d52421726ab2c0209a7de2746dae3311220b5a54e7b5d0dc202d5e063160"}
Nov 03 17:17:14.412940 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:14.412819    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:15.104984 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.104955    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:15.112946 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.112926    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 17:17:15.116547 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.116529    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.154519 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.154481    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.154519 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.154519    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-config-volume\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.154708 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.154558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-web-config\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.154708 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.154611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jp8z\" (UniqueName: \"kubernetes.io/projected/9169e62e-57ab-4322-a0cc-664f54e48556-kube-api-access-5jp8z\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.154708 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.154696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.154839 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.154727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.154839 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.154745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9169e62e-57ab-4322-a0cc-664f54e48556-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.154839 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.154808    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.154982 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.154842    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9169e62e-57ab-4322-a0cc-664f54e48556-config-out\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.154982 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.154860    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9169e62e-57ab-4322-a0cc-664f54e48556-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.154982 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.154877    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9169e62e-57ab-4322-a0cc-664f54e48556-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.154982 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.154945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.154982 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.154972    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9169e62e-57ab-4322-a0cc-664f54e48556-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.172260 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.172231    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 17:17:15.172668 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.172599    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 17:17:15.172728 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.172717    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 17:17:15.172777 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.172741    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rppdf\""
Nov 03 17:17:15.172844 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.172803    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 17:17:15.172891 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.172858    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 17:17:15.172933 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.172914    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 17:17:15.173471 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.173451    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 17:17:15.173632 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.173620    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 17:17:15.176106 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.176080    2578 generic.go:356] "Generic (PLEG): container finished" podID="4af20ec7-7ea1-470f-b238-62e0de9ca527" containerID="f76845680a5523711815a821659ba549c7bfd57da8fbd5c1e2d862b7f315ba6f" exitCode=0
Nov 03 17:17:15.176214 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.176112    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9gj97" event={"ID":"4af20ec7-7ea1-470f-b238-62e0de9ca527","Type":"ContainerDied","Data":"f76845680a5523711815a821659ba549c7bfd57da8fbd5c1e2d862b7f315ba6f"}
Nov 03 17:17:15.182582 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.182560    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg" event={"ID":"117a0c38-48f5-430d-9efe-1a308f725309","Type":"ContainerStarted","Data":"df73b038c3e9f7a57c189956d8388d59f93f2107865d2abffe52038dd4c80210"}
Nov 03 17:17:15.255338 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.255311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.255463 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.255350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9169e62e-57ab-4322-a0cc-664f54e48556-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.255463 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.255423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.255463 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.255448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-config-volume\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.255552 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.255512    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-web-config\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.255597 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.255555    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jp8z\" (UniqueName: \"kubernetes.io/projected/9169e62e-57ab-4322-a0cc-664f54e48556-kube-api-access-5jp8z\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.255645 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.255621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.255710 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.255660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.255710 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.255691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9169e62e-57ab-4322-a0cc-664f54e48556-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.255802 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.255756    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.255893 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.255837    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9169e62e-57ab-4322-a0cc-664f54e48556-config-out\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.255893 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.255871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9169e62e-57ab-4322-a0cc-664f54e48556-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.255975 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.255915    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9169e62e-57ab-4322-a0cc-664f54e48556-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.257646 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.257248    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9169e62e-57ab-4322-a0cc-664f54e48556-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.257988 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.257966    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.258139 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.258008    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9169e62e-57ab-4322-a0cc-664f54e48556-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.259112 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.259088    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-web-config\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.259314 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.259288    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9169e62e-57ab-4322-a0cc-664f54e48556-config-out\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.259859 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.259818    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.260126 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.260101    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-config-volume\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.260305 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.260289    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.260776 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.260757    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.261144 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.261118    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9169e62e-57ab-4322-a0cc-664f54e48556-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.263679 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.263658    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.273499 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.273473    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 17:17:15.274886 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.274866    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 17:17:15.276964 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.276938    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9169e62e-57ab-4322-a0cc-664f54e48556-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.311113 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.311083    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:15.320045 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.319992    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jp8z\" (UniqueName: \"kubernetes.io/projected/9169e62e-57ab-4322-a0cc-664f54e48556-kube-api-access-5jp8z\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.426803 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.426771    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d896bf64d-smcmk"]
Nov 03 17:17:15.429820 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.429804    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.443369 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.443336    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:15.452889 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.452858    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-576dfb567c-pbb4z"]
Nov 03 17:17:15.456466 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.456447    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.456793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.456769    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.456955 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.456845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.456955 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.456907    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.456955 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.456933    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.457123 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.456958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.457123 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.456982    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.457123 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.457083    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00c39778-f1f8-4539-88eb-40d082abdc50-tmp\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.457123 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.457107    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c986b\" (UniqueName: \"kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.489977 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.489938    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d896bf64d-smcmk"]
Nov 03 17:17:15.490841 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.490473    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 17:17:15.490841 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.490614    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9495t\""
Nov 03 17:17:15.490841 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.490697    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 17:17:15.490841 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.490793    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 17:17:15.491502 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.491480    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 17:17:15.491659 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.491644    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 17:17:15.491757 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.491742    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 17:17:15.491886 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.491850    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 17:17:15.492004 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.491982    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 17:17:15.519481 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.519380    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-576dfb567c-pbb4z"]
Nov 03 17:17:15.557629 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.557593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00c39778-f1f8-4539-88eb-40d082abdc50-tmp\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.557629 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.557631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c986b\" (UniqueName: \"kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.557811 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.557658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-grpc-tls\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.557811 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.557697    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.557811 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.557720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/146e21dc-5041-463e-b0e7-9a79f969fa33-tmp\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.557811 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.557745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/146e21dc-5041-463e-b0e7-9a79f969fa33-metrics-client-ca\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.557970 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.557813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.557970 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.557843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.557970 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.557867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66cnr\" (UniqueName: \"kubernetes.io/projected/146e21dc-5041-463e-b0e7-9a79f969fa33-kube-api-access-66cnr\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.557970 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.557896    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.557970 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.557923    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.557970 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.557953    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.558185 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.557978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-tls\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.558185 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.558022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.558185 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.558023    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00c39778-f1f8-4539-88eb-40d082abdc50-tmp\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.558185 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.558075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.558185 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.558110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.558185 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.558139    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.559549 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.558741    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.559549 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.559209    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.559549 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.559263    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.559549 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.559302    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.561542 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.561519    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.561654 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.561620    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.579205 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.579130    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg" podStartSLOduration=2.043464689 podStartE2EDuration="3.579119386s" totalImagesPullingTime="1.535654697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:15.578678247 +0000 UTC m=+51.279216728" watchObservedRunningTime="2026-11-03 17:17:15.579119386 +0000 UTC m=+51.279657867"
Nov 03 17:17:15.587811 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.587786    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 17:17:15.587924 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.587815    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 17:17:15.587924 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.587840    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 17:17:15.587924 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.587870    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 17:17:15.588097 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.588060    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bpr11s5cm6kpg\""
Nov 03 17:17:15.588175 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.588159    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 17:17:15.588367 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.588339    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-h2tc2\""
Nov 03 17:17:15.620492 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.620464    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c986b\" (UniqueName: \"kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.658671 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.658641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66cnr\" (UniqueName: \"kubernetes.io/projected/146e21dc-5041-463e-b0e7-9a79f969fa33-kube-api-access-66cnr\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.658807 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.658682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.658807 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.658709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.658807 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.658790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-tls\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.658988 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.658844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.658988 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.658912    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-grpc-tls\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.658988 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.658946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.658988 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.658962    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/146e21dc-5041-463e-b0e7-9a79f969fa33-tmp\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.658988 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.658985    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/146e21dc-5041-463e-b0e7-9a79f969fa33-metrics-client-ca\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.659588 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.659560    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/146e21dc-5041-463e-b0e7-9a79f969fa33-tmp\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.659803 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.659754    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/146e21dc-5041-463e-b0e7-9a79f969fa33-metrics-client-ca\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.661468 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.661441    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.661553 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.661542    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.661859 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.661776    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.662076 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.662058    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.662259 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.662240    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-tls\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.662955 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.662937    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-grpc-tls\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.721852 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.721822    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66cnr\" (UniqueName: \"kubernetes.io/projected/146e21dc-5041-463e-b0e7-9a79f969fa33-kube-api-access-66cnr\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.740692 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.740663    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:15.778957 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.778926    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:15.883542 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:15.883514    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 17:17:15.886019 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:17:15.885991    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9169e62e_57ab_4322_a0cc_664f54e48556.slice/crio-106da788c38a5f644bcb471f6573512cea9bc0dc229b82880ddb8dfde42b5f3e WatchSource:0}: Error finding container 106da788c38a5f644bcb471f6573512cea9bc0dc229b82880ddb8dfde42b5f3e: Status 404 returned error can't find the container with id 106da788c38a5f644bcb471f6573512cea9bc0dc229b82880ddb8dfde42b5f3e
Nov 03 17:17:16.122627 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:16.122601    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d896bf64d-smcmk"]
Nov 03 17:17:16.126521 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:17:16.126495    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c39778_f1f8_4539_88eb_40d082abdc50.slice/crio-9984b1a24abe3ef16d129e5956dacf85fdbde572d7642a1c35a8b2cd64cffe6d WatchSource:0}: Error finding container 9984b1a24abe3ef16d129e5956dacf85fdbde572d7642a1c35a8b2cd64cffe6d: Status 404 returned error can't find the container with id 9984b1a24abe3ef16d129e5956dacf85fdbde572d7642a1c35a8b2cd64cffe6d
Nov 03 17:17:16.169505 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:16.169480    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-576dfb567c-pbb4z"]
Nov 03 17:17:16.172704 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:17:16.172678    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146e21dc_5041_463e_b0e7_9a79f969fa33.slice/crio-c0fe7d2678f710a09c8cebcf7cc280d5afdb14fc46576208ec3f1cbeaacfacda WatchSource:0}: Error finding container c0fe7d2678f710a09c8cebcf7cc280d5afdb14fc46576208ec3f1cbeaacfacda: Status 404 returned error can't find the container with id c0fe7d2678f710a09c8cebcf7cc280d5afdb14fc46576208ec3f1cbeaacfacda
Nov 03 17:17:16.186839 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:16.186812    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z" event={"ID":"146e21dc-5041-463e-b0e7-9a79f969fa33","Type":"ContainerStarted","Data":"c0fe7d2678f710a09c8cebcf7cc280d5afdb14fc46576208ec3f1cbeaacfacda"}
Nov 03 17:17:16.187977 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:16.187942    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d896bf64d-smcmk" event={"ID":"00c39778-f1f8-4539-88eb-40d082abdc50","Type":"ContainerStarted","Data":"9984b1a24abe3ef16d129e5956dacf85fdbde572d7642a1c35a8b2cd64cffe6d"}
Nov 03 17:17:16.189150 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:16.189131    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9169e62e-57ab-4322-a0cc-664f54e48556","Type":"ContainerStarted","Data":"106da788c38a5f644bcb471f6573512cea9bc0dc229b82880ddb8dfde42b5f3e"}
Nov 03 17:17:16.191319 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:16.191297    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9gj97" event={"ID":"4af20ec7-7ea1-470f-b238-62e0de9ca527","Type":"ContainerStarted","Data":"f834f832f79e2b5b67224ea5cac49d7e947dd16db6c9fee25e27dc6b198824ef"}
Nov 03 17:17:16.191480 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:16.191339    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9gj97" event={"ID":"4af20ec7-7ea1-470f-b238-62e0de9ca527","Type":"ContainerStarted","Data":"784e95bbfeaf86d35f786201acfef2082c533a70df134ab467df3531d8058a40"}
Nov 03 17:17:16.193652 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:16.193631    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc" event={"ID":"08770b22-f9ff-4919-af54-c7916eba9101","Type":"ContainerStarted","Data":"8e09426d65b214efacffb65c87c188d4940982a9927eb771841134e04a127972"}
Nov 03 17:17:16.193755 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:16.193681    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc" event={"ID":"08770b22-f9ff-4919-af54-c7916eba9101","Type":"ContainerStarted","Data":"2a737e555b1b90753e9ce66f662c6b6e076cb139b25d1b0851809a65570b75a5"}
Nov 03 17:17:16.193755 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:16.193713    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc" event={"ID":"08770b22-f9ff-4919-af54-c7916eba9101","Type":"ContainerStarted","Data":"913abef44ac91e41145bc81ed5b4200d88994284b5553bb98c2d059cb14615f4"}
Nov 03 17:17:16.312144 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:16.312094    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9gj97" podStartSLOduration=2.920437261 podStartE2EDuration="4.312080527s" totalImagesPullingTime="1.325695057s" totalInitContainerRuntime="65.948209ms" isStatefulPod=true podCreationTimestamp="2026-11-03 17:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:16.311554615 +0000 UTC m=+52.012093098" watchObservedRunningTime="2026-11-03 17:17:16.312080527 +0000 UTC m=+52.012619009"
Nov 03 17:17:16.432294 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:16.432194    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc" podStartSLOduration=1.931256781 podStartE2EDuration="4.432181134s" totalImagesPullingTime="2.500924353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:16.431836335 +0000 UTC m=+52.132374816" watchObservedRunningTime="2026-11-03 17:17:16.432181134 +0000 UTC m=+52.132719617"
Nov 03 17:17:17.030230 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.030197    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:17.217510 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.217436    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:17.393577 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.393499    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:17.415224 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.415195    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-67sr8"]
Nov 03 17:17:17.418917 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.418893    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-67sr8"
Nov 03 17:17:17.475340 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.474196    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-67sr8"]
Nov 03 17:17:17.475732 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.475692    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-p2lsz\""
Nov 03 17:17:17.475851 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.475748    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 17:17:17.476225 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.476196    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/504075e3-50b2-4616-afea-70d49c4d7293-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-67sr8\" (UID: \"504075e3-50b2-4616-afea-70d49c4d7293\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-67sr8"
Nov 03 17:17:17.476313 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.476254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfb4h\" (UniqueName: \"kubernetes.io/projected/504075e3-50b2-4616-afea-70d49c4d7293-kube-api-access-wfb4h\") pod \"monitoring-plugin-77f4f76594-67sr8\" (UID: \"504075e3-50b2-4616-afea-70d49c4d7293\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-67sr8"
Nov 03 17:17:17.576694 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.576664    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/504075e3-50b2-4616-afea-70d49c4d7293-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-67sr8\" (UID: \"504075e3-50b2-4616-afea-70d49c4d7293\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-67sr8"
Nov 03 17:17:17.576839 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.576721    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfb4h\" (UniqueName: \"kubernetes.io/projected/504075e3-50b2-4616-afea-70d49c4d7293-kube-api-access-wfb4h\") pod \"monitoring-plugin-77f4f76594-67sr8\" (UID: \"504075e3-50b2-4616-afea-70d49c4d7293\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-67sr8"
Nov 03 17:17:17.579371 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.579338    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/504075e3-50b2-4616-afea-70d49c4d7293-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-67sr8\" (UID: \"504075e3-50b2-4616-afea-70d49c4d7293\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-67sr8"
Nov 03 17:17:17.582492 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.582466    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:17.638454 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.638428    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfb4h\" (UniqueName: \"kubernetes.io/projected/504075e3-50b2-4616-afea-70d49c4d7293-kube-api-access-wfb4h\") pod \"monitoring-plugin-77f4f76594-67sr8\" (UID: \"504075e3-50b2-4616-afea-70d49c4d7293\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-67sr8"
Nov 03 17:17:17.734279 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.734242    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-67sr8"
Nov 03 17:17:17.853300 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.851370    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"]
Nov 03 17:17:17.858140 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.858113    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:17.883596 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.883469    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:17.883596 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.883554    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jklgn\" (UniqueName: \"kubernetes.io/projected/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-kube-api-access-jklgn\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:17.883798 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.883667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-secret-metrics-server-client-certs\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:17.883798 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.883729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-metrics-server-audit-profiles\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:17.883909 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.883843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-secret-metrics-server-tls\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:17.883909 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.883885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-audit-log\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:17.884041 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.883958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-client-ca-bundle\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:17.916641 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.915160    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 17:17:17.916641 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.915432    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 17:17:17.916641 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.916578    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"]
Nov 03 17:17:17.974502 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.974465    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 17:17:17.974711 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.974551    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 17:17:17.974711 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.974575    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-k5tst\""
Nov 03 17:17:17.974851 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.974734    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cjbocgem5ujea\""
Nov 03 17:17:17.984966 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.984931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:17.985106 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.984979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jklgn\" (UniqueName: \"kubernetes.io/projected/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-kube-api-access-jklgn\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:17.985106 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.985027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-secret-metrics-server-client-certs\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:17.985106 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.985090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-metrics-server-audit-profiles\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:17.985268 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.985128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-secret-metrics-server-tls\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:17.985268 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.985145    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-audit-log\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:17.985268 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.985164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-client-ca-bundle\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:17.985719 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.985699    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-audit-log\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:17.985816 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.985789    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:17.986049 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.986029    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-metrics-server-audit-profiles\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:17.987935 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.987912    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-secret-metrics-server-tls\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:17.988340 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.988318    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-secret-metrics-server-client-certs\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:17.988426 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:17.988392    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-client-ca-bundle\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:18.047203 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.047167    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jklgn\" (UniqueName: \"kubernetes.io/projected/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-kube-api-access-jklgn\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:18.114051 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.114017    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-67sr8"]
Nov 03 17:17:18.119067 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:17:18.119039    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504075e3_50b2_4616_afea_70d49c4d7293.slice/crio-887131d414daa40def71cf695d9574d77d5b4a3be3393f668be6ceb815b5c0ef WatchSource:0}: Error finding container 887131d414daa40def71cf695d9574d77d5b4a3be3393f668be6ceb815b5c0ef: Status 404 returned error can't find the container with id 887131d414daa40def71cf695d9574d77d5b4a3be3393f668be6ceb815b5c0ef
Nov 03 17:17:18.171951 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.171872    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:18.204933 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.204143    2578 generic.go:356] "Generic (PLEG): container finished" podID="9169e62e-57ab-4322-a0cc-664f54e48556" containerID="05f926711641665314245ac2f0f5f1eb8386119ada7a587f9810c9f3d363c1b9" exitCode=0
Nov 03 17:17:18.204933 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.204237    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9169e62e-57ab-4322-a0cc-664f54e48556","Type":"ContainerDied","Data":"05f926711641665314245ac2f0f5f1eb8386119ada7a587f9810c9f3d363c1b9"}
Nov 03 17:17:18.206426 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.206378    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-67sr8" event={"ID":"504075e3-50b2-4616-afea-70d49c4d7293","Type":"ContainerStarted","Data":"887131d414daa40def71cf695d9574d77d5b4a3be3393f668be6ceb815b5c0ef"}
Nov 03 17:17:18.395704 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.395671    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-548c4c64fb-pgglg"]
Nov 03 17:17:18.399414 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.399390    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.452198 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.452171    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 17:17:18.452198 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.452185    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 17:17:18.452424 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.452268    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zs5l8\""
Nov 03 17:17:18.453956 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.453933    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 17:17:18.454270 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.454252    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 17:17:18.454496 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.454477    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 17:17:18.455488 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.455467    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 17:17:18.462326 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.462248    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-548c4c64fb-pgglg"]
Nov 03 17:17:18.490877 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.490842    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76f649c8-7dad-410e-a2e8-960e639099d5-metrics-client-ca\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.491088 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.490888    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wr9f\" (UniqueName: \"kubernetes.io/projected/76f649c8-7dad-410e-a2e8-960e639099d5-kube-api-access-6wr9f\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.491088 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.491019    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76f649c8-7dad-410e-a2e8-960e639099d5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.491088 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.491042    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-secret-telemeter-client\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.491221 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.491150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-federate-client-tls\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.491221 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.491184    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-telemeter-client-tls\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.491295 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.491284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.491333 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.491311    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76f649c8-7dad-410e-a2e8-960e639099d5-serving-certs-ca-bundle\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.556837 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.556754    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"]
Nov 03 17:17:18.560627 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:17:18.560601    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2d426d_7ee0_48d9_b960_04f7fca5dc29.slice/crio-662e550a5bcd247573f520b50a0974bb04261438fa03101b1d02bc6cd1b4cc6b WatchSource:0}: Error finding container 662e550a5bcd247573f520b50a0974bb04261438fa03101b1d02bc6cd1b4cc6b: Status 404 returned error can't find the container with id 662e550a5bcd247573f520b50a0974bb04261438fa03101b1d02bc6cd1b4cc6b
Nov 03 17:17:18.592446 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.592418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76f649c8-7dad-410e-a2e8-960e639099d5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.592560 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.592463    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-secret-telemeter-client\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.592560 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.592532    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-federate-client-tls\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.592665 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.592567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-telemeter-client-tls\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.592665 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.592638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.592766 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.592669    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76f649c8-7dad-410e-a2e8-960e639099d5-serving-certs-ca-bundle\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.592766 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.592719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76f649c8-7dad-410e-a2e8-960e639099d5-metrics-client-ca\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.592766 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.592758    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wr9f\" (UniqueName: \"kubernetes.io/projected/76f649c8-7dad-410e-a2e8-960e639099d5-kube-api-access-6wr9f\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.593572 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.593545    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76f649c8-7dad-410e-a2e8-960e639099d5-serving-certs-ca-bundle\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.593680 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.593615    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76f649c8-7dad-410e-a2e8-960e639099d5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.593832 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.593801    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76f649c8-7dad-410e-a2e8-960e639099d5-metrics-client-ca\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.595205 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.595180    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-federate-client-tls\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.595638 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.595619    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.595810 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.595791    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-telemeter-client-tls\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.596225 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.596205    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-secret-telemeter-client\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.661205 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.661175    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wr9f\" (UniqueName: \"kubernetes.io/projected/76f649c8-7dad-410e-a2e8-960e639099d5-kube-api-access-6wr9f\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:18.716204 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:18.716158    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:19.152206 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:19.152174    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-548c4c64fb-pgglg"]
Nov 03 17:17:19.157376 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:17:19.157338    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f649c8_7dad_410e_a2e8_960e639099d5.slice/crio-4f23729abf07279979b282fbc3eca67fc51145fe2438609d20644cb97cb197a1 WatchSource:0}: Error finding container 4f23729abf07279979b282fbc3eca67fc51145fe2438609d20644cb97cb197a1: Status 404 returned error can't find the container with id 4f23729abf07279979b282fbc3eca67fc51145fe2438609d20644cb97cb197a1
Nov 03 17:17:19.211107 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:19.211074    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg" event={"ID":"76f649c8-7dad-410e-a2e8-960e639099d5","Type":"ContainerStarted","Data":"4f23729abf07279979b282fbc3eca67fc51145fe2438609d20644cb97cb197a1"}
Nov 03 17:17:19.212278 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:19.212251    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9" event={"ID":"4f2d426d-7ee0-48d9-b960-04f7fca5dc29","Type":"ContainerStarted","Data":"662e550a5bcd247573f520b50a0974bb04261438fa03101b1d02bc6cd1b4cc6b"}
Nov 03 17:17:19.214254 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:19.214233    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z" event={"ID":"146e21dc-5041-463e-b0e7-9a79f969fa33","Type":"ContainerStarted","Data":"3ff76cd2eaf3134be64b0992c1c5ef9bc450179ad4da6e0401faa4921247d9ef"}
Nov 03 17:17:19.214330 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:19.214285    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z" event={"ID":"146e21dc-5041-463e-b0e7-9a79f969fa33","Type":"ContainerStarted","Data":"9b100bdf5021a470454344eba9ecf43ea89af9ac020ffd6b8cad2f68984e8664"}
Nov 03 17:17:19.214330 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:19.214311    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z" event={"ID":"146e21dc-5041-463e-b0e7-9a79f969fa33","Type":"ContainerStarted","Data":"913e777af98972f2dfa7cd2dc01b9d1813d12ab2a3956dc04cf8bb4994df3667"}
Nov 03 17:17:19.866298 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:19.866262    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:20.112567 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:20.112538    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:20.216526 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:20.216492    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:20.893987 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:20.893945    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:21.063174 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.063102    2578 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d896bf64d-smcmk"]
Nov 03 17:17:21.097921 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.097884    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-858bfbb7dc-x6n58"]
Nov 03 17:17:21.103110 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.103078    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.108908 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.108856    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:21.161335 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.161303    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:21.174674 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.172551    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-858bfbb7dc-x6n58"]
Nov 03 17:17:21.220793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.220759    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.220974 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.220797    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.220974 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.220823    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.221098 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.220965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-tmp\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.221098 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.221016    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.221098 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.221056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjrtj\" (UniqueName: \"kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.221256 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.221102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.221256 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.221155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.225900 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.225864    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z" event={"ID":"146e21dc-5041-463e-b0e7-9a79f969fa33","Type":"ContainerStarted","Data":"4ed2923028534622930a05dcf4980e34e4ec0daf48a8a62e05a9cdec21ccf8e1"}
Nov 03 17:17:21.226040 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.225947    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z" event={"ID":"146e21dc-5041-463e-b0e7-9a79f969fa33","Type":"ContainerStarted","Data":"69823144a87972f009a550074c50eb319bca6871f83e61e3eccb97bda08b3b5c"}
Nov 03 17:17:21.226040 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.225997    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:21.226141 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.226040    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z" event={"ID":"146e21dc-5041-463e-b0e7-9a79f969fa33","Type":"ContainerStarted","Data":"b663985c55a21230b32d9b4f8d33bcc4199bd12aec7aa457a0454fe3188725cb"}
Nov 03 17:17:21.325374 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.322203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.325374 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.322247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.325374 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.322277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.325374 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.322311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-tmp\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.325374 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.322333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.325374 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.322389    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjrtj\" (UniqueName: \"kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.325374 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.322418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.325374 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.322485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.325374 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.323517    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.325374 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.323979    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.325374 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.325241    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-tmp\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.325980 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.325944    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.326306 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.326259    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.328326 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.328307    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.329801 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.329778    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.386313 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.385842    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:21.388975 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.388948    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjrtj\" (UniqueName: \"kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.418488 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.418460    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:21.492128 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.492063    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z" podStartSLOduration=2.135815661 podStartE2EDuration="6.49204296s" totalImagesPullingTime="4.356227299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:21.370542713 +0000 UTC m=+57.071081195" watchObservedRunningTime="2026-11-03 17:17:21.49204296 +0000 UTC m=+57.192581444"
Nov 03 17:17:21.572206 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.572011    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:21.822706 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:21.822671    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-858bfbb7dc-x6n58"]
Nov 03 17:17:21.826297 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:17:21.826269    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bfb6a7d_e8f9_434f_acfc_8ac3badf1350.slice/crio-ad6b0fb9aaffef859b91fc640e83dbf07b4c624dd9fe9a618643d5abc60ee82f WatchSource:0}: Error finding container ad6b0fb9aaffef859b91fc640e83dbf07b4c624dd9fe9a618643d5abc60ee82f: Status 404 returned error can't find the container with id ad6b0fb9aaffef859b91fc640e83dbf07b4c624dd9fe9a618643d5abc60ee82f
Nov 03 17:17:22.198520 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.198487    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 17:17:22.206368 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.206336    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.212973 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.212947    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:22.240274 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.240231    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d896bf64d-smcmk" event={"ID":"00c39778-f1f8-4539-88eb-40d082abdc50","Type":"ContainerStarted","Data":"5ce1bc06c7ca0db7013a0414fafca270db33f2b5d1d25f5a2d92d842e969d3c5"}
Nov 03 17:17:22.260388 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.260331    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9169e62e-57ab-4322-a0cc-664f54e48556","Type":"ContainerStarted","Data":"c271fb7eebecdd673f7e04715d9e1b3404251b9c25d93f2885787af4c4a2024f"}
Nov 03 17:17:22.260526 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.260430    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9169e62e-57ab-4322-a0cc-664f54e48556","Type":"ContainerStarted","Data":"c10696182467d9166863ec5815f265ce29cfbac9cd90d29cde647734e12ac50c"}
Nov 03 17:17:22.260526 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.260472    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9169e62e-57ab-4322-a0cc-664f54e48556","Type":"ContainerStarted","Data":"3598a1389cbaab523929e53df155a86a6e5b2f3f4f1e4a9fcddb5a6c10b9353f"}
Nov 03 17:17:22.260526 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.260500    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9169e62e-57ab-4322-a0cc-664f54e48556","Type":"ContainerStarted","Data":"e11f53e8a8e7e9b54a01c8a29f6f7d732cf76dac01bf5d052665cdf3774d9abd"}
Nov 03 17:17:22.260685 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.260526    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9169e62e-57ab-4322-a0cc-664f54e48556","Type":"ContainerStarted","Data":"2f1076bf2750c96b77091df003f21ace6b1f66ab2421ee9a735a8fbc29a894ba"}
Nov 03 17:17:22.261972 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.261943    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9" event={"ID":"4f2d426d-7ee0-48d9-b960-04f7fca5dc29","Type":"ContainerStarted","Data":"8205c91726955a240acc44458d44c1b48bd1e65094cdde0e406c645bfb655549"}
Nov 03 17:17:22.264815 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.264792    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858bfbb7dc-x6n58" event={"ID":"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350","Type":"ContainerStarted","Data":"bd2c1a3533d2ff1149b8ec0fe0ee8605177b76f09ff1efc3e019fcdf9fc504af"}
Nov 03 17:17:22.264886 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.264833    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858bfbb7dc-x6n58" event={"ID":"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350","Type":"ContainerStarted","Data":"ad6b0fb9aaffef859b91fc640e83dbf07b4c624dd9fe9a618643d5abc60ee82f"}
Nov 03 17:17:22.265170 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.265152    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 17:17:22.265282 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.265266    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 17:17:22.265965 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.265950    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 17:17:22.266066 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.266054    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 17:17:22.266235 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.266219    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 17:17:22.266681 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.266662    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-67sr8" event={"ID":"504075e3-50b2-4616-afea-70d49c4d7293","Type":"ContainerStarted","Data":"291466428fa3c754d42b0f5fd841641ff0beb4476d04fc84986cc890ec0bd17c"}
Nov 03 17:17:22.267011 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.266999    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-67sr8"
Nov 03 17:17:22.273454 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.272810    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-67sr8"
Nov 03 17:17:22.313231 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.313204    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 17:17:22.316748 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.316726    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 17:17:22.317304 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.317282    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 17:17:22.317442 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.317424    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 17:17:22.317599 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.317574    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2c6sf\""
Nov 03 17:17:22.330055 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.330028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.330152 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.330068    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.330152 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.330104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f46ad168-4931-41e7-ac08-f6fa965721f9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.330152 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.330123    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.330152 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.330138    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.330287 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.330218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.330287 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.330269    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.330384 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.330303    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-web-config\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.330437 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.330425    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f46ad168-4931-41e7-ac08-f6fa965721f9-config-out\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.330473 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.330452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.330522 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.330471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.330522 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.330488    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.330640 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.330520    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.330640 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.330542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-config\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.330640 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.330558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.330640 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.330635    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.330777 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.330702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.330777 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.330717    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.330839 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.330814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.330873 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.330855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4sxz\" (UniqueName: \"kubernetes.io/projected/f46ad168-4931-41e7-ac08-f6fa965721f9-kube-api-access-q4sxz\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.364118 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.364085    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8kr56n0suls3\""
Nov 03 17:17:22.364118 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.364103    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 17:17:22.364314 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.364235    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 17:17:22.365896 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.365874    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 17:17:22.366137 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.366116    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 17:17:22.366137 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.366143    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 17:17:22.366308 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.366260    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 17:17:22.431879 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.431841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f46ad168-4931-41e7-ac08-f6fa965721f9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.432048 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.431892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.432048 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.431924    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.432048 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.431961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.432207 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.432114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.432207 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.432147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-web-config\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.432207 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.432202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f46ad168-4931-41e7-ac08-f6fa965721f9-config-out\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.432385 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.432227    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.432385 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.432255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.432385 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.432282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.432385 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.432314    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.432385 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.432340    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-config\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.432626 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.432386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.432679 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.432634    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.433267 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.433238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.433399 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.433329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.433399 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.433373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.433505 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.433418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.433505 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.433434    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.433505 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.433449    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4sxz\" (UniqueName: \"kubernetes.io/projected/f46ad168-4931-41e7-ac08-f6fa965721f9-kube-api-access-q4sxz\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.433505 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.433487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.433724 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.433545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.434249 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.434220    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.436245 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.435780    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.436245 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.436008    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.436245 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.436155    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.436245 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.436190    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.436732 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.436638    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f46ad168-4931-41e7-ac08-f6fa965721f9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.436732 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.436652    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.436732 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.436683    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.436871 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.436791    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.437243 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.437162    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.437927 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.437711    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f46ad168-4931-41e7-ac08-f6fa965721f9-config-out\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.438496 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.437958    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.438496 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.438216    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.438496 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.438393    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-web-config\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.438936 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.438908    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.439597 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.439573    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-config\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.440257 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.440238    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.510488 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.510459    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4sxz\" (UniqueName: \"kubernetes.io/projected/f46ad168-4931-41e7-ac08-f6fa965721f9-kube-api-access-q4sxz\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.560675 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.560651    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:22.610335 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.610288    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5d896bf64d-smcmk" podStartSLOduration=2.022791092 podStartE2EDuration="7.610274269s" totalImagesPullingTime="5.587483177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:22.490054317 +0000 UTC m=+58.190592826" watchObservedRunningTime="2026-11-03 17:17:22.610274269 +0000 UTC m=+58.310812751"
Nov 03 17:17:22.611123 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.611091    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-67sr8" podStartSLOduration=2.340873417 podStartE2EDuration="5.611081199s" totalImagesPullingTime="3.270207782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:22.609680303 +0000 UTC m=+58.310218779" watchObservedRunningTime="2026-11-03 17:17:22.611081199 +0000 UTC m=+58.311619717"
Nov 03 17:17:22.668624 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.668595    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:22.731006 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.730950    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9" podStartSLOduration=3.06464295 podStartE2EDuration="5.730934563s" totalImagesPullingTime="2.666291613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:22.730228778 +0000 UTC m=+58.430767260" watchObservedRunningTime="2026-11-03 17:17:22.730934563 +0000 UTC m=+58.431473045"
Nov 03 17:17:22.863769 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.863670    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-858bfbb7dc-x6n58" podStartSLOduration=1.863653497 podStartE2EDuration="1.863653497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:22.86139722 +0000 UTC m=+58.561935713" watchObservedRunningTime="2026-11-03 17:17:22.863653497 +0000 UTC m=+58.564191980"
Nov 03 17:17:22.896450 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:22.896419    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:23.047905 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:23.047874    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 17:17:23.051456 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:17:23.051423    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46ad168_4931_41e7_ac08_f6fa965721f9.slice/crio-1e25b5f52e685e6c8026025d53999cc07360830d7380db2ba5a109782deede16 WatchSource:0}: Error finding container 1e25b5f52e685e6c8026025d53999cc07360830d7380db2ba5a109782deede16: Status 404 returned error can't find the container with id 1e25b5f52e685e6c8026025d53999cc07360830d7380db2ba5a109782deede16
Nov 03 17:17:23.102187 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:23.102160    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:23.272743 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:23.272706    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg" event={"ID":"76f649c8-7dad-410e-a2e8-960e639099d5","Type":"ContainerStarted","Data":"6f7bcdf7a3d7e08c5ab768cb2e58e059d1bf740919a4b439c3e7883b22d0f368"}
Nov 03 17:17:23.273181 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:23.272787    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg" event={"ID":"76f649c8-7dad-410e-a2e8-960e639099d5","Type":"ContainerStarted","Data":"2bbf1bcf3496cee3919b309d7a66d4b47489945a5c4eb806526c9d43b326df65"}
Nov 03 17:17:23.273181 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:23.272824    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg" event={"ID":"76f649c8-7dad-410e-a2e8-960e639099d5","Type":"ContainerStarted","Data":"d34c3acc53c7ce59af8b55f6927bc3d91852cfef6faa105e80e91afd33447b75"}
Nov 03 17:17:23.274286 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:23.274261    2578 generic.go:356] "Generic (PLEG): container finished" podID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerID="ab80803ea666ba4724c980d9da8e5702b1510504878c229ff5dc64415ff94575" exitCode=0
Nov 03 17:17:23.274421 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:23.274342    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f46ad168-4931-41e7-ac08-f6fa965721f9","Type":"ContainerDied","Data":"ab80803ea666ba4724c980d9da8e5702b1510504878c229ff5dc64415ff94575"}
Nov 03 17:17:23.274462 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:23.274432    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f46ad168-4931-41e7-ac08-f6fa965721f9","Type":"ContainerStarted","Data":"1e25b5f52e685e6c8026025d53999cc07360830d7380db2ba5a109782deede16"}
Nov 03 17:17:23.277847 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:23.277825    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9169e62e-57ab-4322-a0cc-664f54e48556","Type":"ContainerStarted","Data":"e439cf07cc380d37b78cd073f932ebe8f830fc79b69591e39985c762935b3a8c"}
Nov 03 17:17:23.405919 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:23.405824    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg" podStartSLOduration=2.062276475 podStartE2EDuration="5.405810769s" totalImagesPullingTime="3.343534294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:23.404620783 +0000 UTC m=+59.105159259" watchObservedRunningTime="2026-11-03 17:17:23.405810769 +0000 UTC m=+59.106349252"
Nov 03 17:17:23.675152 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:23.675051    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.351074233 podStartE2EDuration="8.67503727s" totalImagesPullingTime="5.309724594s" totalInitContainerRuntime="14.238443ms" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:23.67329056 +0000 UTC m=+59.373829065" watchObservedRunningTime="2026-11-03 17:17:23.67503727 +0000 UTC m=+59.375575753"
Nov 03 17:17:24.099874 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:24.099844    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:24.252987 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:24.252955    2578 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-858bfbb7dc-x6n58"]
Nov 03 17:17:24.424722 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:24.424623    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:25.703377 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:25.703314    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:25.741249 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:25.741216    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:25.932180 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:25.932144    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:26.932575 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:26.931490    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:27.129010 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:27.128962    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:27.274785 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:27.273759    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:28.299350 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:28.299318    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f46ad168-4931-41e7-ac08-f6fa965721f9","Type":"ContainerStarted","Data":"807ab55e21372b935ec0f6b61e2a61638523471c8c0f6721ea9fc4fc77bc50c2"}
Nov 03 17:17:28.299725 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:28.299403    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f46ad168-4931-41e7-ac08-f6fa965721f9","Type":"ContainerStarted","Data":"71598ebd4a04a726ae4252ab94a67843e6d610fca75c24d900bb0310185b1726"}
Nov 03 17:17:28.299725 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:28.299438    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f46ad168-4931-41e7-ac08-f6fa965721f9","Type":"ContainerStarted","Data":"235be22017efeb10a48b4d98ec7fc426552e2ab28c203c4df0025046fe88d1fe"}
Nov 03 17:17:28.299725 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:28.299464    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f46ad168-4931-41e7-ac08-f6fa965721f9","Type":"ContainerStarted","Data":"97a0cb86319a8612ce3b35d0a295dc48bd55c92e0b0ee51d9338248cb5503115"}
Nov 03 17:17:28.299725 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:28.299489    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f46ad168-4931-41e7-ac08-f6fa965721f9","Type":"ContainerStarted","Data":"b1b3f036a96309db13f2935405fe341cd62324aec92dacbdcb8265281afa00d5"}
Nov 03 17:17:28.299725 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:28.299528    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f46ad168-4931-41e7-ac08-f6fa965721f9","Type":"ContainerStarted","Data":"2422aa3f96a75eba8798b0f2d19e59a639aefbc0f2868f1381c755c865ff3ba9"}
Nov 03 17:17:28.401683 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:28.401655    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:28.438041 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:28.437988    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.431205865 podStartE2EDuration="6.437973678s" totalImagesPullingTime="3.988882777s" totalInitContainerRuntime="17.885036ms" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:28.437310704 +0000 UTC m=+64.137849208" watchObservedRunningTime="2026-11-03 17:17:28.437973678 +0000 UTC m=+64.138512159"
Nov 03 17:17:29.090595 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:29.090563    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:29.282457 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:29.282431    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:29.811405 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:29.811372    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:17:29.849793 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:29.849762    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:29.863647 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:29.863623    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 17:17:29.873802 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:29.873782    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:17:29.916444 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:29.912625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs\") pod \"network-metrics-daemon-484l6\" (UID: \"bd2a75ff-95c3-4b08-b75b-30059af6bd37\") " pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:17:30.013396 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:30.013342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l8qd\" (UniqueName: \"kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd\") pod \"network-check-target-d252q\" (UID: \"37080dcf-15b0-4d4e-85fa-ff669783f356\") " pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:17:30.038614 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:30.038587    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 17:17:30.045792 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:30.045767    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs\") pod \"network-metrics-daemon-484l6\" (UID: \"bd2a75ff-95c3-4b08-b75b-30059af6bd37\") " pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:17:30.065336 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:30.065278    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 17:17:30.077039 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:30.077018    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:30.125713 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:30.125686    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 17:17:30.137088 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:30.137063    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l8qd\" (UniqueName: \"kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd\") pod \"network-check-target-d252q\" (UID: \"37080dcf-15b0-4d4e-85fa-ff669783f356\") " pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:17:30.195306 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:30.195284    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mr6xr\""
Nov 03 17:17:30.197736 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:30.197718    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-d8tp8\""
Nov 03 17:17:30.201811 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:30.201794    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:17:30.205008 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:30.204995    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x9v6p\""
Nov 03 17:17:30.206971 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:30.206952    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:17:30.213832 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:30.213809    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:17:30.299934 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:30.299892    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:30.497082 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:30.497054    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:30.589225 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:30.589200    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d252q"]
Nov 03 17:17:30.591688 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:17:30.591665    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37080dcf_15b0_4d4e_85fa_ff669783f356.slice/crio-df31fdbf1ba4c585d7350c7c66fb65b142a3c06d2ba1133e28d4a93402d9c384 WatchSource:0}: Error finding container df31fdbf1ba4c585d7350c7c66fb65b142a3c06d2ba1133e28d4a93402d9c384: Status 404 returned error can't find the container with id df31fdbf1ba4c585d7350c7c66fb65b142a3c06d2ba1133e28d4a93402d9c384
Nov 03 17:17:30.640065 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:30.640036    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-484l6"]
Nov 03 17:17:30.642865 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:17:30.642841    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2a75ff_95c3_4b08_b75b_30059af6bd37.slice/crio-1cc1301f076c1e9e0bd08fbed459163f220e07617f012d466e4de555cc41a6c1 WatchSource:0}: Error finding container 1cc1301f076c1e9e0bd08fbed459163f220e07617f012d466e4de555cc41a6c1: Status 404 returned error can't find the container with id 1cc1301f076c1e9e0bd08fbed459163f220e07617f012d466e4de555cc41a6c1
Nov 03 17:17:30.655914 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:30.655891    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rt4lh"]
Nov 03 17:17:30.658881 ip-10-0-143-117 kubenswrapper[2578]: W1103 17:17:30.658853    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902e0389_e70e_4d23_a607_d8f4b9da868d.slice/crio-51fd7415242216b0107d7b353c460edf886a0c33b3c69cf2f9a77918a5348254 WatchSource:0}: Error finding container 51fd7415242216b0107d7b353c460edf886a0c33b3c69cf2f9a77918a5348254: Status 404 returned error can't find the container with id 51fd7415242216b0107d7b353c460edf886a0c33b3c69cf2f9a77918a5348254
Nov 03 17:17:30.904525 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:30.904435    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:31.126584 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:31.126550    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:31.312059 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:31.312024    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d252q" event={"ID":"37080dcf-15b0-4d4e-85fa-ff669783f356","Type":"ContainerStarted","Data":"df31fdbf1ba4c585d7350c7c66fb65b142a3c06d2ba1133e28d4a93402d9c384"}
Nov 03 17:17:31.313152 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:31.313122    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-484l6" event={"ID":"bd2a75ff-95c3-4b08-b75b-30059af6bd37","Type":"ContainerStarted","Data":"1cc1301f076c1e9e0bd08fbed459163f220e07617f012d466e4de555cc41a6c1"}
Nov 03 17:17:31.314237 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:31.314215    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rt4lh" event={"ID":"902e0389-e70e-4d23-a607-d8f4b9da868d","Type":"ContainerStarted","Data":"51fd7415242216b0107d7b353c460edf886a0c33b3c69cf2f9a77918a5348254"}
Nov 03 17:17:31.419507 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:31.419483    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:32.561284 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:32.561256    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:33.325806 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:33.325771    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-484l6" event={"ID":"bd2a75ff-95c3-4b08-b75b-30059af6bd37","Type":"ContainerStarted","Data":"3635f216c5d01093d2a4db6435094725df1252307c5d593669cdb4f81cad8823"}
Nov 03 17:17:33.325970 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:33.325831    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-484l6" event={"ID":"bd2a75ff-95c3-4b08-b75b-30059af6bd37","Type":"ContainerStarted","Data":"ecdf12623c40caad1b7233efed1802fca68955729800113e99573ab120b12307"}
Nov 03 17:17:33.443750 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:33.443688    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-484l6" podStartSLOduration=66.527433206 podStartE2EDuration="1m8.443667342s" totalImagesPullingTime="1.916234136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:33.443555011 +0000 UTC m=+69.144093492" watchObservedRunningTime="2026-11-03 17:17:33.443667342 +0000 UTC m=+69.144205823"
Nov 03 17:17:38.172454 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:38.172408    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:38.172983 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:38.172528    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:38.349574 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:38.349528    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d252q" event={"ID":"37080dcf-15b0-4d4e-85fa-ff669783f356","Type":"ContainerStarted","Data":"a5c5d7cd8fb3c0620a2c8083aed5522981d340a89a2fe47321248bf95a0fb52d"}
Nov 03 17:17:38.349767 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:38.349734    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:17:38.466945 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:38.466885    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-d252q" podStartSLOduration=66.669459865 podStartE2EDuration="1m13.466866358s" totalImagesPullingTime="6.797406493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:38.465094803 +0000 UTC m=+74.165633286" watchObservedRunningTime="2026-11-03 17:17:38.466866358 +0000 UTC m=+74.167404841"
Nov 03 17:17:38.473086 ip-10-0-143-117 kubenswrapper[2578]: E1103 17:17:38.473048    2578 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:17:39.135656 ip-10-0-143-117 kubenswrapper[2578]: I1103 17:17:39.135466    2578 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 17:17:39.135643 ip-10-0-143-117 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 17:17:39.148782 ip-10-0-143-117 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 17:17:39.148952 ip-10-0-143-117 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 17:17:39.149140 ip-10-0-143-117 systemd[1]: kubelet.service: Consumed 5.285s CPU time.
Nov 03 17:17:39.159253 ip-10-0-143-117 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 17:17:39.234964 ip-10-0-143-117 kubenswrapper[6059]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:17:39.234964 ip-10-0-143-117 kubenswrapper[6059]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 17:17:39.234964 ip-10-0-143-117 kubenswrapper[6059]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:17:39.234964 ip-10-0-143-117 kubenswrapper[6059]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:17:39.237248 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237155    6059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:17:39.237248 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237246    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:17:39.237248 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237250    6059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:17:39.237248 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237253    6059 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:17:39.237463 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237257    6059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:17:39.237463 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237261    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:17:39.237463 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237263    6059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:17:39.237463 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237266    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:17:39.237463 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237269    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:17:39.237463 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237273    6059 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:17:39.237463 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237276    6059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:17:39.237463 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237278    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:17:39.237463 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237281    6059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:17:39.237463 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237283    6059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:17:39.237463 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237286    6059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:17:39.237463 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237289    6059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:17:39.237463 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237291    6059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:17:39.237463 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237294    6059 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:17:39.237463 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237297    6059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:17:39.237463 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237299    6059 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:17:39.237463 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237303    6059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:17:39.237463 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237311    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:17:39.237463 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237315    6059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:17:39.237995 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237319    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:17:39.237995 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237322    6059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:17:39.237995 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237324    6059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:17:39.237995 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237327    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:17:39.237995 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237330    6059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:17:39.237995 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237333    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:17:39.237995 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237336    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:17:39.237995 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237338    6059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:17:39.237995 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237341    6059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:17:39.237995 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237344    6059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:17:39.237995 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237346    6059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:17:39.237995 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237349    6059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:17:39.237995 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237367    6059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:17:39.237995 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237372    6059 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:17:39.237995 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237377    6059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:17:39.237995 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237380    6059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:17:39.237995 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237383    6059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:17:39.237995 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237386    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:17:39.237995 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237389    6059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:17:39.237995 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237391    6059 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:17:39.238530 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237394    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:17:39.238530 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237396    6059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:17:39.238530 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237399    6059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:17:39.238530 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237402    6059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:17:39.238530 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237404    6059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:17:39.238530 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237407    6059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:17:39.238530 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237411    6059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:17:39.238530 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237413    6059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:17:39.238530 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237416    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:17:39.238530 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237420    6059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:17:39.238530 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237423    6059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:17:39.238530 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237426    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:17:39.238530 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237429    6059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:17:39.238530 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237432    6059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:17:39.238530 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237435    6059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:17:39.238530 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237438    6059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:17:39.238530 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237444    6059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:17:39.238530 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237448    6059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:17:39.238530 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237453    6059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:17:39.239017 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237456    6059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:17:39.239017 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237459    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:17:39.239017 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237462    6059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:17:39.239017 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237467    6059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:17:39.239017 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237472    6059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:17:39.239017 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237475    6059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:17:39.239017 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237478    6059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:17:39.239017 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237482    6059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:17:39.239017 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237485    6059 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:17:39.239017 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237488    6059 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:17:39.239017 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237490    6059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:17:39.239017 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237493    6059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:17:39.239017 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237496    6059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:17:39.239017 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237498    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:17:39.239017 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237501    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:17:39.239017 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237504    6059 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:17:39.239017 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237507    6059 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:17:39.239017 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237510    6059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:17:39.239017 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237513    6059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:17:39.239017 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237516    6059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:17:39.239641 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237518    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:17:39.239641 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237521    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:17:39.239641 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237524    6059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:17:39.239641 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237527    6059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:17:39.239641 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237530    6059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:17:39.239641 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237533    6059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:17:39.239641 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237537    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:17:39.239641 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237540    6059 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:17:39.239641 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237542    6059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:17:39.239641 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237545    6059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:17:39.239641 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237547    6059 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:17:39.239641 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237550    6059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:17:39.239641 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237552    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:17:39.239641 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237555    6059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:17:39.239641 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237558    6059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:17:39.239641 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237560    6059 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:17:39.239641 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237563    6059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:17:39.239641 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237565    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:17:39.239641 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237568    6059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:17:39.239641 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237571    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:17:39.240233 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237574    6059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:17:39.240233 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237576    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:17:39.240233 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237579    6059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:17:39.240233 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.237582    6059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:17:39.240233 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238540    6059 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:17:39.240233 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238553    6059 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:17:39.240233 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238557    6059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:17:39.240233 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238561    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:17:39.240233 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238565    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:17:39.240233 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238568    6059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:17:39.240233 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238571    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:17:39.240233 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238573    6059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:17:39.240233 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238576    6059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:17:39.240233 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238579    6059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:17:39.240233 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238582    6059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:17:39.240233 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238585    6059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:17:39.240233 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238589    6059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:17:39.240233 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238592    6059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:17:39.240233 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238594    6059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:17:39.240233 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238597    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:17:39.240868 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238601    6059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:17:39.240868 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238605    6059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:17:39.240868 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238609    6059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:17:39.240868 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238614    6059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:17:39.240868 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238618    6059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:17:39.240868 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238621    6059 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:17:39.240868 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238626    6059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:17:39.240868 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238631    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:17:39.240868 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238634    6059 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:17:39.240868 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238637    6059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:17:39.240868 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238639    6059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:17:39.240868 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238643    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:17:39.240868 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238645    6059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:17:39.240868 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238649    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:17:39.240868 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238652    6059 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:17:39.240868 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238654    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:17:39.240868 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238657    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:17:39.240868 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238660    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:17:39.240868 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238664    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:17:39.241446 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238667    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:17:39.241446 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238670    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:17:39.241446 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238672    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:17:39.241446 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238675    6059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:17:39.241446 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238680    6059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:17:39.241446 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238684    6059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:17:39.241446 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238688    6059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:17:39.241446 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238692    6059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:17:39.241446 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238696    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:17:39.241446 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238698    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:17:39.241446 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238702    6059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:17:39.241446 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238705    6059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:17:39.241446 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238708    6059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:17:39.241446 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238711    6059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:17:39.241446 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238713    6059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:17:39.241446 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238716    6059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:17:39.241446 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238719    6059 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:17:39.241446 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238721    6059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:17:39.241446 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238724    6059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:17:39.241446 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238727    6059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:17:39.242033 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238730    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:17:39.242033 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238733    6059 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:17:39.242033 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238735    6059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:17:39.242033 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238738    6059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:17:39.242033 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238741    6059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:17:39.242033 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238744    6059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:17:39.242033 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238747    6059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:17:39.242033 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238750    6059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:17:39.242033 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238753    6059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:17:39.242033 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238757    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:17:39.242033 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238762    6059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:17:39.242033 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238767    6059 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:17:39.242033 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238771    6059 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:17:39.242033 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238774    6059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:17:39.242033 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238778    6059 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:17:39.242033 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238780    6059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:17:39.242033 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238783    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:17:39.242033 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238786    6059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:17:39.242033 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238789    6059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:17:39.242614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238793    6059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:17:39.242614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238798    6059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:17:39.242614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238802    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:17:39.242614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238805    6059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:17:39.242614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238808    6059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:17:39.242614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238811    6059 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:17:39.242614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238814    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:17:39.242614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238816    6059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:17:39.242614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238819    6059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:17:39.242614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238823    6059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:17:39.242614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238825    6059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:17:39.242614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238828    6059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:17:39.242614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238831    6059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:17:39.242614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238838    6059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:17:39.242614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238844    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:17:39.242614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238849    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:17:39.242614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238852    6059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:17:39.242614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238855    6059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:17:39.242614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238858    6059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:17:39.243213 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238861    6059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:17:39.243213 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238863    6059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:17:39.243213 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238866    6059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:17:39.243213 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238869    6059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:17:39.243213 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238871    6059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:17:39.243213 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238874    6059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:17:39.243213 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238876    6059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:17:39.243213 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238879    6059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:17:39.243213 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238882    6059 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:17:39.243213 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238884    6059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:17:39.243213 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238889    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:17:39.243213 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238891    6059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:17:39.243213 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.238894    6059 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:17:39.243213 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239737    6059 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:17:39.243213 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239753    6059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:17:39.243213 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239759    6059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:17:39.243213 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239764    6059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:17:39.243213 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239770    6059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:17:39.243213 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239775    6059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:17:39.243816 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239780    6059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:17:39.243816 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239786    6059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:17:39.243816 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239792    6059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:17:39.243816 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239797    6059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:17:39.243816 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239802    6059 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:17:39.243816 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239805    6059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:17:39.243816 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239809    6059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:17:39.243816 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239813    6059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:17:39.243816 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239817    6059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:17:39.243816 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239822    6059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:17:39.243816 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239827    6059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:17:39.243816 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239831    6059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:17:39.243816 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239834    6059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:17:39.243816 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239836    6059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:17:39.243816 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239839    6059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:17:39.243816 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239842    6059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:17:39.243816 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239845    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:17:39.243816 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239848    6059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:17:39.243816 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239851    6059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:17:39.244386 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239853    6059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:17:39.244386 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239856    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:17:39.244386 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239859    6059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:17:39.244386 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239861    6059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:17:39.244386 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239864    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:17:39.244386 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239866    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:17:39.244386 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239869    6059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:17:39.244386 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239873    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:17:39.244386 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239883    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:17:39.244386 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239887    6059 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:17:39.244386 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239892    6059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:17:39.244386 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239896    6059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:17:39.244386 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239901    6059 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:17:39.244386 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239907    6059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:17:39.244386 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239911    6059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:17:39.244386 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239916    6059 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:17:39.244386 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239920    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:17:39.244386 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239924    6059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:17:39.244386 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239929    6059 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:17:39.244386 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239934    6059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:17:39.245098 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239938    6059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:17:39.245098 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239943    6059 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:17:39.245098 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239947    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:17:39.245098 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239951    6059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:17:39.245098 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239955    6059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:17:39.245098 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239959    6059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:17:39.245098 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239963    6059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:17:39.245098 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239968    6059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:17:39.245098 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239972    6059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:17:39.245098 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239976    6059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:17:39.245098 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239980    6059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:17:39.245098 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239986    6059 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:17:39.245098 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239991    6059 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:17:39.245098 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.239998    6059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:17:39.245098 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240003    6059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:17:39.245098 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240008    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:17:39.245098 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240012    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:17:39.245098 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240017    6059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:17:39.245098 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240021    6059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:17:39.245732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240025    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:17:39.245732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240030    6059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:17:39.245732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240034    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:17:39.245732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240039    6059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:17:39.245732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240043    6059 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:17:39.245732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240047    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:17:39.245732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240052    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:17:39.245732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240056    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:17:39.245732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240060    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:17:39.245732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240064    6059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:17:39.245732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240069    6059 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:17:39.245732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240072    6059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:17:39.245732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240076    6059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:17:39.245732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240081    6059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:17:39.245732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240085    6059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:17:39.245732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240089    6059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:17:39.245732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240094    6059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:17:39.245732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240098    6059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:17:39.245732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240103    6059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:17:39.245732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240108    6059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:17:39.246614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240112    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:17:39.246614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240116    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:17:39.246614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240120    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:17:39.246614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240124    6059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:17:39.246614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240130    6059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:17:39.246614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240137    6059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:17:39.246614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240141    6059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:17:39.246614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240145    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:17:39.246614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240149    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:17:39.246614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240153    6059 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:17:39.246614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240158    6059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:17:39.246614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240162    6059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:17:39.246614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240166    6059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:17:39.246614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240172    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:17:39.246614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240177    6059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:17:39.246614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240181    6059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:17:39.246614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240185    6059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:17:39.246614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240188    6059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:17:39.246614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240193    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:17:39.246614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240198    6059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240202    6059 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.240206    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240406    6059 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240420    6059 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240433    6059 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240441    6059 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240449    6059 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240458    6059 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240466    6059 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240473    6059 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240479    6059 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240485    6059 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240492    6059 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240498    6059 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240503    6059 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240508    6059 flags.go:64] FLAG: --cgroup-root=""
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240514    6059 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240520    6059 flags.go:64] FLAG: --client-ca-file=""
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240525    6059 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240529    6059 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240536    6059 flags.go:64] FLAG: --cluster-domain=""
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240542    6059 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240548    6059 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 17:17:39.247194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240554    6059 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240559    6059 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240567    6059 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240573    6059 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240578    6059 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240584    6059 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240589    6059 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240595    6059 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240600    6059 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240607    6059 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240613    6059 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240621    6059 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240627    6059 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240632    6059 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240637    6059 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240642    6059 flags.go:64] FLAG: --enable-server="true"
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240652    6059 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240661    6059 flags.go:64] FLAG: --event-burst="100"
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240666    6059 flags.go:64] FLAG: --event-qps="50"
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240670    6059 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240675    6059 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240680    6059 flags.go:64] FLAG: --eviction-hard=""
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240687    6059 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240691    6059 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240697    6059 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 17:17:39.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240701    6059 flags.go:64] FLAG: --eviction-soft=""
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240706    6059 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240710    6059 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240714    6059 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240719    6059 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240723    6059 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240728    6059 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240732    6059 flags.go:64] FLAG: --feature-gates=""
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240739    6059 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240743    6059 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240748    6059 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240753    6059 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240757    6059 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240763    6059 flags.go:64] FLAG: --help="false"
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240768    6059 flags.go:64] FLAG: --hostname-override="ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240774    6059 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240777    6059 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240782    6059 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240785    6059 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240789    6059 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240792    6059 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240795    6059 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240798    6059 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 17:17:39.248945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240804    6059 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240807    6059 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240810    6059 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240813    6059 flags.go:64] FLAG: --kube-reserved=""
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240817    6059 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240820    6059 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240823    6059 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240826    6059 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240829    6059 flags.go:64] FLAG: --lock-file=""
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240832    6059 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240836    6059 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240839    6059 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240845    6059 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240849    6059 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240852    6059 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240855    6059 flags.go:64] FLAG: --logging-format="text"
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240858    6059 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240862    6059 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240865    6059 flags.go:64] FLAG: --manifest-url=""
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240868    6059 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240873    6059 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240876    6059 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240880    6059 flags.go:64] FLAG: --max-pods="110"
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240884    6059 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240887    6059 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 17:17:39.250035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240891    6059 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240894    6059 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240898    6059 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240901    6059 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240904    6059 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240912    6059 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240915    6059 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240920    6059 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240924    6059 flags.go:64] FLAG: --pod-cidr=""
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240927    6059 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240930    6059 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240933    6059 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240936    6059 flags.go:64] FLAG: --port="10250"
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240939    6059 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240942    6059 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-00fb8ca6b0119f7b9"
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240946    6059 flags.go:64] FLAG: --qos-reserved=""
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240949    6059 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240952    6059 flags.go:64] FLAG: --register-node="true"
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240955    6059 flags.go:64] FLAG: --register-with-taints=""
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240959    6059 flags.go:64] FLAG: --registry-burst="10"
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240962    6059 flags.go:64] FLAG: --registry-qps="5"
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240965    6059 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240968    6059 flags.go:64] FLAG: --reserved-memory=""
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240972    6059 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240976    6059 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240979    6059 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 17:17:39.250971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240982    6059 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240985    6059 flags.go:64] FLAG: --runonce="false"
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240988    6059 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240991    6059 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240995    6059 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.240998    6059 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241001    6059 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241004    6059 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241009    6059 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241013    6059 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241017    6059 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241020    6059 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241023    6059 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241028    6059 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241031    6059 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241034    6059 flags.go:64] FLAG: --system-cgroups=""
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241038    6059 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241044    6059 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241047    6059 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241050    6059 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241054    6059 flags.go:64] FLAG: --tls-min-version=""
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241057    6059 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241060    6059 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241063    6059 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241066    6059 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 17:17:39.251733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241070    6059 flags.go:64] FLAG: --v="2"
Nov 03 17:17:39.252351 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241074    6059 flags.go:64] FLAG: --version="false"
Nov 03 17:17:39.252351 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241079    6059 flags.go:64] FLAG: --vmodule=""
Nov 03 17:17:39.252351 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241083    6059 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 17:17:39.252351 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241087    6059 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 17:17:39.252351 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241257    6059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:17:39.252351 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241261    6059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:17:39.252351 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241264    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:17:39.252351 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241267    6059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:17:39.252351 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241270    6059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:17:39.252351 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241273    6059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:17:39.252351 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241275    6059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:17:39.252351 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241278    6059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:17:39.252351 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241281    6059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:17:39.252351 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241283    6059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:17:39.252351 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241288    6059 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:17:39.252351 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241291    6059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:17:39.252351 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241299    6059 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:17:39.252351 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241302    6059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:17:39.252351 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241305    6059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:17:39.252351 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241309    6059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:17:39.252351 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241312    6059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:17:39.252880 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241315    6059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:17:39.252880 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241318    6059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:17:39.252880 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241320    6059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:17:39.252880 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241323    6059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:17:39.252880 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241326    6059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:17:39.252880 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241328    6059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:17:39.252880 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241331    6059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:17:39.252880 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241334    6059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:17:39.252880 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241337    6059 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:17:39.252880 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241340    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:17:39.252880 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241342    6059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:17:39.252880 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241345    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:17:39.252880 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241347    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:17:39.252880 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241350    6059 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:17:39.252880 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241370    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:17:39.252880 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241375    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:17:39.252880 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241379    6059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:17:39.252880 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241382    6059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:17:39.252880 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241384    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:17:39.253371 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241387    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:17:39.253371 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241389    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:17:39.253371 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241392    6059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:17:39.253371 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241394    6059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:17:39.253371 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241397    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:17:39.253371 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241400    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:17:39.253371 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241404    6059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:17:39.253371 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241407    6059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:17:39.253371 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241410    6059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:17:39.253371 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241419    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:17:39.253371 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241422    6059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:17:39.253371 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241426    6059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:17:39.253371 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241429    6059 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:17:39.253371 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241431    6059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:17:39.253371 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241434    6059 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:17:39.253371 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241437    6059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:17:39.253371 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241439    6059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:17:39.253371 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241442    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:17:39.253371 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241445    6059 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:17:39.253371 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241448    6059 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:17:39.253871 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241451    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:17:39.253871 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241454    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:17:39.253871 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241457    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:17:39.253871 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241459    6059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:17:39.253871 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241462    6059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:17:39.253871 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241465    6059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:17:39.253871 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241468    6059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:17:39.253871 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241471    6059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:17:39.253871 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241473    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:17:39.253871 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241476    6059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:17:39.253871 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241479    6059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:17:39.253871 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241481    6059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:17:39.253871 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241486    6059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:17:39.253871 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241490    6059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:17:39.253871 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241493    6059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:17:39.253871 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241497    6059 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:17:39.253871 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241500    6059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:17:39.253871 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241503    6059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:17:39.253871 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241508    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:17:39.254369 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241511    6059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:17:39.254369 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241513    6059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:17:39.254369 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241516    6059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:17:39.254369 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241524    6059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:17:39.254369 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241528    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:17:39.254369 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241530    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:17:39.254369 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241533    6059 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:17:39.254369 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241536    6059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:17:39.254369 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241538    6059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:17:39.254369 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241541    6059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:17:39.254369 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241543    6059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:17:39.254369 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241546    6059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:17:39.254369 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241550    6059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:17:39.254369 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241553    6059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:17:39.254369 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241557    6059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:17:39.254369 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241559    6059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:17:39.254369 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241562    6059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:17:39.254369 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241564    6059 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:17:39.254369 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241567    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:17:39.254946 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241569    6059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:17:39.254946 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241572    6059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:17:39.254946 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241574    6059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:17:39.254946 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241577    6059 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:17:39.254946 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241579    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:17:39.254946 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241582    6059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:17:39.254946 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241585    6059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:17:39.254946 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241588    6059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:17:39.254946 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241590    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:17:39.254946 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241593    6059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:17:39.254946 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241595    6059 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:17:39.254946 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241598    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:17:39.254946 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.241629    6059 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:17:39.254946 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241671    6059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:17:39.254946 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241676    6059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:17:39.254946 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241679    6059 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:17:39.255571 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241682    6059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:17:39.255571 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241690    6059 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:17:39.255571 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241692    6059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:17:39.255571 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241695    6059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:17:39.255571 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241698    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:17:39.255571 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241701    6059 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:17:39.255571 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241704    6059 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:17:39.255571 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241707    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:17:39.255571 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241709    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:17:39.255571 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241712    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:17:39.255571 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241715    6059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:17:39.255571 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241717    6059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:17:39.255571 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241720    6059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:17:39.255571 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241722    6059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:17:39.255571 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241725    6059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:17:39.255571 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241728    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:17:39.255571 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241730    6059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:17:39.255571 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241733    6059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:17:39.255571 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241736    6059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:17:39.256146 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241740    6059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:17:39.256146 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241744    6059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:17:39.256146 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241747    6059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:17:39.256146 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241749    6059 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:17:39.256146 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241752    6059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:17:39.256146 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241755    6059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:17:39.256146 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241757    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:17:39.256146 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241760    6059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:17:39.256146 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241762    6059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:17:39.256146 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241766    6059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:17:39.256146 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241769    6059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:17:39.256146 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241772    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:17:39.256146 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241774    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:17:39.256146 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241777    6059 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:17:39.256146 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241779    6059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:17:39.256146 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241787    6059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:17:39.256146 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241790    6059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:17:39.256146 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241793    6059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:17:39.256146 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241795    6059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:17:39.256770 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241799    6059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:17:39.256770 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241802    6059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:17:39.256770 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241805    6059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:17:39.256770 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241807    6059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:17:39.256770 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241810    6059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:17:39.256770 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241812    6059 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:17:39.256770 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241815    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:17:39.256770 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241817    6059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:17:39.256770 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241820    6059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:17:39.256770 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241823    6059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:17:39.256770 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241825    6059 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:17:39.256770 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241828    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:17:39.256770 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241830    6059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:17:39.256770 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241834    6059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:17:39.256770 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241838    6059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:17:39.256770 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241840    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:17:39.256770 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241843    6059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:17:39.256770 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241846    6059 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:17:39.256770 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241848    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:17:39.256770 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241851    6059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:17:39.257339 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241854    6059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:17:39.257339 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241857    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:17:39.257339 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241861    6059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:17:39.257339 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241863    6059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:17:39.257339 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241866    6059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:17:39.257339 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241869    6059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:17:39.257339 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241871    6059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:17:39.257339 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241879    6059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:17:39.257339 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241882    6059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:17:39.257339 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241890    6059 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:17:39.257339 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241893    6059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:17:39.257339 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241896    6059 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:17:39.257339 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241898    6059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:17:39.257339 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241901    6059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:17:39.257339 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241903    6059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:17:39.257339 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241906    6059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:17:39.257339 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241908    6059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:17:39.257339 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241911    6059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:17:39.257339 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241913    6059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:17:39.257339 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241916    6059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:17:39.258241 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241918    6059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:17:39.258241 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241921    6059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:17:39.258241 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241923    6059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:17:39.258241 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241926    6059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:17:39.258241 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241929    6059 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:17:39.258241 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241931    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:17:39.258241 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241936    6059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:17:39.258241 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241938    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:17:39.258241 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241941    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:17:39.258241 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241943    6059 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:17:39.258241 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241946    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:17:39.258241 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241949    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:17:39.258241 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241951    6059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:17:39.258241 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241954    6059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:17:39.258241 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241958    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:17:39.258241 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241960    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:17:39.258241 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241963    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:17:39.258241 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241965    6059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:17:39.258241 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241968    6059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:17:39.259080 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241971    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:17:39.259080 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241973    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:17:39.259080 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241976    6059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:17:39.259080 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241984    6059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:17:39.259080 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241987    6059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:17:39.259080 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.241990    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:17:39.259080 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.248378    6059 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 17:17:39.259080 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.248404    6059 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 17:17:39.259080 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.248432    6059 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 17:17:39.259080 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.248438    6059 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 17:17:39.259080 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248537    6059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:17:39.259080 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248545    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:17:39.259080 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248549    6059 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:17:39.259080 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248552    6059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:17:39.259080 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248555    6059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:17:39.259080 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248558    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:17:39.259080 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248561    6059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:17:39.259080 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248564    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:17:39.259080 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248566    6059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:17:39.259677 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248569    6059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:17:39.259677 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248571    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:17:39.259677 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248574    6059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:17:39.259677 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248577    6059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:17:39.259677 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248580    6059 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:17:39.259677 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248583    6059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:17:39.259677 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248589    6059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:17:39.259677 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248593    6059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:17:39.259677 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248598    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:17:39.259677 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248602    6059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:17:39.259677 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248607    6059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:17:39.259677 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248612    6059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:17:39.259677 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248616    6059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:17:39.259677 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248619    6059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:17:39.259677 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248622    6059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:17:39.259677 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248625    6059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:17:39.259677 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248628    6059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:17:39.259677 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248631    6059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:17:39.259677 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248634    6059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:17:39.260184 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248637    6059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:17:39.260184 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248639    6059 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:17:39.260184 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248642    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:17:39.260184 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248645    6059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:17:39.260184 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248648    6059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:17:39.260184 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248650    6059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:17:39.260184 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248653    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:17:39.260184 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248655    6059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:17:39.260184 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248658    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:17:39.260184 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248661    6059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:17:39.260184 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248663    6059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:17:39.260184 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248667    6059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:17:39.260184 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248669    6059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:17:39.260184 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248672    6059 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:17:39.260184 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248675    6059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:17:39.260184 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248677    6059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:17:39.260184 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248680    6059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:17:39.260184 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248682    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:17:39.260184 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248686    6059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:17:39.260184 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248689    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:17:39.260708 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248692    6059 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:17:39.260708 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248694    6059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:17:39.260708 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248697    6059 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:17:39.260708 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248700    6059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:17:39.260708 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248703    6059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:17:39.260708 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248705    6059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:17:39.260708 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248708    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:17:39.260708 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248711    6059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:17:39.260708 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248713    6059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:17:39.260708 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248717    6059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:17:39.260708 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248722    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:17:39.260708 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248725    6059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:17:39.260708 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248728    6059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:17:39.260708 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248731    6059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:17:39.260708 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248733    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:17:39.260708 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248736    6059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:17:39.260708 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248738    6059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:17:39.260708 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248741    6059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:17:39.260708 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248744    6059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:17:39.260708 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248746    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:17:39.261284 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248749    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:17:39.261284 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248751    6059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:17:39.261284 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248755    6059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:17:39.261284 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248758    6059 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:17:39.261284 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248760    6059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:17:39.261284 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248763    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:17:39.261284 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248766    6059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:17:39.261284 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248768    6059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:17:39.261284 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248773    6059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:17:39.261284 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248777    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:17:39.261284 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248780    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:17:39.261284 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248784    6059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:17:39.261284 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248787    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:17:39.261284 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248789    6059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:17:39.261284 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248792    6059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:17:39.261284 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248795    6059 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:17:39.261284 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248797    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:17:39.261284 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248800    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:17:39.261284 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248803    6059 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:17:39.261810 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248805    6059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:17:39.261810 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248808    6059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:17:39.261810 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248811    6059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:17:39.261810 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248813    6059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:17:39.261810 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248816    6059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:17:39.261810 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248819    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:17:39.261810 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248821    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:17:39.261810 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248824    6059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:17:39.261810 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248826    6059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:17:39.261810 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248829    6059 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:17:39.261810 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248831    6059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:17:39.261810 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248834    6059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:17:39.261810 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248837    6059 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:17:39.261810 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248839    6059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:17:39.261810 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248842    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:17:39.261810 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248845    6059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:17:39.261810 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248847    6059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:17:39.261810 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248850    6059 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:17:39.261810 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.248852    6059 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:17:39.262303 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.248884    6059 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:17:39.262303 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249046    6059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:17:39.262303 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249052    6059 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:17:39.262303 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249056    6059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:17:39.262303 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249060    6059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:17:39.262303 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249065    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:17:39.262303 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249069    6059 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:17:39.262303 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249072    6059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:17:39.262303 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249075    6059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:17:39.262303 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249078    6059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:17:39.262303 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249081    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:17:39.262303 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249085    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:17:39.262303 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249088    6059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:17:39.262303 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249090    6059 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:17:39.262303 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249093    6059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:17:39.262303 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249096    6059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:17:39.262732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249098    6059 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:17:39.262732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249101    6059 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:17:39.262732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249103    6059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:17:39.262732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249106    6059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:17:39.262732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249109    6059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:17:39.262732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249112    6059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:17:39.262732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249115    6059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:17:39.262732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249117    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:17:39.262732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249120    6059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:17:39.262732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249123    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:17:39.262732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249126    6059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:17:39.262732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249129    6059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:17:39.262732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249131    6059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:17:39.262732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249134    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:17:39.262732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249137    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:17:39.262732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249140    6059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:17:39.262732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249142    6059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:17:39.262732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249145    6059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:17:39.262732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249148    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:17:39.262732 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249150    6059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:17:39.263268 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249153    6059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:17:39.263268 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249156    6059 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:17:39.263268 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249158    6059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:17:39.263268 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249167    6059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:17:39.263268 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249170    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:17:39.263268 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249173    6059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:17:39.263268 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249175    6059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:17:39.263268 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249178    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:17:39.263268 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249181    6059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:17:39.263268 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249183    6059 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:17:39.263268 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249186    6059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:17:39.263268 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249189    6059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:17:39.263268 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249192    6059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:17:39.263268 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249194    6059 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:17:39.263268 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249197    6059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:17:39.263268 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249199    6059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:17:39.263268 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249202    6059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:17:39.263268 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249205    6059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:17:39.263268 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249208    6059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:17:39.263268 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249210    6059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:17:39.263885 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249213    6059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:17:39.263885 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249216    6059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:17:39.263885 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249219    6059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:17:39.263885 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249221    6059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:17:39.263885 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249224    6059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:17:39.263885 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249226    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:17:39.263885 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249229    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:17:39.263885 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249232    6059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:17:39.263885 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249235    6059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:17:39.263885 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249237    6059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:17:39.263885 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249240    6059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:17:39.263885 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249243    6059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:17:39.263885 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249245    6059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:17:39.263885 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249248    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:17:39.263885 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249250    6059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:17:39.263885 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249253    6059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:17:39.263885 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249256    6059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:17:39.263885 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249259    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:17:39.263885 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249261    6059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:17:39.264374 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249264    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:17:39.264374 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249266    6059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:17:39.264374 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249268    6059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:17:39.264374 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249271    6059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:17:39.264374 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249274    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:17:39.264374 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249276    6059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:17:39.264374 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249279    6059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:17:39.264374 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249281    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:17:39.264374 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249284    6059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:17:39.264374 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249287    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:17:39.264374 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249290    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:17:39.264374 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249292    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:17:39.264374 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249295    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:17:39.264374 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249298    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:17:39.264374 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249300    6059 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:17:39.264374 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249303    6059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:17:39.264374 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249306    6059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:17:39.264374 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249308    6059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:17:39.264374 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249310    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:17:39.264906 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249313    6059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:17:39.264906 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249316    6059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:17:39.264906 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249318    6059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:17:39.264906 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249321    6059 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:17:39.264906 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249323    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:17:39.264906 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249326    6059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:17:39.264906 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249328    6059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:17:39.264906 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249331    6059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:17:39.264906 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249333    6059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:17:39.264906 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249336    6059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:17:39.264906 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249338    6059 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:17:39.264906 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249341    6059 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:17:39.264906 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249344    6059 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:17:39.264906 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.249387    6059 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:17:39.264906 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249424    6059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:17:39.264906 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249429    6059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:17:39.265307 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249432    6059 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:17:39.265307 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249436    6059 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:17:39.265307 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249438    6059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:17:39.265307 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249442    6059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:17:39.265307 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249445    6059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:17:39.265307 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249448    6059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:17:39.265307 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249450    6059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:17:39.265307 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249456    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:17:39.265307 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249458    6059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:17:39.265307 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249461    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:17:39.265307 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249464    6059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:17:39.265307 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249467    6059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:17:39.265307 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249469    6059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:17:39.265307 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249472    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:17:39.265307 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249475    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:17:39.265307 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249477    6059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:17:39.265307 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249480    6059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:17:39.265307 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249482    6059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:17:39.265307 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249485    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:17:39.265307 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249488    6059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:17:39.265940 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249491    6059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:17:39.265940 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249493    6059 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:17:39.265940 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249496    6059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:17:39.265940 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249498    6059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:17:39.265940 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249501    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:17:39.265940 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249503    6059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:17:39.265940 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249506    6059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:17:39.265940 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249509    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:17:39.265940 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249512    6059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:17:39.265940 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249515    6059 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:17:39.265940 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249518    6059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:17:39.265940 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249520    6059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:17:39.265940 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249523    6059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:17:39.265940 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249528    6059 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:17:39.265940 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249530    6059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:17:39.265940 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249533    6059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:17:39.265940 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249535    6059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:17:39.265940 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249538    6059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:17:39.265940 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249541    6059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:17:39.265940 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249545    6059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:17:39.266788 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249548    6059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:17:39.266788 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249550    6059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:17:39.266788 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249553    6059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:17:39.266788 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249556    6059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:17:39.266788 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249558    6059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:17:39.266788 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249561    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:17:39.266788 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249563    6059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:17:39.266788 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249566    6059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:17:39.266788 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249569    6059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:17:39.266788 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249572    6059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:17:39.266788 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249574    6059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:17:39.266788 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249577    6059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:17:39.266788 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249579    6059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:17:39.266788 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249582    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:17:39.266788 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249585    6059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:17:39.266788 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249587    6059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:17:39.266788 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249590    6059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:17:39.266788 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249592    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:17:39.266788 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249595    6059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:17:39.267406 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249597    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:17:39.267406 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249600    6059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:17:39.267406 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249603    6059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:17:39.267406 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249606    6059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:17:39.267406 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249609    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:17:39.267406 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249611    6059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:17:39.267406 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249616    6059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:17:39.267406 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249618    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:17:39.267406 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249621    6059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:17:39.267406 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249623    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:17:39.267406 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249626    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:17:39.267406 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249628    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:17:39.267406 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249633    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:17:39.267406 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249635    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:17:39.267406 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249638    6059 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:17:39.267406 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249641    6059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:17:39.267406 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249643    6059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:17:39.267406 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249646    6059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:17:39.267406 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249648    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:17:39.267936 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249651    6059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:17:39.267936 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249654    6059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:17:39.267936 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249656    6059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:17:39.267936 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249659    6059 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:17:39.267936 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249662    6059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:17:39.267936 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249664    6059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:17:39.267936 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249667    6059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:17:39.267936 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249669    6059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:17:39.267936 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249672    6059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:17:39.267936 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249674    6059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:17:39.267936 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249677    6059 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:17:39.267936 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249679    6059 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:17:39.267936 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249682    6059 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:17:39.267936 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249684    6059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:17:39.267936 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249687    6059 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:17:39.267936 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249691    6059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:17:39.267936 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249696    6059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:17:39.267936 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249700    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:17:39.267936 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249703    6059 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:17:39.268426 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249707    6059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:17:39.268426 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249709    6059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:17:39.268426 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249712    6059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:17:39.268426 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249714    6059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:17:39.268426 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249717    6059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:17:39.268426 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249721    6059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:17:39.268426 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:17:39.249725    6059 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:17:39.268426 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.249794    6059 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 17:17:39.268426 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.251143    6059 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 17:17:39.268426 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.251228    6059 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 17:17:39.268426 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.251533    6059 server.go:1014] "Starting client certificate rotation"
Nov 03 17:17:39.268426 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.251671    6059 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 17:17:39.268426 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.251730    6059 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 17:11:25 +0000 UTC" deadline="2028-04-05 12:42:46.960466571 +0000 UTC"
Nov 03 17:17:39.268426 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.251763    6059 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12451h25m7.708707124s"
Nov 03 17:17:39.268426 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.252296    6059 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 17:17:39.268426 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.254115    6059 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 17:17:39.268834 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.255707    6059 log.go:25] "Validated CRI v1 runtime API"
Nov 03 17:17:39.268834 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.258073    6059 log.go:25] "Validated CRI v1 image API"
Nov 03 17:17:39.268834 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.259001    6059 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 17:17:39.268834 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.263099    6059 fs.go:135] Filesystem UUIDs: map[279dd0c6-92a3-441b-989b-0d0f46d02635:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d99b5073-8e62-430a-b123-093e8eb6f858:/dev/nvme0n1p3]
Nov 03 17:17:39.268969 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.263129    6059 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0fb2a4c0997232dfde0191c1496a54041963ac981c8993a7d7b4c39f4904085f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0fb2a4c0997232dfde0191c1496a54041963ac981c8993a7d7b4c39f4904085f/userdata/shm major:0 minor:257 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/106da788c38a5f644bcb471f6573512cea9bc0dc229b82880ddb8dfde42b5f3e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/106da788c38a5f644bcb471f6573512cea9bc0dc229b82880ddb8dfde42b5f3e/userdata/shm major:0 minor:307 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1697bed59d554b35eb3bcdf037c62cf04db88502c215f7b4275cc6cbb4f0fcf7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1697bed59d554b35eb3bcdf037c62cf04db88502c215f7b4275cc6cbb4f0fcf7/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1cc1301f076c1e9e0bd08fbed459163f220e07617f012d466e4de555cc41a6c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1cc1301f076c1e9e0bd08fbed459163f220e07617f012d466e4de555cc41a6c1/userdata/shm major:0 minor:531 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e25b5f52e685e6c8026025d53999cc07360830d7380db2ba5a109782deede16/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e25b5f52e685e6c8026025d53999cc07360830d7380db2ba5a109782deede16/userdata/shm major:0 minor:480 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/388325401b9403f65e71e7f80c658c2e479db62a04148a3bda6cf565cc0dee0a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/388325401b9403f65e71e7f80c658c2e479db62a04148a3bda6cf565cc0dee0a/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/38d59c19828a6bacf51ada0988f5e0ec2477f8d6fa87199490886da6f1bf6528/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/38d59c19828a6bacf51ada0988f5e0ec2477f8d6fa87199490886da6f1bf6528/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f23729abf07279979b282fbc3eca67fc51145fe2438609d20644cb97cb197a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f23729abf07279979b282fbc3eca67fc51145fe2438609d20644cb97cb197a1/userdata/shm major:0 minor:364 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51fd7415242216b0107d7b353c460edf886a0c33b3c69cf2f9a77918a5348254/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51fd7415242216b0107d7b353c460edf886a0c33b3c69cf2f9a77918a5348254/userdata/shm major:0 minor:528 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5509ce5deaaa98fcfbfc9bdcc053afcdf3e9d5a0b23a82b025612edf8312b66f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5509ce5deaaa98fcfbfc9bdcc053afcdf3e9d5a0b23a82b025612edf8312b66f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57145a06bca89cdf364ce88792f350f4ffdcc3f29175c41550c42efd09a11b5f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57145a06bca89cdf364ce88792f350f4ffdcc3f29175c41550c42efd09a11b5f/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a509155bce896f34b6b640c82b27e12f25852c8d8b9e7a1b08f2fe8e6d257eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a509155bce896f34b6b640c82b27e12f25852c8d8b9e7a1b08f2fe8e6d257eb/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/662e550a5bcd247573f520b50a0974bb04261438fa03101b1d02bc6cd1b4cc6b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/662e550a5bcd247573f520b50a0974bb04261438fa03101b1d02bc6cd1b4cc6b/userdata/shm major:0 minor:355 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/674240409aea1984b262957e2a7b4b0ee6e9691b2ae3eca66fa5c6c3f997ac7f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/674240409aea1984b262957e2a7b4b0ee6e9691b2ae3eca66fa5c6c3f997ac7f/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74523671170fb1512036d6c47d3b5c4cd185b44d3dc90ff6412360d24a69690f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74523671170fb1512036d6c47d3b5c4cd185b44d3dc90ff6412360d24a69690f/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f9f87517e2657f52914494ba534311726391a06b3ac603e16c528bf27104a3d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f9f87517e2657f52914494ba534311726391a06b3ac603e16c528bf27104a3d/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/87f5d720e3b37e76574e95c4eae2cbc908611871a920c814d52e576eb107e365/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/87f5d720e3b37e76574e95c4eae2cbc908611871a920c814d52e576eb107e365/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/887131d414daa40def71cf695d9574d77d5b4a3be3393f668be6ceb815b5c0ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/887131d414daa40def71cf695d9574d77d5b4a3be3393f668be6ceb815b5c0ef/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9984b1a24abe3ef16d129e5956dacf85fdbde572d7642a1c35a8b2cd64cffe6d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9984b1a24abe3ef16d129e5956dacf85fdbde572d7642a1c35a8b2cd64cffe6d/userdata/shm major:0 minor:319 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b02d52421726ab2c0209a7de2746dae3311220b5a54e7b5d0dc202d5e063160/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b02d52421726ab2c0209a7de2746dae3311220b5a54e7b5d0dc202d5e063160/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a309fe29a795ecf640b84d9617014ca6b16a5866db895234b68de3d37d75632e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a309fe29a795ecf640b84d9617014ca6b16a5866db895234b68de3d37d75632e/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8474860be3e4da5daa7aa3ce533a8b0d1c67601203c301400fa1bcb32fdb559/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8474860be3e4da5daa7aa3ce533a8b0d1c67601203c301400fa1bcb32fdb559/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad6b0fb9aaffef859b91fc640e83dbf07b4c624dd9fe9a618643d5abc60ee82f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad6b0fb9aaffef859b91fc640e83dbf07b4c624dd9fe9a618643d5abc60ee82f/userdata/shm major:0 minor:411 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0fe7d2678f710a09c8cebcf7cc280d5afdb14fc46576208ec3f1cbeaacfacda/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0fe7d2678f710a09c8cebcf7cc280d5afdb14fc46576208ec3f1cbeaacfacda/userdata/shm major:0 minor:321 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d575a9e899a6ee915a4ab04da5ad50997b35b07410a6755aa4fcf3c7a98e9d11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d575a9e899a6ee915a4ab04da5ad50997b35b07410a6755aa4fcf3c7a98e9d11/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df31fdbf1ba4c585d7350c7c66fb65b142a3c06d2ba1133e28d4a93402d9c384/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df31fdbf1ba4c585d7350c7c66fb65b142a3c06d2ba1133e28d4a93402d9c384/userdata/shm major:0 minor:527 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dfe4d3f2f599f651419b79d4758312d379c6a971c82608c6aa290b494658d70c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dfe4d3f2f599f651419b79d4758312d379c6a971c82608c6aa290b494658d70c/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00710f2d-31bb-42da-9ae8-135ddc117df5/volumes/kubernetes.io~projected/kube-api-access-hljmh:{mountpoint:/var/lib/kubelet/pods/00710f2d-31bb-42da-9ae8-135ddc117df5/volumes/kubernetes.io~projected/kube-api-access-hljmh major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00c39778-f1f8-4539-88eb-40d082abdc50/volumes/kubernetes.io~projected/kube-api-access-c986b:{mountpoint:/var/lib/kubelet/pods/00c39778-f1f8-4539-88eb-40d082abdc50/volumes/kubernetes.io~projected/kube-api-access-c986b major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00c39778-f1f8-4539-88eb-40d082abdc50/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/00c39778-f1f8-4539-88eb-40d082abdc50/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00c39778-f1f8-4539-88eb-40d082abdc50/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/00c39778-f1f8-4539-88eb-40d082abdc50/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08770b22-f9ff-4919-af54-c7916eba9101/volumes/kubernetes.io~projected/kube-api-access-46p7r:{mountpoint:/var/lib/kubelet/pods/08770b22-f9ff-4919-af54-c7916eba9101/volumes/kubernetes.io~projected/kube-api-access-46p7r major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08770b22-f9ff-4919-af54-c7916eba9101/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/08770b22-f9ff-4919-af54-c7916eba9101/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08770b22-f9ff-4919-af54-c7916eba9101/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/08770b22-f9ff-4919-af54-c7916eba9101/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/117a0c38-48f5-430d-9efe-1a308f725309/volumes/kubernetes.io~projected/kube-api-access-l26gw:{mountpoint:/var/lib/kubelet/pods/117a0c38-48f5-430d-9efe-1a308f725309/volumes/kubernetes.io~projected/kube-api-access-l26gw major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/117a0c38-48f5-430d-9efe-1a308f725309/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/117a0c38-48f5-430d-9efe-1a308f725309/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/117a0c38-48f5-430d-9efe-1a308f725309/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/117a0c38-48f5-430d-9efe-1a308f725309/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/146e21dc-5041-463e-b0e7-9a79f969fa33/volumes/kubernetes.io~projected/kube-api-access-66cnr:{mountpoint:/var/lib/kubelet/pods/146e21dc-5041-463e-b0e7-9a79f969fa33/volumes/kubernetes.io~projected/kube-api-access-66cnr major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/146e21dc-5041-463e-b0e7-9a79f969fa33/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/146e21dc-5041-463e-b0e7-9a79f969fa33/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/146e21dc-5041-463e-b0e7-9a79f969fa33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/146e21dc-5041-463e-b0e7-9a79f969fa33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/146e21dc-5041-463e-b0e7-9a79f969fa33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/146e21dc-5041-463e-b0e7-9a79f969fa33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/146e21dc-5041-463e-b0e7-9a79f969fa33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/146e21dc-5041-463e-b0e7-9a79f969fa33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/146e21dc-5041-463e-b0e7-9a79f969fa33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/146e21dc-5041-463e-b0e7-9a79f969fa33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/146e21dc-5041-463e-b0e7-9a79f969fa33/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/146e21dc-5041-463e-b0e7-9a79f969fa33/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23a7781b-32fa-4f63-ab0e-663e1f2c6abb/volumes/kubernetes.io~projected/kube-api-access-857nt:{mountpoint:/var/lib/kubelet/pods/23a7781b-32fa-4f63-ab0e-663e1f2c6abb/volumes/kubernetes.io~projected/kube-api-access-857nt major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37080dcf-15b0-4d4e-85fa-ff669783f356/volumes/kubernetes.io~projected/kube-api-access-8l8qd:{mountpoint:/var/lib/kubelet/pods/37080dcf-15b0-4d4e-85fa-ff669783f356/volumes/kubernetes.io~projected/kube-api-access-8l8qd major:0 minor:526 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4af20ec7-7ea1-470f-b238-62e0de9ca527/volumes/kubernetes.io~projected/kube-api-access-jd5nd:{mountpoint:/var/lib/kubelet/pods/4af20ec7-7ea1-470f-b238-62e0de9ca527/volumes/kubernetes.io~projected/kube-api-access-jd5nd major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4af20ec7-7ea1-470f-b238-62e0de9ca527/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/4af20ec7-7ea1-470f-b238-62e0de9ca527/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4af20ec7-7ea1-470f-b238-62e0de9ca527/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/4af20ec7-7ea1-470f-b238-62e0de9ca527/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f2d426d-7ee0-48d9-b960-04f7fca5dc29/volumes/kubernetes.io~projected/kube-api-access-jklgn:{mountpoint:/var/lib/kubelet/pods/4f2d426d-7ee0-48d9-b960-04f7fca5dc29/volumes/kubernetes.io~projected/kube-api-access-jklgn major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f2d426d-7ee0-48d9-b960-04f7fca5dc29/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/4f2d426d-7ee0-48d9-b960-04f7fca5dc29/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f2d426d-7ee0-48d9-b960-04f7fca5dc29/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/4f2d426d-7ee0-48d9-b960-04f7fca5dc29/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f2d426d-7ee0-48d9-b960-04f7fca5dc29/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/4f2d426d-7ee0-48d9-b960-04f7fca5dc29/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/504075e3-50b2-4616-afea-70d49c4d7293/volumes/kubernetes.io~projected/kube-api-access-wfb4h:{mountpoint:/var/lib/kubelet/pods/504075e3-50b2-4616-afea-70d49c4d7293/volumes/kubernetes.io~projected/kube-api-access-wfb4h major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/504075e3-50b2-4616-afea-70d49c4d7293/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/504075e3-50b2-4616-afea-70d49c4d7293/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5923113e-7941-4bf8-bfde-28622a520e86/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/5923113e-7941-4bf8-bfde-28622a520e86/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d9f8f17-bb8e-4f09-ade6-099a78bfe422/volumes/kubernetes.io~projected/kube-api-access-r2wsj:{mountpoint:/var/lib/kubelet/pods/5d9f8f17-bb8e-4f09-ade6-099a78bfe422/volumes/kubernetes.io~projected/kube-api-access-r2wsj major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76f649c8-7dad-410e-a2e8-960e639099d5/volumes/kubernetes.io~projected/kube-api-access-6wr9f:{mountpoint:/var/lib/kubelet/pods/76f649c8-7dad-410e-a2e8-960e639099d5/volumes/kubernetes.io~projected/kube-api-access-6wr9f major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76f649c8-7dad-410e-a2e8-960e639099d5/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/76f649c8-7dad-410e-a2e8-960e639099d5/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76f649c8-7dad-410e-a2e8-960e639099d5/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/76f649c8-7dad-410e-a2e8-960e639099d5/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76f649c8-7dad-410e-a2e8-960e639099d5/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/76f649c8-7dad-410e-a2e8-960e639099d5/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76f649c8-7dad-410e-a2e8-960e639099d5/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/76f649c8-7dad-410e-a2e8-960e639099d5/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88482077-204d-4634-8045-c741d2c10d80/volumes/kubernetes.io~projected/kube-api-access-hl2pt:{mountpoint:/var/lib/kubelet/pods/88482077-204d-4634-8045-c741d2c10d80/volumes/kubernetes.io~projected/kube-api-access-hl2pt major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8906c21b-3233-4214-8ce1-f9189b79ccd3/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/8906c21b-3233-4214-8ce1-f9189b79ccd3/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8906c21b-3233-4214-8ce1-f9189b79ccd3/volumes/kubernetes.io~projected/kube-api-access-jpc58:{mountpoint:/var/lib/kubelet/pods/8906c21b-3233-4214-8ce1-f9189b79ccd3/volumes/kubernetes.io~projected/kube-api-access-jpc58 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8906c21b-3233-4214-8ce1-f9189b79ccd3/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/8906c21b-3233-4214-8ce1-f9189b79ccd3/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a607eca-e6ad-4e5d-aa57-663768133ec0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8a607eca-e6ad-4e5d-aa57-663768133ec0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a607eca-e6ad-4e5d-aa57-663768133ec0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8a607eca-e6ad-4e5d-aa57-663768133ec0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a607eca-e6ad-4e5d-aa57-663768133ec0/volumes/kubernetes.io~projected/kube-api-access-tz4kk:{mountpoint:/var/lib/kubelet/pods/8a607eca-e6ad-4e5d-aa57-663768133ec0/volumes/kubernetes.io~projected/kube-api-access-tz4kk major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350/volumes/kubernetes.io~projected/kube-api-access-cjrtj:{mountpoint:/var/lib/kubelet/pods/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350/volumes/kubernetes.io~projected/kube-api-access-cjrtj major:0 minor:407 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:406 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:401 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/902e0389-e70e-4d23-a607-d8f4b9da868d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/902e0389-e70e-4d23-a607-d8f4b9da868d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:521 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volume-subpaths/web-config/alertmanager/9 major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volume-subpaths/web-config/config-reloader/6 major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~empty-dir/config-out major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~projected/kube-api-access-5jp8z:{mountpoint:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~projected/kube-api-access-5jp8z major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~projected/tls-assets major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~secret/config-volume major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~secret/web-config major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f6cf431-dc3b-478e-81cd-97127a917de3/volumes/kubernetes.io~projected/kube-api-access-qn94x:{mountpoint:/var/lib/kubelet/pods/9f6cf431-dc3b-478e-81cd-97127a917de3/volumes/kubernetes.io~projected/kube-api-access-qn94x major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f6cf431-dc3b-478e-81cd-97127a917de3/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/9f6cf431-dc3b-478e-81cd-97127a917de3/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5b4340c-7e77-4d6e-bdc3-2a6193620f58/volumes/kubernetes.io~projected/kube-api-access-wlwvb:{mountpoint:/var/lib/kubelet/pods/b5b4340c-7e77-4d6e-bdc3-2a6193620f58/volumes/kubernetes.io~projected/kube-api-access-wlwvb major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5b4340c-7e77-4d6e-bdc3-2a6193620f58/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/b5b4340c-7e77-4d6e-bdc3-2a6193620f58/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd2a75ff-95c3-4b08-b75b-30059af6bd37/volumes/kubernetes.io~projected/kube-api-access-z5th7:{mountpoint:/var/lib/kubelet/pods/bd2a75ff-95c3-4b08-b75b-30059af6bd37/volumes/kubernetes.io~projected/kube-api-access-z5th7 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd2a75ff-95c3-4b08-b75b-30059af6bd37/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/bd2a75ff-95c3-4b08-b75b-30059af6bd37/volumes/kubernetes.io~secret/metrics-certs major:0 minor:525 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1e30c92-229f-4030-82eb-9295f5328da4/volumes/kubernetes.io~projected/kube-api-access-bmd9x:{mountpoint:/var/lib/kubelet/pods/c1e30c92-229f-4030-82eb-9295f5328da4/volumes/kubernetes.io~projected/kube-api-access-bmd9x major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e59c4c3e-b582-4b16-a6e3-4aadf0612a32/volumes/kubernetes.io~projected/kube-api-access-2kx94:{mountpoint:/var/lib/kubelet/pods/e59c4c3e-b582-4b16-a6e3-4aadf0612a32/volumes/kubernetes.io~projected/kube-api-access-2kx94 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5c5030e-696b-41dd-a96b-11c9eed11db2/volumes/kubernetes.io~projected/kube-api-access-n7jmf:{mountpoint:/var/lib/kubelet/pods/e5c5030e-696b-41dd-a96b-11c9eed11db2/volumes/kubernetes.io~projected/kube-api-access-n7jmf major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5c5030e-696b-41dd-a96b-11c9eed11db2/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/e5c5030e-696b-41dd-a96b-11c9eed11db2/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volume-subpaths/web-config/config-reloader/2 major:0 minor:464 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:464 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volume-subpaths/web-config/prometheus/15 major:0 minor:464 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~empty-dir/config-out major:0 minor:467 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~projected/kube-api-access-q4sxz:{mountpoint:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~projected/kube-api-access-q4sxz major:0 minor:472 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~projected/tls-assets major:0 minor:462 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/config major:0 minor:470 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:466 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:458 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:465 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:469 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:463 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:471 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:468 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/web-config major:0 minor:464 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/9fc816d6dd5e198fadb53f26e6d92f077143e02b9231325764115f50079c5b81/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d9a92c03248da3ff18a61975d3c1273bcaac38cb45bda9a6ccdc5619cde9421d/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/7d2c49adf1dcbd10b421bfb9d17f46162cbc98a6fe2db71a22f5f047af462a69/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/7ebb80afbb69554f29e7e4494603591cce8d14d5f7f5b4a73a76c84c85537364/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/70dad57f3a081714f215d363d0f46be6991f94c1a5345fefe18013652e3516be/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/e77933f1ef6c9a7cbcb2afacc8cdacaf63699c61047c2c4a6f80f16703eea8e6/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/98dfbcc8a489e08b275e6ba10836ea59e22c01f41ce277d5cfa315d2f6e47d85/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/988843251e3688004d4f5e28823edde6a4925e06e05013f8712489d4640cbb6e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/bec73e525c52c0f589320359da10af31b01b69b6ca1915179bd48e31aaf0500b/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/8f0ff4655d0dcd77bd93db096f05235dfe91621ced55619f87b8bbb71d189849/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/39f66e19a367907d390cdb65c588b6208f48929197427ae7c2040526237025c7/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/2ac4eaca84eb028cd4b321b4c7c77948c01a6f6fc10ff19fe158d5d18b4accbb/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/042c9fdd1198b684bb2e5ad8473c53bc43cc180a54e1b810ec346ee7631c5b20/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/f48c31dc67a4f2af67c8b807455635cc6b2fa0894c4a58e7216cb36cb76d3f61/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/266029283e6ff32c59a936542dec9e17118c1387a2a875ebe6256c568f80fbbc/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/9b15c176884fe902d5d3f9f305a6750284acad99fbe927071b826410656edcd2/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/cf79e3ca32918298a51f71f3aa1616272a6b9100980e7110683f3447545adbd6/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/fdf2b7a2dc65392fa96c14bff1967063a3e8fe2adb0d564f099c0e037548b1f8/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/02d4cfa9045a25e39459937f6619adbc1ef07113ac53c6155b2181d753967db1/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/a9e6a7601db44054b1f9715cccd2477b2d52c977bae1f0e8d818835f10a73af0/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/234d0ab00eb0c97e7f04025eb2dc44964a9745f2f2ea645a1d5010d942ce022c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/c055e8ca2285340c801e68d4efdebdd714e222fc4467965044f9d32bafc83e5d/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/bd3da87f7c8b4b29dace7787dc97005930764242402f89b3ce8c355dbcaa67b1/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/278eeb2874270890a92bd7c579c1ff3a5758ee4c9c4bee72a6609ca5d4a14216/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/9ef1af8015da61a0b25ea14c0b42f4b163cd6d656d159fc5269cf52e6c32f413/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/ff4a243cf6f792e630ed322aeb8cb40617bfefcca35c26719536ac553631a946/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/5180f9137c1a05f01edfebaf7658734132fde5501e6394e7623b64188fdc2553/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/708d9e8caff7bd9c32e32c1545e261d5336df2aa6b061f5c0d6c01c5d4fdea33/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/5070b501d31485491a2580cac0d25b20d345d4bef58b7b03c8504d5caf7974d3/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/987744ce22e8a9730342208fcc81812523b7105072bac9deb4483d06de4b80cd/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/b0217185b28ccc7ae21f800c03c57af2f649989de67868c5c2078fa2d5b5bffe/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/57300985e0cebb22f1251ece1a5351ab974835c908193fab475f7b28627f6f38/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/06201720596c022c7ff56c3dcc8870da93467d0098f5804f6ddf0e1b8fdfc746/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/44c921980894f3852c1f3b81f1068eb9b4b8239b168f81e1affa3065c54d57c2/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/778821cef4cc6bfcdd603d7629f3615fd3b586e4d4e44eae63a12bc20827d67b/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/cdda0f9a7e93de2f9fd5f35ab0f0a661757090e2896896bdf64367227057542b/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/ddaeff3f91bf3a9849ddeaaa81d3a78dcd7c4147d2a79df3a5793ec47131394b/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/8f90d2899e5e629745536c1ff8276b0f81912050e2a2529ace0f378520869d6a/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/bf05762cfdc33a9d4f0d9e7f68441c6dd8a7944023f0dca5010df3b1a11b7811/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/e93fd0c2a941b9a4827b0942e7cb71d8b30d8e95597de2917bb86f5fccc4294a/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/c45041fd3ffa5f59a43f0ea64aedff5c51096ccdb196943f4e29ed9c6d784a7b/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/9fdb5c6fbc78c5b2a91b0b9726720270987af129804174f3f9c1b64a7deaa1af/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/52d61d93d321dda02ae7be15ee3eda8eff79be00f31387ff165a5bbdbb83f767/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/f2f6fe259ec013111ae9c780c546f9e43f539ed002172631f65990572c12a5de/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/6340e4a3c14808c94db66e49dbf6dd6635d933ab29c44868892c74075f895538/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/5e4be1175bc31f67878e4d39eb787bc66f33997250487369c0e80e4be8a7f211/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/91cc4d8d2f01f863df8769ecace872e8496c1b3d657995d23b57abaacb269eb3/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/bb054ea1fe9edebc59d198923cd6f40f37ee7ddaf81bddd3e421c6b44e1c6a27/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/56cd1f62fa2cbac36fc5795fccc91037ec5f552e244531ce4a6b50e43db60744/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/56b758a59e43e7d17dd231d427c09bf4442a6dbfe57919839ff83ead22ad7900/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/57cd8c3207f5f889ea045c9dd7fa59d817212ac874f2555e712551ba0e559f54/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/e45a26060b64a1805030a70f7d9cffdd78416f91631bf9324ba284f36f5867c5/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/a8ee95de66aaaf00330fbd4120fd69f5cc6b6e8e346022fae881643187b3b74a/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/badaae77486cae90f8cd5a104f6b5d2e7c945779284868f62b7690583fd72d24/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/fb69d633fa2653691cea7d604371d0848e7b989299ec4733bdc5c97f1ff1e6ca/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/a4dfee8c5e798b9efb135d43d7500e7eaa15dfcf9546f9c988b045be2981b791/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/34624c7866afb0044289aec1f86ac53aadecbdb7a5a28acdecf6d8d09a7a5921/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-424:{mountpoint:/var/lib/containers/storage/overlay/77c3a48de5c1910ce0204f771088536532fe26fed0d6e7dee841c330800bf10a/merged major:0 minor:424 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/e3aa42a201e718d42ec54f3e188b58107639aa948f8dab4c4fb97689dfc543b0/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-428:{mountpoint:/var/lib/containers/storage/overlay/09e994d8e78ffecb0bc7a078854e3437c12d4d552443f4c64500a7cc8f9a1134/merged major:0 minor:428 fsType:overlay blockSize:0} overlay_0-438:{mountpoint:/var/lib/containers/storage/overlay/2446dc5d0437b6b4e68e3a06e557d1932e1d93b8958126e7662e300b153e40ba/merged major:0 minor:438 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/2bc592e296575c32900ba3c53ea2576d23c8cb0228e2646d2e3d57630ad084c2/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-448:{mountpoint:/var/lib/containers/storage/overlay/0baddebb01c7ffc67b87a625ada925a035689220752ee832d349a470ad03e8d7/merged major:0 minor:448 fsType:overlay blockSize:0} overlay_0-453:{mountpoint:/var/lib/containers/storage/overlay/14504b1a6c5b221c491faa92d1cf32d90db53b006b8e2603309d1d761de7c7d9/merged major:0 minor:453 fsType:overlay blockSize:0} overlay_0-473:{mountpoint:/var/lib/containers/storage/overlay/c45b6881c54af30f5ddf1fe12a103b9228b5b5dc31fefb2c803212b33d28574a/merged major:0 minor:473 fsType:overlay blockSize:0} overlay_0-481:{mountpoint:/var/lib/containers/storage/overlay/f84dbca23d971a316ff05d96386624a146dde7b4ebad3e8f351f5a31ca20243c/merged major:0 minor:481 fsType:overlay blockSize:0} overlay_0-483:{mountpoint:/var/lib/containers/storage/overlay/3ee9a6a735baa7dc2aa6849d4043cd0a8e162c30e9ddc6f966395f6be4430a49/merged major:0 minor:483 fsType:overlay blockSize:0} overlay_0-487:{mountpoint:/var/lib/containers/storage/overlay/9182f2b927bc1e5d8974f1fd50dd50f783033e13bb3f1ca53c234035851dac9c/merged major:0 minor:487 fsType:overlay blockSize:0} overlay_0-493:{mountpoint:/var/lib/containers/storage/overlay/80c07a8ecd015cba184a937a7fe40ea3f0d74ce7319273181dfdf8f3087c166b/merged major:0 minor:493 fsType:overlay blockSize:0} overlay_0-495:{mountpoint:/var/lib/containers/storage/overlay/fe72f0c1369c7096101ee8651b9881dcfadf7105731fb2107950a7fa785e9a75/merged major:0 minor:495 fsType:overlay blockSize:0} overlay_0-501:{mountpoint:/var/lib/containers/storage/overlay/faa217bda698a62a689959cef59cf8fe1b581d616b88956a398481ecd09c4ba5/merged major:0 minor:501 fsType:overlay blockSize:0} overlay_0-506:{mountpoint:/var/lib/containers/storage/overlay/636339a9bbe83f2e2187c46baf45a9fdcd5501bddcb449a2b49104138c8a134c/merged major:0 minor:506 fsType:overlay blockSize:0} overlay_0-511:{mountpoint:/var/lib/containers/storage/overlay/5b7c5951508e27a55695af9dea4f4b96a6ec38c9e69c02fcc7f4a7c44ae73e3c/merged major:0 minor:511 fsType:overlay blockSize:0} overlay_0-516:{mountpoint:/var/lib/containers/storage/overlay/fa2035818c805674ca400445a1bee786fbc88cb194bea445ceb3138af7035cfb/merged major:0 minor:516 fsType:overlay blockSize:0} overlay_0-533:{mountpoint:/var/lib/containers/storage/overlay/6b3b86a9897d2c7b25e08517ce588488e0687ad33222ef190a2de4ac1860eb07/merged major:0 minor:533 fsType:overlay blockSize:0} overlay_0-535:{mountpoint:/var/lib/containers/storage/overlay/f2c38ff7236e99c8fb7feb30cebcdbb4baed5b04a2273bdbee0aca16681f577c/merged major:0 minor:535 fsType:overlay blockSize:0} overlay_0-537:{mountpoint:/var/lib/containers/storage/overlay/ea89e47f0239e91ee5d329fc60f988fefe56ce2644722fecddcccce9d7c773c6/merged major:0 minor:537 fsType:overlay blockSize:0} overlay_0-539:{mountpoint:/var/lib/containers/storage/overlay/2d7d44fc5f17a4f9fdd43637cf51d7eb6fdb67f80f5cfc54d86e43a7e6a2c913/merged major:0 minor:539 fsType:overlay blockSize:0} overlay_0-541:{mountpoint:/var/lib/containers/storage/overlay/04c7b6d85511ffe58ad5e9e2d87f00fb11e69c752428a0f3f6597a990d4fabbc/merged major:0 minor:541 fsType:overlay blockSize:0} overlay_0-547:{mountpoint:/var/lib/containers/storage/overlay/cb9c2fe6f5c59ad4331aad7ca1bb62c4dfc340a553096dc360356e7b6180f38d/merged major:0 minor:547 fsType:overlay blockSize:0} overlay_0-552:{mountpoint:/var/lib/containers/storage/overlay/998ae5364c84e6aab2950dc8b09cfff14c5088dcb3fafee6f177bbfe03a7b661/merged major:0 minor:552 fsType:overlay blockSize:0} overlay_0-63:{mountpoint:/var/lib/containers/storage/overlay/e56a90ffbd2a787aad5510cf65861ff8d1f7bb450c6d8480b116feb3112d3a07/merged major:0 minor:63 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/3b8b361697c824f24721c09c85d6ef96029a8dd731588d2d2abac15b443a8aa2/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/0bb6823aea93a02a0b9065f06ff34e57fad54b00ff0d6085e4c8faa30375308b/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c120ac091b7aad975b9177fa0878b1dfef0ce00db9ca85458211026e98fac4d9/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/91dd1e22237e0dd0167f572722c1466b04cacb14c0e9cab592c64b74fb1aad8d/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/774ac72f5b0fcac8a3dee7edd942344dd5754ed3bf1bd9a070f162c7e0484089/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/10c83848f026160a064b98df99bd54dd79144174ff9440d8dc11ff886524b696/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/d4d7c98cd49d698bebfee6b98fcf9dda8283d7654bc00c465ba5b1a3c7c0f75d/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/33cd9e5e1225e76abd7e1033571796aa9a45b22e36a1269b77d4e67d49721667/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/3aee294cfbbd4255de092e6522d25ac22921112c8c6822656b0fc0e67be84695/merged major:0 minor:98 fsType:overlay blockSize:0}]
Nov 03 17:17:39.275870 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.275290    6059 manager.go:217] Machine: {Timestamp:2026-11-03 17:17:39.274817895 +0000 UTC m=+0.072530667 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100126 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ccaacf930ee484cd321e8f630becd SystemUUID:ec2ccaac-f930-ee48-4cd3-21e8f630becd BootID:a7c7c148-c3c6-40c2-bac8-029ad736844b Filesystems:[{Device:/run/containers/storage/overlay-containers/1e25b5f52e685e6c8026025d53999cc07360830d7380db2ba5a109782deede16/userdata/shm DeviceMajor:0 DeviceMinor:480 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-493 DeviceMajor:0 DeviceMinor:493 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-516 DeviceMajor:0 DeviceMinor:516 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-533 DeviceMajor:0 DeviceMinor:533 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:301 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d575a9e899a6ee915a4ab04da5ad50997b35b07410a6755aa4fcf3c7a98e9d11/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/38d59c19828a6bacf51ada0988f5e0ec2477f8d6fa87199490886da6f1bf6528/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/87f5d720e3b37e76574e95c4eae2cbc908611871a920c814d52e576eb107e365/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f6cf431-dc3b-478e-81cd-97127a917de3/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b5b4340c-7e77-4d6e-bdc3-2a6193620f58/volumes/kubernetes.io~projected/kube-api-access-wlwvb DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-481 DeviceMajor:0 DeviceMinor:481 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/00c39778-f1f8-4539-88eb-40d082abdc50/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:310 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/76f649c8-7dad-410e-a2e8-960e639099d5/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:360 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0fb2a4c0997232dfde0191c1496a54041963ac981c8993a7d7b4c39f4904085f/userdata/shm DeviceMajor:0 DeviceMinor:257 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74523671170fb1512036d6c47d3b5c4cd185b44d3dc90ff6412360d24a69690f/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5509ce5deaaa98fcfbfc9bdcc053afcdf3e9d5a0b23a82b025612edf8312b66f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/146e21dc-5041-463e-b0e7-9a79f969fa33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4f2d426d-7ee0-48d9-b960-04f7fca5dc29/volumes/kubernetes.io~projected/kube-api-access-jklgn DeviceMajor:0 DeviceMinor:351 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08770b22-f9ff-4919-af54-c7916eba9101/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:253 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:294 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/662e550a5bcd247573f520b50a0974bb04261438fa03101b1d02bc6cd1b4cc6b/userdata/shm DeviceMajor:0 DeviceMinor:355 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:465 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:470 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:464 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-473 DeviceMajor:0 DeviceMinor:473 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-541 DeviceMajor:0 DeviceMinor:541 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4af20ec7-7ea1-470f-b238-62e0de9ca527/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:256 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b02d52421726ab2c0209a7de2746dae3311220b5a54e7b5d0dc202d5e063160/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/00c39778-f1f8-4539-88eb-40d082abdc50/volumes/kubernetes.io~projected/kube-api-access-c986b DeviceMajor:0 DeviceMinor:311 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:466 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-511 DeviceMajor:0 DeviceMinor:511 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51fd7415242216b0107d7b353c460edf886a0c33b3c69cf2f9a77918a5348254/userdata/shm DeviceMajor:0 DeviceMinor:528 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-539 DeviceMajor:0 DeviceMinor:539 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57145a06bca89cdf364ce88792f350f4ffdcc3f29175c41550c42efd09a11b5f/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a509155bce896f34b6b640c82b27e12f25852c8d8b9e7a1b08f2fe8e6d257eb/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/674240409aea1984b262957e2a7b4b0ee6e9691b2ae3eca66fa5c6c3f997ac7f/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-448 DeviceMajor:0 DeviceMinor:448 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5c5030e-696b-41dd-a96b-11c9eed11db2/volumes/kubernetes.io~projected/kube-api-access-n7jmf DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/08770b22-f9ff-4919-af54-c7916eba9101/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:254 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:300 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:458 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:471 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-487 DeviceMajor:0 DeviceMinor:487 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1e30c92-229f-4030-82eb-9295f5328da4/volumes/kubernetes.io~projected/kube-api-access-bmd9x DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:463 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/902e0389-e70e-4d23-a607-d8f4b9da868d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:521 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1cc1301f076c1e9e0bd08fbed459163f220e07617f012d466e4de555cc41a6c1/userdata/shm DeviceMajor:0 DeviceMinor:531 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1697bed59d554b35eb3bcdf037c62cf04db88502c215f7b4275cc6cbb4f0fcf7/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5c5030e-696b-41dd-a96b-11c9eed11db2/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:467 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f6cf431-dc3b-478e-81cd-97127a917de3/volumes/kubernetes.io~projected/kube-api-access-qn94x DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/146e21dc-5041-463e-b0e7-9a79f969fa33/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:317 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/504075e3-50b2-4616-afea-70d49c4d7293/volumes/kubernetes.io~projected/kube-api-access-wfb4h DeviceMajor:0 DeviceMinor:353 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/117a0c38-48f5-430d-9efe-1a308f725309/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/117a0c38-48f5-430d-9efe-1a308f725309/volumes/kubernetes.io~projected/kube-api-access-l26gw DeviceMajor:0 DeviceMinor:252 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4af20ec7-7ea1-470f-b238-62e0de9ca527/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:255 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/08770b22-f9ff-4919-af54-c7916eba9101/volumes/kubernetes.io~projected/kube-api-access-46p7r DeviceMajor:0 DeviceMinor:260 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f9f87517e2657f52914494ba534311726391a06b3ac603e16c528bf27104a3d/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/146e21dc-5041-463e-b0e7-9a79f969fa33/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:316 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/887131d414daa40def71cf695d9574d77d5b4a3be3393f668be6ceb815b5c0ef/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8906c21b-3233-4214-8ce1-f9189b79ccd3/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a309fe29a795ecf640b84d9617014ca6b16a5866db895234b68de3d37d75632e/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/76f649c8-7dad-410e-a2e8-960e639099d5/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:361 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:401 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-424 DeviceMajor:0 DeviceMinor:424 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:469 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/37080dcf-15b0-4d4e-85fa-ff669783f356/volumes/kubernetes.io~projected/kube-api-access-8l8qd DeviceMajor:0 DeviceMinor:526 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/8a607eca-e6ad-4e5d-aa57-663768133ec0/volumes/kubernetes.io~projected/kube-api-access-tz4kk DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8474860be3e4da5daa7aa3ce533a8b0d1c67601203c301400fa1bcb32fdb559/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-438 DeviceMajor:0 DeviceMinor:438 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:302 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/00c39778-f1f8-4539-88eb-40d082abdc50/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:309 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/88482077-204d-4634-8045-c741d2c10d80/volumes/kubernetes.io~projected/kube-api-access-hl2pt DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e59c4c3e-b582-4b16-a6e3-4aadf0612a32/volumes/kubernetes.io~projected/kube-api-access-2kx94 DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:296 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:297 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/146e21dc-5041-463e-b0e7-9a79f969fa33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:315 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0fe7d2678f710a09c8cebcf7cc280d5afdb14fc46576208ec3f1cbeaacfacda/userdata/shm DeviceMajor:0 DeviceMinor:321 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76f649c8-7dad-410e-a2e8-960e639099d5/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:359 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f23729abf07279979b282fbc3eca67fc51145fe2438609d20644cb97cb197a1/userdata/shm DeviceMajor:0 DeviceMinor:364 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/00710f2d-31bb-42da-9ae8-135ddc117df5/volumes/kubernetes.io~projected/kube-api-access-hljmh DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-483 DeviceMajor:0 DeviceMinor:483 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:464 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9984b1a24abe3ef16d129e5956dacf85fdbde572d7642a1c35a8b2cd64cffe6d/userdata/shm DeviceMajor:0 DeviceMinor:319 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350/volumes/kubernetes.io~projected/kube-api-access-cjrtj DeviceMajor:0 DeviceMinor:407 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bd2a75ff-95c3-4b08-b75b-30059af6bd37/volumes/kubernetes.io~projected/kube-api-access-z5th7 DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/23a7781b-32fa-4f63-ab0e-663e1f2c6abb/volumes/kubernetes.io~projected/kube-api-access-857nt DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~projected/kube-api-access-5jp8z DeviceMajor:0 DeviceMinor:303 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad6b0fb9aaffef859b91fc640e83dbf07b4c624dd9fe9a618643d5abc60ee82f/userdata/shm DeviceMajor:0 DeviceMinor:411 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:468 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8a607eca-e6ad-4e5d-aa57-663768133ec0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5923113e-7941-4bf8-bfde-28622a520e86/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/106da788c38a5f644bcb471f6573512cea9bc0dc229b82880ddb8dfde42b5f3e/userdata/shm DeviceMajor:0 DeviceMinor:307 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:464 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~projected/kube-api-access-q4sxz DeviceMajor:0 DeviceMinor:472 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:464 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df31fdbf1ba4c585d7350c7c66fb65b142a3c06d2ba1133e28d4a93402d9c384/userdata/shm DeviceMajor:0 DeviceMinor:527 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8a607eca-e6ad-4e5d-aa57-663768133ec0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5d9f8f17-bb8e-4f09-ade6-099a78bfe422/volumes/kubernetes.io~projected/kube-api-access-r2wsj DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4f2d426d-7ee0-48d9-b960-04f7fca5dc29/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:349 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-535 DeviceMajor:0 DeviceMinor:535 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4af20ec7-7ea1-470f-b238-62e0de9ca527/volumes/kubernetes.io~projected/kube-api-access-jd5nd DeviceMajor:0 DeviceMinor:259 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/146e21dc-5041-463e-b0e7-9a79f969fa33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:313 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4f2d426d-7ee0-48d9-b960-04f7fca5dc29/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:348 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-453 DeviceMajor:0 DeviceMinor:453 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-506 DeviceMajor:0 DeviceMinor:506 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd2a75ff-95c3-4b08-b75b-30059af6bd37/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:525 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-63 DeviceMajor:0 DeviceMinor:63 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-501 DeviceMajor:0 DeviceMinor:501 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-547 DeviceMajor:0 DeviceMinor:547 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dfe4d3f2f599f651419b79d4758312d379c6a971c82608c6aa290b494658d70c/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:462 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/117a0c38-48f5-430d-9efe-1a308f725309/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:247 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/146e21dc-5041-463e-b0e7-9a79f969fa33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:314 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/76f649c8-7dad-410e-a2e8-960e639099d5/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:362 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:406 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-495 DeviceMajor:0 DeviceMinor:495 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-552 DeviceMajor:0 DeviceMinor:552 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8906c21b-3233-4214-8ce1-f9189b79ccd3/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/388325401b9403f65e71e7f80c658c2e479db62a04148a3bda6cf565cc0dee0a/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/504075e3-50b2-4616-afea-70d49c4d7293/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:340 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4f2d426d-7ee0-48d9-b960-04f7fca5dc29/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:350 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-428 DeviceMajor:0 DeviceMinor:428 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8906c21b-3233-4214-8ce1-f9189b79ccd3/volumes/kubernetes.io~projected/kube-api-access-jpc58 DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5b4340c-7e77-4d6e-bdc3-2a6193620f58/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/146e21dc-5041-463e-b0e7-9a79f969fa33/volumes/kubernetes.io~projected/kube-api-access-66cnr DeviceMajor:0 DeviceMinor:318 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/76f649c8-7dad-410e-a2e8-960e639099d5/volumes/kubernetes.io~projected/kube-api-access-6wr9f DeviceMajor:0 DeviceMinor:363 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-537 DeviceMajor:0 DeviceMinor:537 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:298 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0fb2a4c0997232d MacAddress:ae:d8:71:d1:96:a0 Speed:10000 Mtu:8901} {Name:106da788c38a5f6 MacAddress:92:81:44:42:5c:d6 Speed:10000 Mtu:8901} {Name:1cc1301f076c1e9 MacAddress:82:83:5a:82:d4:f2 Speed:10000 Mtu:8901} {Name:1e25b5f52e685e6 MacAddress:aa:b0:b7:75:3f:56 Speed:10000 Mtu:8901} {Name:388325401b9403f MacAddress:fa:01:45:65:19:69 Speed:10000 Mtu:8901} {Name:4f23729abf07279 MacAddress:36:b4:0d:c4:94:ff Speed:10000 Mtu:8901} {Name:51fd7415242216b MacAddress:d6:ad:d9:85:b9:09 Speed:10000 Mtu:8901} {Name:662e550a5bcd247 MacAddress:ea:90:e3:a2:50:55 Speed:10000 Mtu:8901} {Name:674240409aea198 MacAddress:ae:1a:35:17:bb:34 Speed:10000 Mtu:8901} {Name:887131d414daa40 MacAddress:9e:5f:36:f7:30:a0 Speed:10000 Mtu:8901} {Name:9984b1a24abe3ef MacAddress:6a:88:d6:1d:de:00 Speed:10000 Mtu:8901} {Name:9b02d52421726ab MacAddress:76:d9:80:fd:4b:b8 Speed:10000 Mtu:8901} {Name:a8474860be3e4da MacAddress:76:de:db:c1:71:0c Speed:10000 Mtu:8901} {Name:ad6b0fb9aaffef8 MacAddress:de:b8:60:5e:07:af Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d9:0c:5c:37 Speed:0 Mtu:9001} {Name:br-int MacAddress:6a:d9:57:57:f5:65 Speed:0 Mtu:8901} {Name:c0fe7d2678f710a MacAddress:9e:86:ca:63:46:ae Speed:10000 Mtu:8901} {Name:df31fdbf1ba4c58 MacAddress:02:40:ab:57:f6:93 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:d9:0c:5c:37 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:72:cc:be:68:b1:a3 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7a:e7:26:dd:08:6d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 17:17:39.275870 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.275875    6059 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 17:17:39.276330 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.275932    6059 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 17:17:39.276483 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.276441    6059 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 17:17:39.276664 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.276484    6059 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-117.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 17:17:39.276746 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.276669    6059 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 17:17:39.276746 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.276682    6059 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 17:17:39.276746 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.276698    6059 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 17:17:39.276746 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.276727    6059 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 17:17:39.276908 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.276885    6059 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 17:17:39.277150 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.277134    6059 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 17:17:39.277284 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.277272    6059 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 17:17:39.277351 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.277326    6059 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 17:17:39.277351 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.277342    6059 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 17:17:39.277468 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.277398    6059 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 17:17:39.277468 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.277428    6059 kubelet.go:405] "Adding apiserver pod source"
Nov 03 17:17:39.277468 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.277442    6059 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 17:17:39.278563 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.278545    6059 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 17:17:39.278563 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.278561    6059 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 17:17:39.279297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.279277    6059 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 17:17:39.282221 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.282204    6059 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 17:17:39.282221 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.282221    6059 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 17:17:39.282327 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.282317    6059 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 17:17:39.282531 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.282520    6059 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 17:17:39.282575 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.282549    6059 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 17:17:39.282648 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.282638    6059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 17:17:39.282683 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.282654    6059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 17:17:39.282683 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.282661    6059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 17:17:39.282683 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.282667    6059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 17:17:39.282683 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.282680    6059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 17:17:39.282785 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.282686    6059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 17:17:39.282785 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.282692    6059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 17:17:39.282785 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.282697    6059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 17:17:39.282785 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.282704    6059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 17:17:39.282785 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.282710    6059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 17:17:39.282785 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.282722    6059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 17:17:39.282785 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.282731    6059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 17:17:39.282785 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.282754    6059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 17:17:39.282785 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.282760    6059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 17:17:39.286229 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.286214    6059 server.go:1263] "Started kubelet"
Nov 03 17:17:39.286384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.286306    6059 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 17:17:39.286446 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.286378    6059 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 17:17:39.286494 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.286470    6059 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 17:17:39.286846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.286825    6059 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 17:17:39.286845 ip-10-0-143-117 systemd[1]: Started Kubernetes Kubelet.
Nov 03 17:17:39.292270 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.292243    6059 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 17:17:39.292937 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.292916    6059 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.293083    6059 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.293169    6059 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:11:26 +0000 UTC" deadline="2028-06-28 07:33:42.239972476 +0000 UTC"
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.293192    6059 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14462h16m2.946783788s"
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:39.293251    6059 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-117.us-west-2.compute.internal\" not found"
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.293258    6059 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.293283    6059 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.293380    6059 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.294404    6059 factory.go:55] Registering systemd factory
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.294422    6059 factory.go:223] Registration of the systemd container factory successfully
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.294717    6059 factory.go:153] Registering CRI-O factory
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.294738    6059 factory.go:223] Registration of the crio container factory successfully
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.294803    6059 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.294838    6059 factory.go:103] Registering Raw factory
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.294853    6059 manager.go:1199] Started watching for new ooms in manager
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.295162    6059 manager.go:319] Starting recovery of all containers
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297372    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00710f2d-31bb-42da-9ae8-135ddc117df5" volumeName="kubernetes.io/configmap/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-daemon-config" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297420    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f2d426d-7ee0-48d9-b960-04f7fca5dc29" volumeName="kubernetes.io/secret/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-secret-metrics-server-tls" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297434    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8906c21b-3233-4214-8ce1-f9189b79ccd3" volumeName="kubernetes.io/configmap/8906c21b-3233-4214-8ce1-f9189b79ccd3-ovnkube-config" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297458    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd2a75ff-95c3-4b08-b75b-30059af6bd37" volumeName="kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297471    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e59c4c3e-b582-4b16-a6e3-4aadf0612a32" volumeName="kubernetes.io/empty-dir/e59c4c3e-b582-4b16-a6e3-4aadf0612a32-tmp-dir" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297483    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f2d426d-7ee0-48d9-b960-04f7fca5dc29" volumeName="kubernetes.io/configmap/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297497    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9169e62e-57ab-4322-a0cc-664f54e48556" volumeName="kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297508    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f6cf431-dc3b-478e-81cd-97127a917de3" volumeName="kubernetes.io/secret/9f6cf431-dc3b-478e-81cd-97127a917de3-cert" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297523    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="146e21dc-5041-463e-b0e7-9a79f969fa33" volumeName="kubernetes.io/projected/146e21dc-5041-463e-b0e7-9a79f969fa33-kube-api-access-66cnr" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297535    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="146e21dc-5041-463e-b0e7-9a79f969fa33" volumeName="kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297547    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4af20ec7-7ea1-470f-b238-62e0de9ca527" volumeName="kubernetes.io/secret/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297559    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a607eca-e6ad-4e5d-aa57-663768133ec0" volumeName="kubernetes.io/empty-dir/8a607eca-e6ad-4e5d-aa57-663768133ec0-tmp" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297571    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46ad168-4931-41e7-ac08-f6fa965721f9" volumeName="kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297582    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46ad168-4931-41e7-ac08-f6fa965721f9" volumeName="kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297597    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="146e21dc-5041-463e-b0e7-9a79f969fa33" volumeName="kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297613    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5c5030e-696b-41dd-a96b-11c9eed11db2" volumeName="kubernetes.io/secret/e5c5030e-696b-41dd-a96b-11c9eed11db2-metrics-tls" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297624    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46ad168-4931-41e7-ac08-f6fa965721f9" volumeName="kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297637    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9169e62e-57ab-4322-a0cc-664f54e48556" volumeName="kubernetes.io/configmap/9169e62e-57ab-4322-a0cc-664f54e48556-metrics-client-ca" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297649    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46ad168-4931-41e7-ac08-f6fa965721f9" volumeName="kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297661    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00710f2d-31bb-42da-9ae8-135ddc117df5" volumeName="kubernetes.io/projected/00710f2d-31bb-42da-9ae8-135ddc117df5-kube-api-access-hljmh" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297674    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8906c21b-3233-4214-8ce1-f9189b79ccd3" volumeName="kubernetes.io/projected/8906c21b-3233-4214-8ce1-f9189b79ccd3-kube-api-access-jpc58" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297696    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bfb6a7d-e8f9-434f-acfc-8ac3badf1350" volumeName="kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297708    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9169e62e-57ab-4322-a0cc-664f54e48556" volumeName="kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-web-config" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297731    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46ad168-4931-41e7-ac08-f6fa965721f9" volumeName="kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-config" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297745    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bfb6a7d-e8f9-434f-acfc-8ac3badf1350" volumeName="kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297756    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46ad168-4931-41e7-ac08-f6fa965721f9" volumeName="kubernetes.io/projected/f46ad168-4931-41e7-ac08-f6fa965721f9-kube-api-access-q4sxz" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297770    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23a7781b-32fa-4f63-ab0e-663e1f2c6abb" volumeName="kubernetes.io/projected/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-kube-api-access-857nt" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297789    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88482077-204d-4634-8045-c741d2c10d80" volumeName="kubernetes.io/projected/88482077-204d-4634-8045-c741d2c10d80-kube-api-access-hl2pt" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297808    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9169e62e-57ab-4322-a0cc-664f54e48556" volumeName="kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-config-volume" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297820    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46ad168-4931-41e7-ac08-f6fa965721f9" volumeName="kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-web-config" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297833    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00c39778-f1f8-4539-88eb-40d082abdc50" volumeName="kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297844    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="146e21dc-5041-463e-b0e7-9a79f969fa33" volumeName="kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-grpc-tls" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297856    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76f649c8-7dad-410e-a2e8-960e639099d5" volumeName="kubernetes.io/configmap/76f649c8-7dad-410e-a2e8-960e639099d5-telemeter-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297869    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8906c21b-3233-4214-8ce1-f9189b79ccd3" volumeName="kubernetes.io/configmap/8906c21b-3233-4214-8ce1-f9189b79ccd3-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297881    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a607eca-e6ad-4e5d-aa57-663768133ec0" volumeName="kubernetes.io/empty-dir/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-tuned" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297892    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9169e62e-57ab-4322-a0cc-664f54e48556" volumeName="kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297906    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="117a0c38-48f5-430d-9efe-1a308f725309" volumeName="kubernetes.io/secret/117a0c38-48f5-430d-9efe-1a308f725309-openshift-state-metrics-tls" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297941    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f2d426d-7ee0-48d9-b960-04f7fca5dc29" volumeName="kubernetes.io/empty-dir/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-audit-log" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297952    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f2d426d-7ee0-48d9-b960-04f7fca5dc29" volumeName="kubernetes.io/secret/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297965    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76f649c8-7dad-410e-a2e8-960e639099d5" volumeName="kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297976    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5b4340c-7e77-4d6e-bdc3-2a6193620f58" volumeName="kubernetes.io/projected/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-kube-api-access-wlwvb" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.297988    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46ad168-4931-41e7-ac08-f6fa965721f9" volumeName="kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.298001    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="146e21dc-5041-463e-b0e7-9a79f969fa33" volumeName="kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.298012    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23a7781b-32fa-4f63-ab0e-663e1f2c6abb" volumeName="kubernetes.io/configmap/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.298028    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76f649c8-7dad-410e-a2e8-960e639099d5" volumeName="kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-secret-telemeter-client" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.298039    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88482077-204d-4634-8045-c741d2c10d80" volumeName="kubernetes.io/configmap/88482077-204d-4634-8045-c741d2c10d80-iptables-alerter-script" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.298052    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00c39778-f1f8-4539-88eb-40d082abdc50" volumeName="kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert" seLinuxMountContext=""
Nov 03 17:17:39.298089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.298063    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9169e62e-57ab-4322-a0cc-664f54e48556" volumeName="kubernetes.io/empty-dir/9169e62e-57ab-4322-a0cc-664f54e48556-config-out" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.300593    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5b4340c-7e77-4d6e-bdc3-2a6193620f58" volumeName="kubernetes.io/secret/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301203    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46ad168-4931-41e7-ac08-f6fa965721f9" volumeName="kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-metrics-client-certs" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301222    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08770b22-f9ff-4919-af54-c7916eba9101" volumeName="kubernetes.io/projected/08770b22-f9ff-4919-af54-c7916eba9101-kube-api-access-46p7r" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301236    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="117a0c38-48f5-430d-9efe-1a308f725309" volumeName="kubernetes.io/projected/117a0c38-48f5-430d-9efe-1a308f725309-kube-api-access-l26gw" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301254    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f2d426d-7ee0-48d9-b960-04f7fca5dc29" volumeName="kubernetes.io/projected/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-kube-api-access-jklgn" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301268    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76f649c8-7dad-410e-a2e8-960e639099d5" volumeName="kubernetes.io/projected/76f649c8-7dad-410e-a2e8-960e639099d5-kube-api-access-6wr9f" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301286    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46ad168-4931-41e7-ac08-f6fa965721f9" volumeName="kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-grpc-tls" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301299    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08770b22-f9ff-4919-af54-c7916eba9101" volumeName="kubernetes.io/secret/08770b22-f9ff-4919-af54-c7916eba9101-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301336    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46ad168-4931-41e7-ac08-f6fa965721f9" volumeName="kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-tls" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301389    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37080dcf-15b0-4d4e-85fa-ff669783f356" volumeName="kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301405    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4af20ec7-7ea1-470f-b238-62e0de9ca527" volumeName="kubernetes.io/configmap/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301418    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f2d426d-7ee0-48d9-b960-04f7fca5dc29" volumeName="kubernetes.io/configmap/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301434    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76f649c8-7dad-410e-a2e8-960e639099d5" volumeName="kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-telemeter-client-tls" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301447    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8906c21b-3233-4214-8ce1-f9189b79ccd3" volumeName="kubernetes.io/configmap/8906c21b-3233-4214-8ce1-f9189b79ccd3-env-overrides" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301465    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bfb6a7d-e8f9-434f-acfc-8ac3badf1350" volumeName="kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301477    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46ad168-4931-41e7-ac08-f6fa965721f9" volumeName="kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301495    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23a7781b-32fa-4f63-ab0e-663e1f2c6abb" volumeName="kubernetes.io/configmap/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301518    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8906c21b-3233-4214-8ce1-f9189b79ccd3" volumeName="kubernetes.io/secret/8906c21b-3233-4214-8ce1-f9189b79ccd3-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301531    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1e30c92-229f-4030-82eb-9295f5328da4" volumeName="kubernetes.io/projected/c1e30c92-229f-4030-82eb-9295f5328da4-kube-api-access-bmd9x" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301547    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46ad168-4931-41e7-ac08-f6fa965721f9" volumeName="kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-thanos-prometheus-http-client-file" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301563    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23a7781b-32fa-4f63-ab0e-663e1f2c6abb" volumeName="kubernetes.io/configmap/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-cni-binary-copy" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301579    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4af20ec7-7ea1-470f-b238-62e0de9ca527" volumeName="kubernetes.io/empty-dir/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-textfile" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301591    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d9f8f17-bb8e-4f09-ade6-099a78bfe422" volumeName="kubernetes.io/configmap/5d9f8f17-bb8e-4f09-ade6-099a78bfe422-serviceca" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301603    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9169e62e-57ab-4322-a0cc-664f54e48556" volumeName="kubernetes.io/empty-dir/9169e62e-57ab-4322-a0cc-664f54e48556-alertmanager-main-db" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301621    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e59c4c3e-b582-4b16-a6e3-4aadf0612a32" volumeName="kubernetes.io/projected/e59c4c3e-b582-4b16-a6e3-4aadf0612a32-kube-api-access-2kx94" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301637    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00710f2d-31bb-42da-9ae8-135ddc117df5" volumeName="kubernetes.io/configmap/00710f2d-31bb-42da-9ae8-135ddc117df5-cni-binary-copy" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301653    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08770b22-f9ff-4919-af54-c7916eba9101" volumeName="kubernetes.io/configmap/08770b22-f9ff-4919-af54-c7916eba9101-metrics-client-ca" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301665    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9169e62e-57ab-4322-a0cc-664f54e48556" volumeName="kubernetes.io/projected/9169e62e-57ab-4322-a0cc-664f54e48556-kube-api-access-5jp8z" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301676    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9169e62e-57ab-4322-a0cc-664f54e48556" volumeName="kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301692    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f6cf431-dc3b-478e-81cd-97127a917de3" volumeName="kubernetes.io/projected/9f6cf431-dc3b-478e-81cd-97127a917de3-kube-api-access-qn94x" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301705    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00c39778-f1f8-4539-88eb-40d082abdc50" volumeName="kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301723    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="504075e3-50b2-4616-afea-70d49c4d7293" volumeName="kubernetes.io/projected/504075e3-50b2-4616-afea-70d49c4d7293-kube-api-access-wfb4h" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301735    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5923113e-7941-4bf8-bfde-28622a520e86" volumeName="kubernetes.io/configmap/5923113e-7941-4bf8-bfde-28622a520e86-konnectivity-ca" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301748    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bfb6a7d-e8f9-434f-acfc-8ac3badf1350" volumeName="kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301766    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902e0389-e70e-4d23-a607-d8f4b9da868d" volumeName="kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301777    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9169e62e-57ab-4322-a0cc-664f54e48556" volumeName="kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301793    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5c5030e-696b-41dd-a96b-11c9eed11db2" volumeName="kubernetes.io/projected/e5c5030e-696b-41dd-a96b-11c9eed11db2-kube-api-access-n7jmf" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301806    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46ad168-4931-41e7-ac08-f6fa965721f9" volumeName="kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301818    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76f649c8-7dad-410e-a2e8-960e639099d5" volumeName="kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-federate-client-tls" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301835    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46ad168-4931-41e7-ac08-f6fa965721f9" volumeName="kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301847    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="117a0c38-48f5-430d-9efe-1a308f725309" volumeName="kubernetes.io/configmap/117a0c38-48f5-430d-9efe-1a308f725309-metrics-client-ca" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301864    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="146e21dc-5041-463e-b0e7-9a79f969fa33" volumeName="kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301877    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4af20ec7-7ea1-470f-b238-62e0de9ca527" volumeName="kubernetes.io/projected/4af20ec7-7ea1-470f-b238-62e0de9ca527-kube-api-access-jd5nd" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301889    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f2d426d-7ee0-48d9-b960-04f7fca5dc29" volumeName="kubernetes.io/secret/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-client-ca-bundle" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301905    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bfb6a7d-e8f9-434f-acfc-8ac3badf1350" volumeName="kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301917    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46ad168-4931-41e7-ac08-f6fa965721f9" volumeName="kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-metrics-client-ca" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301933    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bfb6a7d-e8f9-434f-acfc-8ac3badf1350" volumeName="kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301945    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bfb6a7d-e8f9-434f-acfc-8ac3badf1350" volumeName="kubernetes.io/empty-dir/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-tmp" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301958    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd2a75ff-95c3-4b08-b75b-30059af6bd37" volumeName="kubernetes.io/projected/bd2a75ff-95c3-4b08-b75b-30059af6bd37-kube-api-access-z5th7" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301976    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46ad168-4931-41e7-ac08-f6fa965721f9" volumeName="kubernetes.io/projected/f46ad168-4931-41e7-ac08-f6fa965721f9-tls-assets" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.301987    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4af20ec7-7ea1-470f-b238-62e0de9ca527" volumeName="kubernetes.io/secret/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-tls" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302004    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d9f8f17-bb8e-4f09-ade6-099a78bfe422" volumeName="kubernetes.io/projected/5d9f8f17-bb8e-4f09-ade6-099a78bfe422-kube-api-access-r2wsj" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302016    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9169e62e-57ab-4322-a0cc-664f54e48556" volumeName="kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-cluster-tls-config" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302031    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46ad168-4931-41e7-ac08-f6fa965721f9" volumeName="kubernetes.io/empty-dir/f46ad168-4931-41e7-ac08-f6fa965721f9-config-out" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302047    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00c39778-f1f8-4539-88eb-40d082abdc50" volumeName="kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302059    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08770b22-f9ff-4919-af54-c7916eba9101" volumeName="kubernetes.io/configmap/08770b22-f9ff-4919-af54-c7916eba9101-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302076    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bfb6a7d-e8f9-434f-acfc-8ac3badf1350" volumeName="kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302089    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46ad168-4931-41e7-ac08-f6fa965721f9" volumeName="kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302101    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00c39778-f1f8-4539-88eb-40d082abdc50" volumeName="kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302124    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="117a0c38-48f5-430d-9efe-1a308f725309" volumeName="kubernetes.io/secret/117a0c38-48f5-430d-9efe-1a308f725309-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302137    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00c39778-f1f8-4539-88eb-40d082abdc50" volumeName="kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302155    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00c39778-f1f8-4539-88eb-40d082abdc50" volumeName="kubernetes.io/empty-dir/00c39778-f1f8-4539-88eb-40d082abdc50-tmp" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302167    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08770b22-f9ff-4919-af54-c7916eba9101" volumeName="kubernetes.io/empty-dir/08770b22-f9ff-4919-af54-c7916eba9101-volume-directive-shadow" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302182    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="146e21dc-5041-463e-b0e7-9a79f969fa33" volumeName="kubernetes.io/configmap/146e21dc-5041-463e-b0e7-9a79f969fa33-metrics-client-ca" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302239    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="146e21dc-5041-463e-b0e7-9a79f969fa33" volumeName="kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302252    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5923113e-7941-4bf8-bfde-28622a520e86" volumeName="kubernetes.io/secret/5923113e-7941-4bf8-bfde-28622a520e86-agent-certs" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302264    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9169e62e-57ab-4322-a0cc-664f54e48556" volumeName="kubernetes.io/projected/9169e62e-57ab-4322-a0cc-664f54e48556-tls-assets" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302281    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5b4340c-7e77-4d6e-bdc3-2a6193620f58" volumeName="kubernetes.io/empty-dir/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-data-volume" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302293    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08770b22-f9ff-4919-af54-c7916eba9101" volumeName="kubernetes.io/secret/08770b22-f9ff-4919-af54-c7916eba9101-kube-state-metrics-tls" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302309    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="146e21dc-5041-463e-b0e7-9a79f969fa33" volumeName="kubernetes.io/empty-dir/146e21dc-5041-463e-b0e7-9a79f969fa33-tmp" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302322    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="504075e3-50b2-4616-afea-70d49c4d7293" volumeName="kubernetes.io/secret/504075e3-50b2-4616-afea-70d49c4d7293-monitoring-plugin-cert" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302334    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76f649c8-7dad-410e-a2e8-960e639099d5" volumeName="kubernetes.io/configmap/76f649c8-7dad-410e-a2e8-960e639099d5-metrics-client-ca" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302351    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5c5030e-696b-41dd-a96b-11c9eed11db2" volumeName="kubernetes.io/configmap/e5c5030e-696b-41dd-a96b-11c9eed11db2-config-volume" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302381    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5c5030e-696b-41dd-a96b-11c9eed11db2" volumeName="kubernetes.io/empty-dir/e5c5030e-696b-41dd-a96b-11c9eed11db2-tmp-dir" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302402    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46ad168-4931-41e7-ac08-f6fa965721f9" volumeName="kubernetes.io/empty-dir/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-db" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302414    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00c39778-f1f8-4539-88eb-40d082abdc50" volumeName="kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302426    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4af20ec7-7ea1-470f-b238-62e0de9ca527" volumeName="kubernetes.io/configmap/4af20ec7-7ea1-470f-b238-62e0de9ca527-metrics-client-ca" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302444    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a607eca-e6ad-4e5d-aa57-663768133ec0" volumeName="kubernetes.io/projected/8a607eca-e6ad-4e5d-aa57-663768133ec0-kube-api-access-tz4kk" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302456    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9169e62e-57ab-4322-a0cc-664f54e48556" volumeName="kubernetes.io/configmap/9169e62e-57ab-4322-a0cc-664f54e48556-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302473    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5b4340c-7e77-4d6e-bdc3-2a6193620f58" volumeName="kubernetes.io/configmap/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302485    6059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76f649c8-7dad-410e-a2e8-960e639099d5" volumeName="kubernetes.io/configmap/76f649c8-7dad-410e-a2e8-960e639099d5-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302498    6059 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 17:17:39.304835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.302505    6059 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 17:17:39.326428 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.326135    6059 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 17:17:39.327897 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.327865    6059 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 17:17:39.327897 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.327900    6059 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 17:17:39.328076 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.327928    6059 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 17:17:39.328076 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:39.327981    6059 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 17:17:39.336315 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.336286    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:17:39.336866 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.336848    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:17:39.339874 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.339843    6059 generic.go:356] "Generic (PLEG): container finished" podID="971b27eeac85072c20cb928b063f959b" containerID="9fe2e9daacd45c17ae4f9856b9be09a3afcb913896a511c25813f09043deb1c6" exitCode=0
Nov 03 17:17:39.359517 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.359482    6059 generic.go:356] "Generic (PLEG): container finished" podID="9169e62e-57ab-4322-a0cc-664f54e48556" containerID="05f926711641665314245ac2f0f5f1eb8386119ada7a587f9810c9f3d363c1b9" exitCode=0
Nov 03 17:17:39.374167 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.374134    6059 generic.go:356] "Generic (PLEG): container finished" podID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerID="ab80803ea666ba4724c980d9da8e5702b1510504878c229ff5dc64415ff94575" exitCode=0
Nov 03 17:17:39.391770 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.391691    6059 generic.go:356] "Generic (PLEG): container finished" podID="4af20ec7-7ea1-470f-b238-62e0de9ca527" containerID="f76845680a5523711815a821659ba549c7bfd57da8fbd5c1e2d862b7f315ba6f" exitCode=0
Nov 03 17:17:39.393338 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:39.393309    6059 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-117.us-west-2.compute.internal\" not found"
Nov 03 17:17:39.397092 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.397061    6059 generic.go:356] "Generic (PLEG): container finished" podID="23a7781b-32fa-4f63-ab0e-663e1f2c6abb" containerID="9873e8f5a2fb31a0f8b82fd8aff2bfdbe9197254c7ce312c55a37a2267fa5679" exitCode=0
Nov 03 17:17:39.397092 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.397090    6059 generic.go:356] "Generic (PLEG): container finished" podID="23a7781b-32fa-4f63-ab0e-663e1f2c6abb" containerID="c693989e9116ca8b52467d296d65554356e96f00eac19da8271143d564307219" exitCode=0
Nov 03 17:17:39.397259 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.397100    6059 generic.go:356] "Generic (PLEG): container finished" podID="23a7781b-32fa-4f63-ab0e-663e1f2c6abb" containerID="d099c0eeeb96a2e03e5f70e64eef6ddbeab21dbf1d5c725b0aaa68fa7dafc753" exitCode=0
Nov 03 17:17:39.397259 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.397112    6059 generic.go:356] "Generic (PLEG): container finished" podID="23a7781b-32fa-4f63-ab0e-663e1f2c6abb" containerID="b25a3b9416b77a869f89c508ea92eaaeb69ded4c5ccb7626f7b8328781bcdee3" exitCode=0
Nov 03 17:17:39.397259 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.397124    6059 generic.go:356] "Generic (PLEG): container finished" podID="23a7781b-32fa-4f63-ab0e-663e1f2c6abb" containerID="92c2310c2040a1bb7d8976202fb78354aac3e1f9084cf6645f3f61cc4f32e630" exitCode=0
Nov 03 17:17:39.397259 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.397133    6059 generic.go:356] "Generic (PLEG): container finished" podID="23a7781b-32fa-4f63-ab0e-663e1f2c6abb" containerID="abb7f6775b662f2a5802b444939bd70ee54bcaeb07ef69e27c1e8e798b9daedd" exitCode=0
Nov 03 17:17:39.404202 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.404178    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:17:39.404534 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.404511    6059 generic.go:356] "Generic (PLEG): container finished" podID="8906c21b-3233-4214-8ce1-f9189b79ccd3" containerID="a45894f72c3f44cabca1dd1e1c377e0d970282e716a3c5aea600e6d3d1cf6f39" exitCode=1
Nov 03 17:17:39.428243 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.428214    6059 manager.go:324] Recovery completed
Nov 03 17:17:39.429402 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:39.429344    6059 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 17:17:39.453531 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.453506    6059 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:17:39.455609 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.455578    6059 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-117.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:17:39.455743 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.455620    6059 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-117.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:17:39.455743 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.455631    6059 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-117.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:17:39.459406 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.459385    6059 cpu_manager.go:226] "Starting" policy="none"
Nov 03 17:17:39.459406 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.459400    6059 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 17:17:39.459552 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.459408    6059 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:17:39.459552 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.459422    6059 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 17:17:39.459622 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.459557    6059 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 17:17:39.459622 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.459568    6059 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 17:17:39.459622 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.459595    6059 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 17:17:39.459622 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.459603    6059 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 17:17:39.459622 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.459611    6059 policy_none.go:50] "Start"
Nov 03 17:17:39.459622 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.459621    6059 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 17:17:39.459779 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.459631    6059 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 17:17:39.459808 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.459784    6059 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 17:17:39.459808 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.459792    6059 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 17:17:39.459808 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.459798    6059 policy_none.go:44] "Start"
Nov 03 17:17:39.460766 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.460739    6059 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-117.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:17:39.460828 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.460773    6059 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-117.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:17:39.460828 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.460784    6059 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-117.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:17:39.465081 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.465048    6059 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 17:17:39.470892 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.470877    6059 manager.go:341] "Starting Device Plugin manager"
Nov 03 17:17:39.470995 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:39.470907    6059 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 17:17:39.470995 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.470918    6059 server.go:87] "Starting device plugin registration server"
Nov 03 17:17:39.471195 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.471183    6059 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 17:17:39.471260 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.471195    6059 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 17:17:39.471309 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.471293    6059 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 17:17:39.471458 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.471415    6059 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 17:17:39.471571 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.471465    6059 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 17:17:39.471571 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.471481    6059 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 17:17:39.471675 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.471548    6059 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T17:17:39.471452784Z","UUID":"9300ae8d-0009-4c70-ba6c-ad11b59ac5c1","Handler":null,"Name":"","Endpoint":""}
Nov 03 17:17:39.473008 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.472981    6059 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 17:17:39.473112 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.473018    6059 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 17:17:39.473475 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:39.473416    6059 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 17:17:39.473475 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:39.473456    6059 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-117.us-west-2.compute.internal\" not found"
Nov 03 17:17:39.485690 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.485661    6059 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 17:17:39.486063 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.486040    6059 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:17:39.486821 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.486803    6059 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:17:39.508814 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.508776    6059 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:17:39.571597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.571552    6059 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:17:39.572869 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.572847    6059 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-117.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:17:39.572976 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.572887    6059 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-117.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:17:39.572976 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.572900    6059 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-117.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:17:39.573083 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.573006    6059 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:17:39.580619 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.580595    6059 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:17:39.630244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.630187    6059 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal"]
Nov 03 17:17:39.630965 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.630940    6059 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:17:39.631070 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.630981    6059 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:17:39.631070 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.630915    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal" event={"ID":"971b27eeac85072c20cb928b063f959b","Type":"ContainerStarted","Data":"a4fdfc3081da1d52d09832c26820b77ea8b2e84474e8e88a977d95bb2b7f1b16"}
Nov 03 17:17:39.631154 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.631065    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal" event={"ID":"971b27eeac85072c20cb928b063f959b","Type":"ContainerDied","Data":"9fe2e9daacd45c17ae4f9856b9be09a3afcb913896a511c25813f09043deb1c6"}
Nov 03 17:17:39.631154 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.631102    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal" event={"ID":"971b27eeac85072c20cb928b063f959b","Type":"ContainerStarted","Data":"d575a9e899a6ee915a4ab04da5ad50997b35b07410a6755aa4fcf3c7a98e9d11"}
Nov 03 17:17:39.631211 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.631183    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal" event={"ID":"01359929955a1f3cff637d7cc239a2bc","Type":"ContainerStarted","Data":"95c2a442a3b634b410e3063b219ac4139120fce65246186bca68c30886d2c8eb"}
Nov 03 17:17:39.631211 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.631206    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal" event={"ID":"01359929955a1f3cff637d7cc239a2bc","Type":"ContainerStarted","Data":"74523671170fb1512036d6c47d3b5c4cd185b44d3dc90ff6412360d24a69690f"}
Nov 03 17:17:39.685310 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.685232    6059 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:17:39.685449 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.685320    6059 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:17:39.688183 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.688165    6059 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:17:39.688272 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:39.688209    6059 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:17:39.688696 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.688681    6059 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:17:39.688762 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:39.688707    6059 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:17:39.787747 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.787709    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/971b27eeac85072c20cb928b063f959b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal\" (UID: \"971b27eeac85072c20cb928b063f959b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:17:39.787747 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.787758    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/971b27eeac85072c20cb928b063f959b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal\" (UID: \"971b27eeac85072c20cb928b063f959b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:17:39.787983 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.787853    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01359929955a1f3cff637d7cc239a2bc-config\") pod \"kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal\" (UID: \"01359929955a1f3cff637d7cc239a2bc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:17:39.888026 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.887988    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/971b27eeac85072c20cb928b063f959b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal\" (UID: \"971b27eeac85072c20cb928b063f959b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:17:39.888026 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.888028    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01359929955a1f3cff637d7cc239a2bc-config\") pod \"kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal\" (UID: \"01359929955a1f3cff637d7cc239a2bc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:17:39.888026 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.888046    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/971b27eeac85072c20cb928b063f959b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal\" (UID: \"971b27eeac85072c20cb928b063f959b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:17:39.888301 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.888090    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/971b27eeac85072c20cb928b063f959b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal\" (UID: \"971b27eeac85072c20cb928b063f959b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:17:39.888301 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.888107    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/971b27eeac85072c20cb928b063f959b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal\" (UID: \"971b27eeac85072c20cb928b063f959b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:17:39.888301 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:39.888113    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01359929955a1f3cff637d7cc239a2bc-config\") pod \"kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal\" (UID: \"01359929955a1f3cff637d7cc239a2bc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:17:40.277749 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.277706    6059 apiserver.go:52] "Watching apiserver"
Nov 03 17:17:40.377638 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.377596    6059 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 17:17:40.378323 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.378296    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2","openshift-dns/dns-default-vpzpk","openshift-monitoring/prometheus-k8s-0","openshift-dns/node-resolver-6l5k8","kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-d252q","openshift-cluster-node-tuning-operator/tuned-4dtnl","openshift-monitoring/alertmanager-main-0","openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc","openshift-multus/multus-additional-cni-plugins-b99b8","openshift-console/console-5d896bf64d-smcmk","openshift-image-registry/node-ca-bkmvr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal","openshift-monitoring/thanos-querier-576dfb567c-pbb4z","kube-system/global-pull-secret-syncer-rt4lh","openshift-ingress-canary/ingress-canary-dv7x7","openshift-monitoring/metrics-server-865c7f6b8c-gq7x9","openshift-monitoring/telemeter-client-548c4c64fb-pgglg","openshift-monitoring/monitoring-plugin-77f4f76594-67sr8","openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg","openshift-insights/insights-runtime-extractor-5gv26","openshift-multus/network-metrics-daemon-484l6","openshift-ovn-kubernetes/ovnkube-node-c5n7n","openshift-monitoring/node-exporter-9gj97","openshift-multus/multus-9cgng","openshift-network-operator/iptables-alerter-z5v9n","kube-system/konnectivity-agent-dzlp4","openshift-console/console-858bfbb7dc-x6n58"]
Nov 03 17:17:40.409885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.409859    6059 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:17:40.410055 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.410035    6059 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:17:40.431010 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.430969    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 17:17:40.431010 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.430973    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 17:17:40.431281 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.430969    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x9v6p\""
Nov 03 17:17:40.431281 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.430995    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h78gf\""
Nov 03 17:17:40.432909 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.432872    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 17:17:40.433026 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.432992    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 17:17:40.433081 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.433056    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 17:17:40.433179 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.433156    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 17:17:40.433563 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.433544    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 17:17:40.433878 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.433860    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 17:17:40.434017 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.433997    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 17:17:40.434232 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.434216    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-d8tp8\""
Nov 03 17:17:40.434348 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.434331    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hwgrr\""
Nov 03 17:17:40.435138 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.435117    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 17:17:40.435272 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.435230    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-czdwp\""
Nov 03 17:17:40.435407 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.435388    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mr6xr\""
Nov 03 17:17:40.435617 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.435600    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fmcvz\""
Nov 03 17:17:40.435742 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.435621    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9lkll\""
Nov 03 17:17:40.435835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.435654    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 17:17:40.435923 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.435885    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 17:17:40.436053 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.436023    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c6dxq\""
Nov 03 17:17:40.436456 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.436437    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h6tl4\""
Nov 03 17:17:40.436587 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.436494    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 17:17:40.436729 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.436549    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-whvtr\""
Nov 03 17:17:40.436871 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.436581    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:17:40.437003 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.436908    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 17:17:40.437244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.437187    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 17:17:40.437464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.437294    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 17:17:40.437464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.437304    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 17:17:40.437464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.437330    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 17:17:40.437464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.437380    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 17:17:40.437464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.437414    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 17:17:40.437653 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.437472    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:17:40.437653 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.437477    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 17:17:40.437653 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.437504    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 17:17:40.437653 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.437581    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 17:17:40.437789 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.437679    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 17:17:40.438260 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.438244    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 17:17:40.477527 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.477461    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 17:17:40.477872 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.477579    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 17:17:40.477872 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.477657    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 17:17:40.477872 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.477665    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 17:17:40.477872 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.477678    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 17:17:40.477872 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.477687    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 17:17:40.477872 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.477704    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 17:17:40.477872 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.477763    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 17:17:40.477872 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.477787    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 17:17:40.477872 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.477793    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 17:17:40.477872 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.477841    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 17:17:40.478464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.477919    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gnkq2\""
Nov 03 17:17:40.478464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.477993    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qk7gm\""
Nov 03 17:17:40.478464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.478025    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 17:17:40.478464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.478029    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 17:17:40.478464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.478221    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 17:17:40.478464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.478243    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 17:17:40.478464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.478420    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 17:17:40.479027 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.479000    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 17:17:40.479027 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.479011    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 17:17:40.479345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.479063    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 17:17:40.479345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.479077    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-h2tc2\""
Nov 03 17:17:40.479345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.479119    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rppdf\""
Nov 03 17:17:40.479345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.479142    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 17:17:40.479345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.479165    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2ptg9\""
Nov 03 17:17:40.479345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.479185    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bpr11s5cm6kpg\""
Nov 03 17:17:40.479345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.479190    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jrssc\""
Nov 03 17:17:40.479345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.479238    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 17:17:40.479814 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.479498    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 17:17:40.479814 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.479522    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 17:17:40.479891 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.479854    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 17:17:40.479891 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.479855    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 17:17:40.480404 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.480390    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 17:17:40.480404 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.480398    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-p2lsz\""
Nov 03 17:17:40.480549 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.480510    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 17:17:40.480874 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.480858    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 17:17:40.480931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.480865    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 17:17:40.481072 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.481058    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cjbocgem5ujea\""
Nov 03 17:17:40.481224 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.481213    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 17:17:40.487782 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.487760    6059 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:17:40.487857 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.487815    6059 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:17:40.488337 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.488322    6059 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:17:40.488423 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.488388    6059 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-117.us-west-2.compute.internal"
Nov 03 17:17:40.491564 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.491543    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/902e0389-e70e-4d23-a607-d8f4b9da868d-kubelet-config\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:17:40.491655 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.491577    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5923113e-7941-4bf8-bfde-28622a520e86-agent-certs\") pod \"konnectivity-agent-dzlp4\" (UID: \"5923113e-7941-4bf8-bfde-28622a520e86\") " pod="kube-system/konnectivity-agent-dzlp4"
Nov 03 17:17:40.491655 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.491607    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-data-volume\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:40.491727 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.491708    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-federate-client-tls\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:40.491767 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.491751    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.491798 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.491782    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:40.491832 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.491811    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:40.491832 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.491712    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-data-volume\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:40.491923 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.491837    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfb4h\" (UniqueName: \"kubernetes.io/projected/504075e3-50b2-4616-afea-70d49c4d7293-kube-api-access-wfb4h\") pod \"monitoring-plugin-77f4f76594-67sr8\" (UID: \"504075e3-50b2-4616-afea-70d49c4d7293\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-67sr8"
Nov 03 17:17:40.491923 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.491864    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-tls\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:40.491923 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.491890    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-client-ca-bundle\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:40.491923 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.491899    6059 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 17:17:40.491923 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.491906    6059 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:40.492144 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.491920    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e59c4c3e-b582-4b16-a6e3-4aadf0612a32-tmp-dir\") pod \"node-resolver-6l5k8\" (UID: \"e59c4c3e-b582-4b16-a6e3-4aadf0612a32\") " pod="openshift-dns/node-resolver-6l5k8"
Nov 03 17:17:40.492144 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.491940    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.492144 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.491949    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs\") pod \"network-metrics-daemon-484l6\" (UID: \"bd2a75ff-95c3-4b08-b75b-30059af6bd37\") " pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:17:40.492144 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.491967    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:40.991946263 +0000 UTC m=+1.789659024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"service-ca" not registered
Nov 03 17:17:40.492144 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.491968    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5923113e-7941-4bf8-bfde-28622a520e86-agent-certs\") pod \"konnectivity-agent-dzlp4\" (UID: \"5923113e-7941-4bf8-bfde-28622a520e86\") " pod="kube-system/konnectivity-agent-dzlp4"
Nov 03 17:17:40.492144 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.492011    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:40.99199795 +0000 UTC m=+1.789710723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:40.492144 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492026    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e59c4c3e-b582-4b16-a6e3-4aadf0612a32-tmp-dir\") pod \"node-resolver-6l5k8\" (UID: \"e59c4c3e-b582-4b16-a6e3-4aadf0612a32\") " pod="openshift-dns/node-resolver-6l5k8"
Nov 03 17:17:40.492144 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492032    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:17:40.492144 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492058    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-etc-kubernetes\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.492514 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492165    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-textfile\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:40.492514 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492171    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd2a75ff-95c3-4b08-b75b-30059af6bd37-metrics-certs\") pod \"network-metrics-daemon-484l6\" (UID: \"bd2a75ff-95c3-4b08-b75b-30059af6bd37\") " pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:17:40.492514 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492186    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-client-ca-bundle\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:40.492514 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492229    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.492514 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492251    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-textfile\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:40.492514 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492289    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-web-config\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.492514 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492351    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-secret-telemeter-client\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:40.492514 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492466    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-telemeter-client-tls\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:40.492514 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492488    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-var-lib-kubelet\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.492514 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492505    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.492847 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492525    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4af20ec7-7ea1-470f-b238-62e0de9ca527-sys\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:40.492847 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492551    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-systemd\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.492847 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492586    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/117a0c38-48f5-430d-9efe-1a308f725309-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5srqg\" (UID: \"117a0c38-48f5-430d-9efe-1a308f725309\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"
Nov 03 17:17:40.492847 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492654    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-run-netns\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.492847 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492682    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-slash\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.492847 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492736    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8906c21b-3233-4214-8ce1-f9189b79ccd3-ovn-node-metrics-cert\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.492847 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492773    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-socket-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:17:40.492847 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492840    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/117a0c38-48f5-430d-9efe-1a308f725309-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5srqg\" (UID: \"117a0c38-48f5-430d-9efe-1a308f725309\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"
Nov 03 17:17:40.493102 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492863    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-registration-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:17:40.493102 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492892    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-socket-dir-parent\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.493102 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492917    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-conf-dir\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.493102 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492931    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8906c21b-3233-4214-8ce1-f9189b79ccd3-ovn-node-metrics-cert\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.493102 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492952    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.493102 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492975    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.493102 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.492997    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-cni-bin\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.493102 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493083    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlwvb\" (UniqueName: \"kubernetes.io/projected/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-kube-api-access-wlwvb\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:40.493454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493140    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:40.493454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493234    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kx94\" (UniqueName: \"kubernetes.io/projected/e59c4c3e-b582-4b16-a6e3-4aadf0612a32-kube-api-access-2kx94\") pod \"node-resolver-6l5k8\" (UID: \"e59c4c3e-b582-4b16-a6e3-4aadf0612a32\") " pod="openshift-dns/node-resolver-6l5k8"
Nov 03 17:17:40.493454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493254    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-host\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.493454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493286    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00710f2d-31bb-42da-9ae8-135ddc117df5-cni-binary-copy\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.493454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493332    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:40.493454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493378    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c986b\" (UniqueName: \"kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:40.493454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493382    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.493454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493407    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-run-openvswitch\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.493454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493432    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/08770b22-f9ff-4919-af54-c7916eba9101-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:40.493454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493456    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/88482077-204d-4634-8045-c741d2c10d80-iptables-alerter-script\") pod \"iptables-alerter-z5v9n\" (UID: \"88482077-204d-4634-8045-c741d2c10d80\") " pod="openshift-network-operator/iptables-alerter-z5v9n"
Nov 03 17:17:40.493912 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.493514    6059 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:40.493912 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493524    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88482077-204d-4634-8045-c741d2c10d80-host-slash\") pod \"iptables-alerter-z5v9n\" (UID: \"88482077-204d-4634-8045-c741d2c10d80\") " pod="openshift-network-operator/iptables-alerter-z5v9n"
Nov 03 17:17:40.493912 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493556    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76f649c8-7dad-410e-a2e8-960e639099d5-serving-certs-ca-bundle\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:40.493912 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493562    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/08770b22-f9ff-4919-af54-c7916eba9101-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:40.493912 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.493587    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:40.993571132 +0000 UTC m=+1.791283914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:40.493912 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493610    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:40.493912 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493642    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-run\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.493912 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493665    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00710f2d-31bb-42da-9ae8-135ddc117df5-cni-binary-copy\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.493912 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493719    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-system-cni-dir\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.493912 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.493722    6059 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:40.493912 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493746    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-hostroot\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.493912 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493758    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/88482077-204d-4634-8045-c741d2c10d80-iptables-alerter-script\") pod \"iptables-alerter-z5v9n\" (UID: \"88482077-204d-4634-8045-c741d2c10d80\") " pod="openshift-network-operator/iptables-alerter-z5v9n"
Nov 03 17:17:40.493912 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493791    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00c39778-f1f8-4539-88eb-40d082abdc50-tmp\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:40.493912 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.493844    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:40.993827926 +0000 UTC m=+1.791540699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:40.493912 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493878    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.493912 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493911    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-web-config\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.493912 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493913    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00c39778-f1f8-4539-88eb-40d082abdc50-tmp\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:40.494597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493939    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd5nd\" (UniqueName: \"kubernetes.io/projected/4af20ec7-7ea1-470f-b238-62e0de9ca527-kube-api-access-jd5nd\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:40.494597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493969    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:17:40.494597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.493998    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-sys\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.494597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494024    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz4kk\" (UniqueName: \"kubernetes.io/projected/8a607eca-e6ad-4e5d-aa57-663768133ec0-kube-api-access-tz4kk\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.494597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494128    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-daemon-config\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.494597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494224    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:17:40.494597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494257    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/504075e3-50b2-4616-afea-70d49c4d7293-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-67sr8\" (UID: \"504075e3-50b2-4616-afea-70d49c4d7293\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-67sr8"
Nov 03 17:17:40.494597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494280    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-config\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.494597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494336    6059 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 17:17:40.494597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494347    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4af20ec7-7ea1-470f-b238-62e0de9ca527-metrics-client-ca\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:40.494597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494401    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-accelerators-collector-config\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:40.494597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494453    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c5030e-696b-41dd-a96b-11c9eed11db2-tmp-dir\") pod \"dns-default-vpzpk\" (UID: \"e5c5030e-696b-41dd-a96b-11c9eed11db2\") " pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:40.494597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494485    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-run-k8s-cni-cncf-io\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.494597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494489    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-daemon-config\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.494597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494513    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/504075e3-50b2-4616-afea-70d49c4d7293-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-67sr8\" (UID: \"504075e3-50b2-4616-afea-70d49c4d7293\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-67sr8"
Nov 03 17:17:40.494597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494518    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2wsj\" (UniqueName: \"kubernetes.io/projected/5d9f8f17-bb8e-4f09-ade6-099a78bfe422-kube-api-access-r2wsj\") pod \"node-ca-bkmvr\" (UID: \"5d9f8f17-bb8e-4f09-ade6-099a78bfe422\") " pod="openshift-image-registry/node-ca-bkmvr"
Nov 03 17:17:40.494597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494538    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c5030e-696b-41dd-a96b-11c9eed11db2-tmp-dir\") pod \"dns-default-vpzpk\" (UID: \"e5c5030e-696b-41dd-a96b-11c9eed11db2\") " pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:40.494597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494540    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4af20ec7-7ea1-470f-b238-62e0de9ca527-metrics-client-ca\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:40.494597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494555    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.494597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494606    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4sxz\" (UniqueName: \"kubernetes.io/projected/f46ad168-4931-41e7-ac08-f6fa965721f9-kube-api-access-q4sxz\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494645    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-etc-selinux\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494650    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494663    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9169e62e-57ab-4322-a0cc-664f54e48556-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494681    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-cni-binary-copy\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494714    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494784    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9169e62e-57ab-4322-a0cc-664f54e48556-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494796    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494839    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-kubelet\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.494873    6059 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494930    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-sys-fs\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.494936    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:40.994922861 +0000 UTC m=+1.792635642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494950    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-cni-binary-copy\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.494979    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/08770b22-f9ff-4919-af54-c7916eba9101-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495025    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9169e62e-57ab-4322-a0cc-664f54e48556-config-out\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495045    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495043    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/902e0389-e70e-4d23-a607-d8f4b9da868d-original-pull-secret\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495109    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9169e62e-57ab-4322-a0cc-664f54e48556-config-out\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495136    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-secret-metrics-server-tls\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495154    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/902e0389-e70e-4d23-a607-d8f4b9da868d-dbus\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495171    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hljmh\" (UniqueName: \"kubernetes.io/projected/00710f2d-31bb-42da-9ae8-135ddc117df5-kube-api-access-hljmh\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495190    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-cni-netd\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495224    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495229    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/08770b22-f9ff-4919-af54-c7916eba9101-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495243    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9169e62e-57ab-4322-a0cc-664f54e48556-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495275    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-857nt\" (UniqueName: \"kubernetes.io/projected/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-kube-api-access-857nt\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495270    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495310    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495376    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e59c4c3e-b582-4b16-a6e3-4aadf0612a32-hosts-file\") pod \"node-resolver-6l5k8\" (UID: \"e59c4c3e-b582-4b16-a6e3-4aadf0612a32\") " pod="openshift-dns/node-resolver-6l5k8"
Nov 03 17:17:40.495394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495414    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-lib-modules\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.495445    6059 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495460    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495446    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-systemd-units\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495445    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9169e62e-57ab-4322-a0cc-664f54e48556-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495472    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-secret-metrics-server-tls\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.495494    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:40.995480931 +0000 UTC m=+1.793193694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495529    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8906c21b-3233-4214-8ce1-f9189b79ccd3-ovnkube-config\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495559    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/08770b22-f9ff-4919-af54-c7916eba9101-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495595    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-tls\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495636    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.495692    6059 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495693    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495716    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-tuned\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495731    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f6cf431-dc3b-478e-81cd-97127a917de3-cert\") pod \"ingress-canary-dv7x7\" (UID: \"9f6cf431-dc3b-478e-81cd-97127a917de3\") " pod="openshift-ingress-canary/ingress-canary-dv7x7"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.495752    6059 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495692    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8906c21b-3233-4214-8ce1-f9189b79ccd3-ovnkube-config\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.495774    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:40.995730808 +0000 UTC m=+1.793443585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.495800    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:40.995784274 +0000 UTC m=+1.793497056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495819    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4af20ec7-7ea1-470f-b238-62e0de9ca527-root\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495824    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-tuned\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495863    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d9f8f17-bb8e-4f09-ade6-099a78bfe422-serviceca\") pod \"node-ca-bkmvr\" (UID: \"5d9f8f17-bb8e-4f09-ade6-099a78bfe422\") " pod="openshift-image-registry/node-ca-bkmvr"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495880    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a607eca-e6ad-4e5d-aa57-663768133ec0-tmp\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495896    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/117a0c38-48f5-430d-9efe-1a308f725309-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5srqg\" (UID: \"117a0c38-48f5-430d-9efe-1a308f725309\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495915    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f6cf431-dc3b-478e-81cd-97127a917de3-cert\") pod \"ingress-canary-dv7x7\" (UID: \"9f6cf431-dc3b-478e-81cd-97127a917de3\") " pod="openshift-ingress-canary/ingress-canary-dv7x7"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495924    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5923113e-7941-4bf8-bfde-28622a520e86-konnectivity-ca\") pod \"konnectivity-agent-dzlp4\" (UID: \"5923113e-7941-4bf8-bfde-28622a520e86\") " pod="kube-system/konnectivity-agent-dzlp4"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495950    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d9f8f17-bb8e-4f09-ade6-099a78bfe422-host\") pod \"node-ca-bkmvr\" (UID: \"5d9f8f17-bb8e-4f09-ade6-099a78bfe422\") " pod="openshift-image-registry/node-ca-bkmvr"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495952    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a607eca-e6ad-4e5d-aa57-663768133ec0-tmp\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.495992    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-log-socket\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496036    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5923113e-7941-4bf8-bfde-28622a520e86-konnectivity-ca\") pod \"konnectivity-agent-dzlp4\" (UID: \"5923113e-7941-4bf8-bfde-28622a520e86\") " pod="kube-system/konnectivity-agent-dzlp4"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496053    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9169e62e-57ab-4322-a0cc-664f54e48556-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496078    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/117a0c38-48f5-430d-9efe-1a308f725309-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5srqg\" (UID: \"117a0c38-48f5-430d-9efe-1a308f725309\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496080    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjrtj\" (UniqueName: \"kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496127    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-run-netns\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496156    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7jmf\" (UniqueName: \"kubernetes.io/projected/e5c5030e-696b-41dd-a96b-11c9eed11db2-kube-api-access-n7jmf\") pod \"dns-default-vpzpk\" (UID: \"e5c5030e-696b-41dd-a96b-11c9eed11db2\") " pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496181    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-sysctl-conf\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496232    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-var-lib-kubelet\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496266    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5th7\" (UniqueName: \"kubernetes.io/projected/bd2a75ff-95c3-4b08-b75b-30059af6bd37-kube-api-access-z5th7\") pod \"network-metrics-daemon-484l6\" (UID: \"bd2a75ff-95c3-4b08-b75b-30059af6bd37\") " pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496306    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-cni-dir\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496328    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496349    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496445    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5d9f8f17-bb8e-4f09-ade6-099a78bfe422-serviceca\") pod \"node-ca-bkmvr\" (UID: \"5d9f8f17-bb8e-4f09-ade6-099a78bfe422\") " pod="openshift-image-registry/node-ca-bkmvr"
Nov 03 17:17:40.496495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496505    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76f649c8-7dad-410e-a2e8-960e639099d5-metrics-client-ca\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496533    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-sysctl-d\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496558    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l26gw\" (UniqueName: \"kubernetes.io/projected/117a0c38-48f5-430d-9efe-1a308f725309-kube-api-access-l26gw\") pod \"openshift-state-metrics-7bd7489957-5srqg\" (UID: \"117a0c38-48f5-430d-9efe-1a308f725309\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496573    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-cnibin\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496616    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f46ad168-4931-41e7-ac08-f6fa965721f9-config-out\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496651    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496680    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496706    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f46ad168-4931-41e7-ac08-f6fa965721f9-config-out\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496710    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-etc-openvswitch\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496685    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76f649c8-7dad-410e-a2e8-960e639099d5-metrics-client-ca\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496755    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-system-cni-dir\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496783    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496833    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.496852    6059 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496865    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.496909    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:40.996895831 +0000 UTC m=+1.794608606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496939    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-run-systemd\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.496971    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497002    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/146e21dc-5041-463e-b0e7-9a79f969fa33-tmp\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497054    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/146e21dc-5041-463e-b0e7-9a79f969fa33-tmp\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497076    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497086    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66cnr\" (UniqueName: \"kubernetes.io/projected/146e21dc-5041-463e-b0e7-9a79f969fa33-kube-api-access-66cnr\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497111    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497129    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76f649c8-7dad-410e-a2e8-960e639099d5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497146    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5c5030e-696b-41dd-a96b-11c9eed11db2-metrics-tls\") pod \"dns-default-vpzpk\" (UID: \"e5c5030e-696b-41dd-a96b-11c9eed11db2\") " pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497172    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/117a0c38-48f5-430d-9efe-1a308f725309-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5srqg\" (UID: \"117a0c38-48f5-430d-9efe-1a308f725309\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497198    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497224    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-var-lib-openvswitch\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497252    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmd9x\" (UniqueName: \"kubernetes.io/projected/c1e30c92-229f-4030-82eb-9295f5328da4-kube-api-access-bmd9x\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.497266    6059 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497282    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08770b22-f9ff-4919-af54-c7916eba9101-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.497309    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:40.997296339 +0000 UTC m=+1.795009113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"console-config" not registered
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497316    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5c5030e-696b-41dd-a96b-11c9eed11db2-metrics-tls\") pod \"dns-default-vpzpk\" (UID: \"e5c5030e-696b-41dd-a96b-11c9eed11db2\") " pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497328    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-grpc-tls\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497378    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497377    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/117a0c38-48f5-430d-9efe-1a308f725309-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5srqg\" (UID: \"117a0c38-48f5-430d-9efe-1a308f725309\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497439    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-var-lib-cni-bin\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497471    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497489    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08770b22-f9ff-4919-af54-c7916eba9101-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497500    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497543    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-grpc-tls\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497545    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497577    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-secret-metrics-server-client-certs\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497587    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497595    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-metrics-server-audit-profiles\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497613    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497630    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-tmp\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497646    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-modprobe-d\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497661    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497670    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-var-lib-cni-multus\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497704    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497732    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8906c21b-3233-4214-8ce1-f9189b79ccd3-ovnkube-script-lib\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497765    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/146e21dc-5041-463e-b0e7-9a79f969fa33-metrics-client-ca\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497772    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-secret-metrics-server-client-certs\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497803    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-cnibin\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497804    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-tmp\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497843    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c5030e-696b-41dd-a96b-11c9eed11db2-config-volume\") pod \"dns-default-vpzpk\" (UID: \"e5c5030e-696b-41dd-a96b-11c9eed11db2\") " pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497865    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497890    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497921    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl2pt\" (UniqueName: \"kubernetes.io/projected/88482077-204d-4634-8045-c741d2c10d80-kube-api-access-hl2pt\") pod \"iptables-alerter-z5v9n\" (UID: \"88482077-204d-4634-8045-c741d2c10d80\") " pod="openshift-network-operator/iptables-alerter-z5v9n"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497953    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497979    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8906c21b-3233-4214-8ce1-f9189b79ccd3-ovnkube-script-lib\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497984    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/146e21dc-5041-463e-b0e7-9a79f969fa33-metrics-client-ca\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.497981    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-run-ovn\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498031    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-node-log\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498058    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c5030e-696b-41dd-a96b-11c9eed11db2-config-volume\") pod \"dns-default-vpzpk\" (UID: \"e5c5030e-696b-41dd-a96b-11c9eed11db2\") " pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498072    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498099    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08770b22-f9ff-4919-af54-c7916eba9101-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498128    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9169e62e-57ab-4322-a0cc-664f54e48556-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.498104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498151    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jklgn\" (UniqueName: \"kubernetes.io/projected/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-kube-api-access-jklgn\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498179    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l8qd\" (UniqueName: \"kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd\") pod \"network-check-target-d252q\" (UID: \"37080dcf-15b0-4d4e-85fa-ff669783f356\") " pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498308    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498324    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08770b22-f9ff-4919-af54-c7916eba9101-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498331    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpc58\" (UniqueName: \"kubernetes.io/projected/8906c21b-3233-4214-8ce1-f9189b79ccd3-kube-api-access-jpc58\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498351    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-crio-socket\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498393    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498398    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9169e62e-57ab-4322-a0cc-664f54e48556-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498410    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.498498    6059 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498502    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498557    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-config-volume\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498587    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.498615    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:40.998601414 +0000 UTC m=+1.796314207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"service-ca" not registered
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498638    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-kubernetes\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.498662    6059 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498669    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-run-ovn-kubernetes\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.498698    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:40.998686805 +0000 UTC m=+1.796399581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"console-config" not registered
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498716    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-wtmp\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498748    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wr9f\" (UniqueName: \"kubernetes.io/projected/76f649c8-7dad-410e-a2e8-960e639099d5-kube-api-access-6wr9f\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498777    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-sysconfig\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498795    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498814    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46p7r\" (UniqueName: \"kubernetes.io/projected/08770b22-f9ff-4919-af54-c7916eba9101-kube-api-access-46p7r\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498842    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498872    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jp8z\" (UniqueName: \"kubernetes.io/projected/9169e62e-57ab-4322-a0cc-664f54e48556-kube-api-access-5jp8z\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498937    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-run-multus-certs\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.498966    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f46ad168-4931-41e7-ac08-f6fa965721f9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.499004    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8906c21b-3233-4214-8ce1-f9189b79ccd3-env-overrides\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.499032    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.499131    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.499171    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn94x\" (UniqueName: \"kubernetes.io/projected/9f6cf431-dc3b-478e-81cd-97127a917de3-kube-api-access-qn94x\") pod \"ingress-canary-dv7x7\" (UID: \"9f6cf431-dc3b-478e-81cd-97127a917de3\") " pod="openshift-ingress-canary/ingress-canary-dv7x7"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.499181    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8906c21b-3233-4214-8ce1-f9189b79ccd3-env-overrides\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.499204    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-audit-log\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.499211    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.499238    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-device-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:17:40.500285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.499256    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-audit-log\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:40.530881 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.530803    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 17:17:40.530881 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.530808    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 17:17:40.530881 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.530852    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 17:17:40.531072 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.531038    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 17:17:40.531110 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.531069    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 17:17:40.531174 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.531157    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 17:17:40.531208 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.531184    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 17:17:40.531303 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.531287    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 17:17:40.531391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.531294    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zs5l8\""
Nov 03 17:17:40.531391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.531311    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 17:17:40.531391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.531352    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 17:17:40.531502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.531382    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 17:17:40.531502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.531403    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 17:17:40.531502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.531448    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 17:17:40.531502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.531474    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 17:17:40.531502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.531490    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 17:17:40.531664 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.531545    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xr7h8\""
Nov 03 17:17:40.531664 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.531577    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 17:17:40.531751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.531670    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4vh8j\""
Nov 03 17:17:40.531751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.531708    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 17:17:40.531751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.531719    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 17:17:40.531751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.531710    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 17:17:40.532042 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.532023    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-federate-client-tls\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:40.532111 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.532095    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 17:17:40.532157 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.532143    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 17:17:40.532206 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.532154    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 17:17:40.532206 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.532177    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8kr56n0suls3\""
Nov 03 17:17:40.532206 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.532196    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 17:17:40.532349 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.532210    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 17:17:40.532349 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.532234    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 17:17:40.532349 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.532237    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2c6sf\""
Nov 03 17:17:40.532349 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.532277    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 17:17:40.532349 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.532271    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 17:17:40.532349 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.532296    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 17:17:40.532349 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.532325    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nxfj8\""
Nov 03 17:17:40.532650 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.532380    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-k5tst\""
Nov 03 17:17:40.532650 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.532426    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 17:17:40.532650 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.532552    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 17:17:40.532650 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.532599    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-web-config\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.532650 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.532625    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-secret-telemeter-client\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:40.532885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.532769    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-telemeter-client-tls\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:40.532885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.532838    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.533152 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.533051    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.533152 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.533151    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.533451 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.533424    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:40.533796 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.533779    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76f649c8-7dad-410e-a2e8-960e639099d5-serving-certs-ca-bundle\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:40.534069 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.534051    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 17:17:40.534174 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.534159    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-web-config\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.534208 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.534180    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 17:17:40.534208 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.534187    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.534435 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.534419    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 17:17:40.534515 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.534501    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-config\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.534637 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.534625    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-accelerators-collector-config\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:40.536651 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.536633    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9169e62e-57ab-4322-a0cc-664f54e48556-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.536724 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.536666    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-tls\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:40.536772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.536722    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.536816 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.536800    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:40.536856 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.536846    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/08770b22-f9ff-4919-af54-c7916eba9101-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:40.536933 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.536920    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.537946 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.537930    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:40.538009 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.537988    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:40.538050 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.538016    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-metrics-server-audit-profiles\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:40.538050 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.538029    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76f649c8-7dad-410e-a2e8-960e639099d5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:40.538543 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.538042    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.542519 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.538864    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76f649c8-7dad-410e-a2e8-960e639099d5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:40.542519 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.538951    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-config-volume\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.542519 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.538968    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.542519 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.539087    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.542519 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.539119    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/146e21dc-5041-463e-b0e7-9a79f969fa33-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:40.542519 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.539149    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.542519 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.539193    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f46ad168-4931-41e7-ac08-f6fa965721f9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.542519 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.539716    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.542519 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.540228    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.542977 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.542958    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-tls\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:40.548855 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.548834    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfb4h\" (UniqueName: \"kubernetes.io/projected/504075e3-50b2-4616-afea-70d49c4d7293-kube-api-access-wfb4h\") pod \"monitoring-plugin-77f4f76594-67sr8\" (UID: \"504075e3-50b2-4616-afea-70d49c4d7293\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-67sr8"
Nov 03 17:17:40.558184 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.558158    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlwvb\" (UniqueName: \"kubernetes.io/projected/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-kube-api-access-wlwvb\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:40.559663 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.559646    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2wsj\" (UniqueName: \"kubernetes.io/projected/5d9f8f17-bb8e-4f09-ade6-099a78bfe422-kube-api-access-r2wsj\") pod \"node-ca-bkmvr\" (UID: \"5d9f8f17-bb8e-4f09-ade6-099a78bfe422\") " pod="openshift-image-registry/node-ca-bkmvr"
Nov 03 17:17:40.559936 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.559918    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5th7\" (UniqueName: \"kubernetes.io/projected/bd2a75ff-95c3-4b08-b75b-30059af6bd37-kube-api-access-z5th7\") pod \"network-metrics-daemon-484l6\" (UID: \"bd2a75ff-95c3-4b08-b75b-30059af6bd37\") " pod="openshift-multus/network-metrics-daemon-484l6"
Nov 03 17:17:40.561633 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.561615    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl2pt\" (UniqueName: \"kubernetes.io/projected/88482077-204d-4634-8045-c741d2c10d80-kube-api-access-hl2pt\") pod \"iptables-alerter-z5v9n\" (UID: \"88482077-204d-4634-8045-c741d2c10d80\") " pod="openshift-network-operator/iptables-alerter-z5v9n"
Nov 03 17:17:40.561752 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.561737    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4sxz\" (UniqueName: \"kubernetes.io/projected/f46ad168-4931-41e7-ac08-f6fa965721f9-kube-api-access-q4sxz\") pod \"prometheus-k8s-0\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:40.561821 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.561804    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66cnr\" (UniqueName: \"kubernetes.io/projected/146e21dc-5041-463e-b0e7-9a79f969fa33-kube-api-access-66cnr\") pod \"thanos-querier-576dfb567c-pbb4z\" (UID: \"146e21dc-5041-463e-b0e7-9a79f969fa33\") " pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:40.561954 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.561940    6059 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 17:17:40.561993 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.561959    6059 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 17:17:40.561993 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.561970    6059 projected.go:196] Error preparing data for projected volume kube-api-access-cjrtj for pod openshift-console/console-858bfbb7dc-x6n58: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:40.562059 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.562035    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:41.062020145 +0000 UTC m=+1.859732926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cjrtj" (UniqueName: "kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:40.565020 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.565003    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpc58\" (UniqueName: \"kubernetes.io/projected/8906c21b-3233-4214-8ce1-f9189b79ccd3-kube-api-access-jpc58\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.565309 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.565295    6059 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 17:17:40.565344 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.565312    6059 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 17:17:40.565344 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.565321    6059 projected.go:196] Error preparing data for projected volume kube-api-access-c986b for pod openshift-console/console-5d896bf64d-smcmk: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:40.565438 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:40.565374    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:41.065349656 +0000 UTC m=+1.863062415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-c986b" (UniqueName: "kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:40.566676 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.566658    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmd9x\" (UniqueName: \"kubernetes.io/projected/c1e30c92-229f-4030-82eb-9295f5328da4-kube-api-access-bmd9x\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:17:40.566752 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.566709    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wr9f\" (UniqueName: \"kubernetes.io/projected/76f649c8-7dad-410e-a2e8-960e639099d5-kube-api-access-6wr9f\") pod \"telemeter-client-548c4c64fb-pgglg\" (UID: \"76f649c8-7dad-410e-a2e8-960e639099d5\") " pod="openshift-monitoring/telemeter-client-548c4c64fb-pgglg"
Nov 03 17:17:40.567340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.567319    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7jmf\" (UniqueName: \"kubernetes.io/projected/e5c5030e-696b-41dd-a96b-11c9eed11db2-kube-api-access-n7jmf\") pod \"dns-default-vpzpk\" (UID: \"e5c5030e-696b-41dd-a96b-11c9eed11db2\") " pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:40.568029 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.567786    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jklgn\" (UniqueName: \"kubernetes.io/projected/4f2d426d-7ee0-48d9-b960-04f7fca5dc29-kube-api-access-jklgn\") pod \"metrics-server-865c7f6b8c-gq7x9\" (UID: \"4f2d426d-7ee0-48d9-b960-04f7fca5dc29\") " pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:40.568029 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.567797    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd5nd\" (UniqueName: \"kubernetes.io/projected/4af20ec7-7ea1-470f-b238-62e0de9ca527-kube-api-access-jd5nd\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:40.568029 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.567861    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-857nt\" (UniqueName: \"kubernetes.io/projected/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-kube-api-access-857nt\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:17:40.568029 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.567876    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hljmh\" (UniqueName: \"kubernetes.io/projected/00710f2d-31bb-42da-9ae8-135ddc117df5-kube-api-access-hljmh\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.568029 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.567950    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz4kk\" (UniqueName: \"kubernetes.io/projected/8a607eca-e6ad-4e5d-aa57-663768133ec0-kube-api-access-tz4kk\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.568234 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.568055    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kx94\" (UniqueName: \"kubernetes.io/projected/e59c4c3e-b582-4b16-a6e3-4aadf0612a32-kube-api-access-2kx94\") pod \"node-resolver-6l5k8\" (UID: \"e59c4c3e-b582-4b16-a6e3-4aadf0612a32\") " pod="openshift-dns/node-resolver-6l5k8"
Nov 03 17:17:40.568234 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.568128    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jp8z\" (UniqueName: \"kubernetes.io/projected/9169e62e-57ab-4322-a0cc-664f54e48556-kube-api-access-5jp8z\") pod \"alertmanager-main-0\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:17:40.568234 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.568225    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l26gw\" (UniqueName: \"kubernetes.io/projected/117a0c38-48f5-430d-9efe-1a308f725309-kube-api-access-l26gw\") pod \"openshift-state-metrics-7bd7489957-5srqg\" (UID: \"117a0c38-48f5-430d-9efe-1a308f725309\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5srqg"
Nov 03 17:17:40.569190 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.569169    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn94x\" (UniqueName: \"kubernetes.io/projected/9f6cf431-dc3b-478e-81cd-97127a917de3-kube-api-access-qn94x\") pod \"ingress-canary-dv7x7\" (UID: \"9f6cf431-dc3b-478e-81cd-97127a917de3\") " pod="openshift-ingress-canary/ingress-canary-dv7x7"
Nov 03 17:17:40.570059 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.570039    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l8qd\" (UniqueName: \"kubernetes.io/projected/37080dcf-15b0-4d4e-85fa-ff669783f356-kube-api-access-8l8qd\") pod \"network-check-target-d252q\" (UID: \"37080dcf-15b0-4d4e-85fa-ff669783f356\") " pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:17:40.572572 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.572554    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46p7r\" (UniqueName: \"kubernetes.io/projected/08770b22-f9ff-4919-af54-c7916eba9101-kube-api-access-46p7r\") pod \"kube-state-metrics-5dc8d7d89b-c9nbc\" (UID: \"08770b22-f9ff-4919-af54-c7916eba9101\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-c9nbc"
Nov 03 17:17:40.600003 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.599962    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-device-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:17:40.600003 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600003    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/902e0389-e70e-4d23-a607-d8f4b9da868d-kubelet-config\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:17:40.600205 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600070    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:17:40.600205 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600092    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-etc-kubernetes\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.600205 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600121    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-var-lib-kubelet\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.600205 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600092    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/902e0389-e70e-4d23-a607-d8f4b9da868d-kubelet-config\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:17:40.600205 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600133    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:17:40.600205 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600143    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4af20ec7-7ea1-470f-b238-62e0de9ca527-sys\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:40.600205 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600091    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-device-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:17:40.600205 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600163    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-var-lib-kubelet\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.600205 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600151    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-etc-kubernetes\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.600205 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600166    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-systemd\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.600205 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600197    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4af20ec7-7ea1-470f-b238-62e0de9ca527-sys\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:40.600598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600215    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-run-netns\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.600598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600241    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-slash\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.600598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600257    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-systemd\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.600598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600279    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-run-netns\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.600598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600294    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-slash\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.600598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600307    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-socket-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:17:40.600598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600329    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-registration-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:17:40.600598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600345    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-socket-dir-parent\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.600598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600383    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-conf-dir\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.600598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600396    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-socket-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:17:40.600598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600414    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-cni-bin\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.600598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600419    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-socket-dir-parent\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.600598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600412    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-registration-dir\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:17:40.600598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600432    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-conf-dir\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.600598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600453    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-cni-bin\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.600598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600519    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-host\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.600598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600563    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-run-openvswitch\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.600598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600592    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-run-openvswitch\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.600598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600595    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88482077-204d-4634-8045-c741d2c10d80-host-slash\") pod \"iptables-alerter-z5v9n\" (UID: \"88482077-204d-4634-8045-c741d2c10d80\") " pod="openshift-network-operator/iptables-alerter-z5v9n"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600613    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-host\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600624    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88482077-204d-4634-8045-c741d2c10d80-host-slash\") pod \"iptables-alerter-z5v9n\" (UID: \"88482077-204d-4634-8045-c741d2c10d80\") " pod="openshift-network-operator/iptables-alerter-z5v9n"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600642    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-run\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600662    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-system-cni-dir\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600675    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-hostroot\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600693    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-sys\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600735    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-hostroot\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600740    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-system-cni-dir\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600749    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-sys\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600724    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-run-k8s-cni-cncf-io\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600770    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-run\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600798    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-run-k8s-cni-cncf-io\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600819    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-etc-selinux\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600851    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-kubelet\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600853    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-etc-selinux\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600865    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-sys-fs\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600896    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/902e0389-e70e-4d23-a607-d8f4b9da868d-dbus\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600912    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-cni-netd\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600937    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e59c4c3e-b582-4b16-a6e3-4aadf0612a32-hosts-file\") pod \"node-resolver-6l5k8\" (UID: \"e59c4c3e-b582-4b16-a6e3-4aadf0612a32\") " pod="openshift-dns/node-resolver-6l5k8"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600957    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-lib-modules\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600981    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-systemd-units\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.600987    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-cni-netd\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601001    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e59c4c3e-b582-4b16-a6e3-4aadf0612a32-hosts-file\") pod \"node-resolver-6l5k8\" (UID: \"e59c4c3e-b582-4b16-a6e3-4aadf0612a32\") " pod="openshift-dns/node-resolver-6l5k8"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601004    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/902e0389-e70e-4d23-a607-d8f4b9da868d-dbus\") pod \"global-pull-secret-syncer-rt4lh\" (UID: \"902e0389-e70e-4d23-a607-d8f4b9da868d\") " pod="kube-system/global-pull-secret-syncer-rt4lh"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601046    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-kubelet\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601062    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4af20ec7-7ea1-470f-b238-62e0de9ca527-root\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601011    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c1e30c92-229f-4030-82eb-9295f5328da4-sys-fs\") pod \"aws-ebs-csi-driver-node-c9fm2\" (UID: \"c1e30c92-229f-4030-82eb-9295f5328da4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9fm2"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601079    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-lib-modules\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601093    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d9f8f17-bb8e-4f09-ade6-099a78bfe422-host\") pod \"node-ca-bkmvr\" (UID: \"5d9f8f17-bb8e-4f09-ade6-099a78bfe422\") " pod="openshift-image-registry/node-ca-bkmvr"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601111    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-systemd-units\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601119    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-log-socket\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601124    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d9f8f17-bb8e-4f09-ade6-099a78bfe422-host\") pod \"node-ca-bkmvr\" (UID: \"5d9f8f17-bb8e-4f09-ade6-099a78bfe422\") " pod="openshift-image-registry/node-ca-bkmvr"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601140    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4af20ec7-7ea1-470f-b238-62e0de9ca527-root\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601142    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-run-netns\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601165    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-log-socket\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601173    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-sysctl-conf\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601165    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-run-netns\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601200    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-var-lib-kubelet\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601227    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-cni-dir\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601254    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-sysctl-d\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601260    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-var-lib-kubelet\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601270    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-sysctl-conf\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601282    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-cnibin\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601301    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-multus-cni-dir\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601309    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-etc-openvswitch\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601334    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-cnibin\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601333    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-system-cni-dir\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601347    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-sysctl-d\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601378    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-etc-openvswitch\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.601345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601389    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-system-cni-dir\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601399    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-run-systemd\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601443    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-var-lib-openvswitch\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601471    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-run-systemd\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601472    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-var-lib-cni-bin\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601498    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-var-lib-openvswitch\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601506    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-modprobe-d\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601514    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-var-lib-cni-bin\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601532    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-var-lib-cni-multus\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601560    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-cnibin\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601586    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-var-lib-cni-multus\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601585    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601603    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-modprobe-d\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601614    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-cnibin\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601615    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-run-ovn\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601628    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601639    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-node-log\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601651    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-run-ovn\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601660    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-crio-socket\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601690    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-kubernetes\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601715    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-node-log\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601722    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-kubernetes\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601724    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5b4340c-7e77-4d6e-bdc3-2a6193620f58-crio-socket\") pod \"insights-runtime-extractor-5gv26\" (UID: \"b5b4340c-7e77-4d6e-bdc3-2a6193620f58\") " pod="openshift-insights/insights-runtime-extractor-5gv26"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601735    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-run-ovn-kubernetes\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601765    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-wtmp\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601782    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-sysconfig\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601800    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-run-multus-certs\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601810    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8906c21b-3233-4214-8ce1-f9189b79ccd3-host-run-ovn-kubernetes\") pod \"ovnkube-node-c5n7n\" (UID: \"8906c21b-3233-4214-8ce1-f9189b79ccd3\") " pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601819    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4af20ec7-7ea1-470f-b238-62e0de9ca527-node-exporter-wtmp\") pod \"node-exporter-9gj97\" (UID: \"4af20ec7-7ea1-470f-b238-62e0de9ca527\") " pod="openshift-monitoring/node-exporter-9gj97"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601817    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601834    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a607eca-e6ad-4e5d-aa57-663768133ec0-etc-sysconfig\") pod \"tuned-4dtnl\" (UID: \"8a607eca-e6ad-4e5d-aa57-663768133ec0\") " pod="openshift-cluster-node-tuning-operator/tuned-4dtnl"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601849    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00710f2d-31bb-42da-9ae8-135ddc117df5-host-run-multus-certs\") pod \"multus-9cgng\" (UID: \"00710f2d-31bb-42da-9ae8-135ddc117df5\") " pod="openshift-multus/multus-9cgng"
Nov 03 17:17:40.602895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:40.601853    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23a7781b-32fa-4f63-ab0e-663e1f2c6abb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b99b8\" (UID: \"23a7781b-32fa-4f63-ab0e-663e1f2c6abb\") " pod="openshift-multus/multus-additional-cni-plugins-b99b8"
Nov 03 17:17:41.006822 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:41.006785    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:41.006988 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:41.006848    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:41.006988 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:41.006884    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:41.006988 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:41.006909    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:41.006988 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.006941    6059 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:41.006988 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:41.006972    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:41.007239 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.006987    6059 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:41.007239 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.007005    6059 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:41.007239 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:41.007008    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:41.007239 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.007048    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:42.00702587 +0000 UTC m=+2.804738646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:41.007239 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.007064    6059 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 17:17:41.007239 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.007083    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:42.007076313 +0000 UTC m=+2.804789073 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:41.007239 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.007093    6059 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:41.007239 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.007096    6059 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:41.007239 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.007098    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:42.007092855 +0000 UTC m=+2.804805613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:41.007239 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.007154    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:42.007138076 +0000 UTC m=+2.804850843 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"console-config" not registered
Nov 03 17:17:41.007239 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.007174    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:42.007167812 +0000 UTC m=+2.804880607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:41.007239 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.007191    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:42.007183979 +0000 UTC m=+2.804896782 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:41.007239 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:41.007230    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:41.007752 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:41.007260    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:41.007752 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:41.007294    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:41.007752 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:41.007317    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:41.007752 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.007337    6059 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 17:17:41.007752 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.007345    6059 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 17:17:41.007752 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:41.007381    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:41.007752 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.007400    6059 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 17:17:41.007752 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.007406    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:42.00739323 +0000 UTC m=+2.805106004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"service-ca" not registered
Nov 03 17:17:41.007752 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.007445    6059 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:41.007752 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:41.007481    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:41.007752 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.007455    6059 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:41.007752 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.007494    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:42.00747979 +0000 UTC m=+2.805192558 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"console-config" not registered
Nov 03 17:17:41.007752 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.007517    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:42.007508212 +0000 UTC m=+2.805220970 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"service-ca" not registered
Nov 03 17:17:41.007752 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.007535    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:42.007527734 +0000 UTC m=+2.805240494 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:41.007752 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.007539    6059 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:41.007752 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.007547    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:42.007541647 +0000 UTC m=+2.805254406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:41.007752 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.007608    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:42.00759912 +0000 UTC m=+2.805311883 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:41.108694 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:41.108650    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c986b\" (UniqueName: \"kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:41.108872 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.108840    6059 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 17:17:41.108872 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.108868    6059 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 17:17:41.109003 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.108881    6059 projected.go:196] Error preparing data for projected volume kube-api-access-c986b for pod openshift-console/console-5d896bf64d-smcmk: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:41.109003 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:41.108877    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjrtj\" (UniqueName: \"kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:41.109003 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.108941    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:42.108922402 +0000 UTC m=+2.906635178 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-c986b" (UniqueName: "kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:41.109003 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.108954    6059 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 17:17:41.109003 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.108976    6059 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 17:17:41.109003 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.108985    6059 projected.go:196] Error preparing data for projected volume kube-api-access-cjrtj for pod openshift-console/console-858bfbb7dc-x6n58: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:41.109195 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:41.109027    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:42.10901174 +0000 UTC m=+2.906724513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cjrtj" (UniqueName: "kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:41.466059 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:41.466033    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-67sr8"
Nov 03 17:17:42.018159 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:42.018118    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:42.018333 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:42.018173    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:42.018333 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:42.018224    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:42.018333 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:42.018250    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:42.018333 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.018270    6059 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:42.018333 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:42.018292    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:42.018333 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:42.018316    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:42.018659 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.018339    6059 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 17:17:42.018659 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.018383    6059 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:42.018659 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.018395    6059 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 17:17:42.018659 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.018339    6059 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 17:17:42.018659 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.018345    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:44.018326057 +0000 UTC m=+4.816038841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:42.018659 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.018343    6059 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 17:17:42.018659 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.018494    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:44.018463687 +0000 UTC m=+4.816176463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"service-ca" not registered
Nov 03 17:17:42.018659 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.018534    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:44.018520608 +0000 UTC m=+4.816233381 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:42.018659 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.018558    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:44.018548734 +0000 UTC m=+4.816261497 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"service-ca" not registered
Nov 03 17:17:42.018659 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.018583    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:44.01857356 +0000 UTC m=+4.816286324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"console-config" not registered
Nov 03 17:17:42.018659 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.018606    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:44.018597396 +0000 UTC m=+4.816310160 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"console-config" not registered
Nov 03 17:17:42.019233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:42.018685    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:42.019233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:42.018739    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:42.019233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:42.018792    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:42.019233 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.018803    6059 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:42.019233 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.018839    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:44.018824458 +0000 UTC m=+4.816537236 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:42.019233 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.018866    6059 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:42.019233 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.018889    6059 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:42.019233 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.018894    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:44.018882817 +0000 UTC m=+4.816595591 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:42.019233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:42.018962    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:42.019233 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.018989    6059 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:42.019233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:42.019000    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:42.019233 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.019014    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:44.019007611 +0000 UTC m=+4.816720370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:42.019233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:42.019035    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:42.019233 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.019045    6059 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:42.019233 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.019049    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:44.019042703 +0000 UTC m=+4.816755462 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:42.019233 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.019069    6059 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:42.019233 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.019094    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:44.019081457 +0000 UTC m=+4.816794228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:42.019233 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.019125    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:44.019114608 +0000 UTC m=+4.816827367 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:42.120128 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:42.120082    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c986b\" (UniqueName: \"kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:42.120340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:42.120216    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjrtj\" (UniqueName: \"kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:42.120340 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.120247    6059 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 17:17:42.120340 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.120273    6059 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 17:17:42.120340 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.120282    6059 projected.go:196] Error preparing data for projected volume kube-api-access-c986b for pod openshift-console/console-5d896bf64d-smcmk: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:42.120340 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.120340    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:44.120326415 +0000 UTC m=+4.918039195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-c986b" (UniqueName: "kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:42.120589 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.120390    6059 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 17:17:42.120589 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.120407    6059 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 17:17:42.120589 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.120417    6059 projected.go:196] Error preparing data for projected volume kube-api-access-cjrtj for pod openshift-console/console-858bfbb7dc-x6n58: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:42.120589 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:42.120466    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:44.120454272 +0000 UTC m=+4.918167043 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cjrtj" (UniqueName: "kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:42.904617 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:42.904585    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vpzpk"
Nov 03 17:17:44.042559 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:44.042517    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:44.042569    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:44.042600    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:44.042624    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:44.042645    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:44.042660    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.042680    6059 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:44.042718    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.042724    6059 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.042763    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:48.042741824 +0000 UTC m=+8.840454600 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.042728    6059 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.042773    6059 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.042783    6059 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.042796    6059 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.042802    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:48.042787607 +0000 UTC m=+8.840500380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.042778    6059 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.042826    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:48.042816995 +0000 UTC m=+8.840529767 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.042861    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:48.042840359 +0000 UTC m=+8.840553125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.042896    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:48.042886165 +0000 UTC m=+8.840598956 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.042912    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:48.042904173 +0000 UTC m=+8.840616934 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.042937    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:48.04292728 +0000 UTC m=+8.840640040 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:44.042956    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:44.043013    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.043038    6059 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:44.043041    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:44.043058 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.043068    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:48.043060361 +0000 UTC m=+8.840773155 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"console-config" not registered
Nov 03 17:17:44.043918 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.043091    6059 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 17:17:44.043918 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:44.043096    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:44.043918 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:44.043114    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:44.043918 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.043098    6059 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 17:17:44.043918 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.043155    6059 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 17:17:44.043918 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.043124    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:48.043115535 +0000 UTC m=+8.840828327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"service-ca" not registered
Nov 03 17:17:44.043918 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.043138    6059 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:44.043918 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.043202    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:48.043189499 +0000 UTC m=+8.840902260 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"service-ca" not registered
Nov 03 17:17:44.043918 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.043227    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:48.043218068 +0000 UTC m=+8.840930834 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"console-config" not registered
Nov 03 17:17:44.043918 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.043247    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:48.043237845 +0000 UTC m=+8.840950611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:44.144523 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:44.144480    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c986b\" (UniqueName: \"kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:44.144702 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:44.144582    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjrtj\" (UniqueName: \"kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:44.144702 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.144676    6059 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 17:17:44.144702 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.144678    6059 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 17:17:44.144846 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.144709    6059 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 17:17:44.144846 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.144688    6059 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 17:17:44.144846 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.144723    6059 projected.go:196] Error preparing data for projected volume kube-api-access-c986b for pod openshift-console/console-5d896bf64d-smcmk: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:44.144846 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.144786    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:48.144768214 +0000 UTC m=+8.942480975 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-c986b" (UniqueName: "kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:44.144846 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.144724    6059 projected.go:196] Error preparing data for projected volume kube-api-access-cjrtj for pod openshift-console/console-858bfbb7dc-x6n58: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:44.144846 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:44.144839    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:48.144824264 +0000 UTC m=+8.942537022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cjrtj" (UniqueName: "kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:44.461424 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:44.461386    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:44.493386 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:44.493342    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:17:44.636190 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:44.636153    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:44.919696 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:44.919611    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:45.250763 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:45.250728    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:45.435733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:45.435695    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:45.659031 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:45.658951    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-576dfb567c-pbb4z"
Nov 03 17:17:45.785565 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:45.785533    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-865c7f6b8c-gq7x9"
Nov 03 17:17:46.947452 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:46.947412    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:17:47.351619 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:47.351525    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:47.546835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:47.546799    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:47.935318 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:47.935278    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:47.935562 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:47.935546    6059 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 17:17:48.107895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:48.107851    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:48.108348 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:48.107918    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:48.108348 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:48.107968    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:48.108348 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:48.108009    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:48.108348 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108016    6059 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:48.108348 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:48.108043    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:48.108348 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108054    6059 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:48.108348 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:48.108075    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:48.108348 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108089    6059 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:48.108348 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108104    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:56.108082683 +0000 UTC m=+16.905795446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:48.108348 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108132    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:56.108122699 +0000 UTC m=+16.905835463 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:48.108348 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108132    6059 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:48.108348 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108151    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:56.108142113 +0000 UTC m=+16.905854875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:48.108348 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108151    6059 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:48.108348 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108137    6059 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:48.108348 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108169    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:56.108159462 +0000 UTC m=+16.905872240 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:48.108348 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108197    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:56.108185571 +0000 UTC m=+16.905898333 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:48.108348 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108215    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:56.108207425 +0000 UTC m=+16.905920195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:48.108348 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:48.108268    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:48.108348 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:48.108291    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:48.108348 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:48.108327    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:48.108348 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:48.108346    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:48.108348 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108386    6059 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:48.109074 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108394    6059 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 17:17:48.109074 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108414    6059 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 17:17:48.109074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:48.108420    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:48.109074 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108449    6059 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 17:17:48.109074 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108456    6059 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 17:17:48.109074 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108804    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:56.108415735 +0000 UTC m=+16.906128502 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:48.109074 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108839    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:56.108830608 +0000 UTC m=+16.906543368 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"console-config" not registered
Nov 03 17:17:48.109074 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108850    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:56.108844756 +0000 UTC m=+16.906557515 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"service-ca" not registered
Nov 03 17:17:48.109074 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108859    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:56.108854427 +0000 UTC m=+16.906567186 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"service-ca" not registered
Nov 03 17:17:48.109074 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108868    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:56.108863401 +0000 UTC m=+16.906576161 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"console-config" not registered
Nov 03 17:17:48.109074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:48.108882    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:48.109074 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108949    6059 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:48.109074 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.108976    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:56.108970262 +0000 UTC m=+16.906683020 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:48.210202 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:48.210169    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjrtj\" (UniqueName: \"kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:48.210395 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:48.210315    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c986b\" (UniqueName: \"kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:48.210464 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.210401    6059 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 17:17:48.210464 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.210432    6059 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 17:17:48.210464 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.210447    6059 projected.go:196] Error preparing data for projected volume kube-api-access-cjrtj for pod openshift-console/console-858bfbb7dc-x6n58: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:48.210608 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.210469    6059 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 17:17:48.210608 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.210487    6059 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 17:17:48.210608 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.210495    6059 projected.go:196] Error preparing data for projected volume kube-api-access-c986b for pod openshift-console/console-5d896bf64d-smcmk: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:48.210608 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.210527    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:56.210505067 +0000 UTC m=+17.008217846 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cjrtj" (UniqueName: "kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:48.210608 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:48.210550    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:56.210539491 +0000 UTC m=+17.008252265 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-c986b" (UniqueName: "kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:50.330025 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:50.329975    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c5n7n"
Nov 03 17:17:51.817467 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:51.817434    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:51.998407 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:51.998371    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:53.418866 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:53.418838    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dzlp4"
Nov 03 17:17:54.316803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:54.316771    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:54.506424 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:54.506384    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:55.568729 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:55.568686    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:55.760988 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:55.760953    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:56.196426 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:56.196381    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:56.196618 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:56.196457    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:56.196618 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.196484    6059 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:56.196618 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:56.196511    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:56.196618 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:56.196537    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:56.196618 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.196557    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:18:12.196542386 +0000 UTC m=+32.994255145 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:56.196618 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.196583    6059 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:56.196618 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.196601    6059 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:56.196916 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.196622    6059 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:56.196916 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.196635    6059 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:56.196916 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:56.196599    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:56.196916 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.196645    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:18:12.196630743 +0000 UTC m=+32.994343525 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:56.196916 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.196671    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:18:12.196664816 +0000 UTC m=+32.994377574 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 17:17:56.196916 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.196682    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:18:12.19667585 +0000 UTC m=+32.994388608 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:56.196916 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.196699    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:18:12.196690774 +0000 UTC m=+32.994403533 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 17:17:56.196916 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:56.196713    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:56.196916 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:56.196757    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:56.196916 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:56.196778    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:56.196916 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.196826    6059 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:56.196916 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.196848    6059 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 17:17:56.196916 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.196863    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:18:12.196851687 +0000 UTC m=+32.994564448 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:56.196916 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.196881    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:18:12.196871857 +0000 UTC m=+32.994584619 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"console-config" not registered
Nov 03 17:17:56.196916 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.196882    6059 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:56.196916 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:56.196900    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:56.197537 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:56.196932    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:56.197537 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.196962    6059 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 17:17:56.197537 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.196991    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:18:12.196972924 +0000 UTC m=+32.994685708 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 17:17:56.197537 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.197006    6059 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 17:17:56.197537 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:56.197021    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:56.197537 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.197039    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:18:12.197028744 +0000 UTC m=+32.994741506 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"console-config" not registered
Nov 03 17:17:56.197537 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.197071    6059 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 17:17:56.197537 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.197077    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:18:12.197055081 +0000 UTC m=+32.994767857 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : object "openshift-console"/"service-ca" not registered
Nov 03 17:17:56.197537 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:56.197096    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:56.197537 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.197126    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:18:12.197111957 +0000 UTC m=+32.994824734 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"service-ca" not registered
Nov 03 17:17:56.197537 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.197143    6059 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:56.197537 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.197185    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:18:12.197172715 +0000 UTC m=+32.994885493 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 17:17:56.297828 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:56.297790    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjrtj\" (UniqueName: \"kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj\") pod \"console-858bfbb7dc-x6n58\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") " pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:17:56.297982 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.297966    6059 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 17:17:56.298022 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.297991    6059 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 17:17:56.298022 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.298004    6059 projected.go:196] Error preparing data for projected volume kube-api-access-cjrtj for pod openshift-console/console-858bfbb7dc-x6n58: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:56.298084 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.298029    6059 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 17:17:56.298084 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.298047    6059 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 17:17:56.298084 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.298056    6059 projected.go:196] Error preparing data for projected volume kube-api-access-c986b for pod openshift-console/console-5d896bf64d-smcmk: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:56.298084 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.298074    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj podName:8bfb6a7d-e8f9-434f-acfc-8ac3badf1350 nodeName:}" failed. No retries permitted until 2026-11-03 17:18:12.298055063 +0000 UTC m=+33.095767844 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cjrtj" (UniqueName: "kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj") pod "console-858bfbb7dc-x6n58" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:56.298084 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:56.297966    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c986b\" (UniqueName: \"kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b\") pod \"console-5d896bf64d-smcmk\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") " pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:17:56.298242 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:56.298092    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b podName:00c39778-f1f8-4539-88eb-40d082abdc50 nodeName:}" failed. No retries permitted until 2026-11-03 17:18:12.298083861 +0000 UTC m=+33.095796632 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-c986b" (UniqueName: "kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b") pod "console-5d896bf64d-smcmk" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 17:17:57.787827 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:57.787746    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:57.977232 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:57.977188    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:17:59.538956 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:59.538920    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 17:17:59.539452 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:59.539309    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="alertmanager" containerID="cri-o://2f1076bf2750c96b77091df003f21ace6b1f66ab2421ee9a735a8fbc29a894ba" gracePeriod=120
Nov 03 17:17:59.539452 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:59.539379    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="kube-rbac-proxy-metric" containerID="cri-o://c271fb7eebecdd673f7e04715d9e1b3404251b9c25d93f2885787af4c4a2024f" gracePeriod=120
Nov 03 17:17:59.539452 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:59.539395    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="config-reloader" containerID="cri-o://e11f53e8a8e7e9b54a01c8a29f6f7d732cf76dac01bf5d052665cdf3774d9abd" gracePeriod=120
Nov 03 17:17:59.539452 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:59.539379    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="kube-rbac-proxy-web" containerID="cri-o://3598a1389cbaab523929e53df155a86a6e5b2f3f4f1e4a9fcddb5a6c10b9353f" gracePeriod=120
Nov 03 17:17:59.539657 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:59.539487    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="kube-rbac-proxy" containerID="cri-o://c10696182467d9166863ec5815f265ce29cfbac9cd90d29cde647734e12ac50c" gracePeriod=120
Nov 03 17:17:59.539657 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:17:59.539512    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="prom-label-proxy" containerID="cri-o://e439cf07cc380d37b78cd073f932ebe8f830fc79b69591e39985c762935b3a8c" gracePeriod=120
Nov 03 17:17:59.564224 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:17:59.564184    6059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9169e62e_57ab_4322_a0cc_664f54e48556.slice/crio-c10696182467d9166863ec5815f265ce29cfbac9cd90d29cde647734e12ac50c.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 17:18:00.268023 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.267990    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:18:00.454283 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.454248    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:18:00.491029 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.490995    6059 generic.go:356] "Generic (PLEG): container finished" podID="9169e62e-57ab-4322-a0cc-664f54e48556" containerID="e439cf07cc380d37b78cd073f932ebe8f830fc79b69591e39985c762935b3a8c" exitCode=0
Nov 03 17:18:00.491029 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.491021    6059 generic.go:356] "Generic (PLEG): container finished" podID="9169e62e-57ab-4322-a0cc-664f54e48556" containerID="c271fb7eebecdd673f7e04715d9e1b3404251b9c25d93f2885787af4c4a2024f" exitCode=0
Nov 03 17:18:00.491029 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.491029    6059 generic.go:356] "Generic (PLEG): container finished" podID="9169e62e-57ab-4322-a0cc-664f54e48556" containerID="c10696182467d9166863ec5815f265ce29cfbac9cd90d29cde647734e12ac50c" exitCode=0
Nov 03 17:18:00.491029 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.491037    6059 generic.go:356] "Generic (PLEG): container finished" podID="9169e62e-57ab-4322-a0cc-664f54e48556" containerID="e11f53e8a8e7e9b54a01c8a29f6f7d732cf76dac01bf5d052665cdf3774d9abd" exitCode=0
Nov 03 17:18:00.491289 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.491044    6059 generic.go:356] "Generic (PLEG): container finished" podID="9169e62e-57ab-4322-a0cc-664f54e48556" containerID="2f1076bf2750c96b77091df003f21ace6b1f66ab2421ee9a735a8fbc29a894ba" exitCode=0
Nov 03 17:18:00.491289 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.491064    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9169e62e-57ab-4322-a0cc-664f54e48556","Type":"ContainerDied","Data":"e439cf07cc380d37b78cd073f932ebe8f830fc79b69591e39985c762935b3a8c"}
Nov 03 17:18:00.491289 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.491158    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9169e62e-57ab-4322-a0cc-664f54e48556","Type":"ContainerDied","Data":"c271fb7eebecdd673f7e04715d9e1b3404251b9c25d93f2885787af4c4a2024f"}
Nov 03 17:18:00.491289 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.491189    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9169e62e-57ab-4322-a0cc-664f54e48556","Type":"ContainerDied","Data":"c10696182467d9166863ec5815f265ce29cfbac9cd90d29cde647734e12ac50c"}
Nov 03 17:18:00.491289 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.491212    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9169e62e-57ab-4322-a0cc-664f54e48556","Type":"ContainerDied","Data":"e11f53e8a8e7e9b54a01c8a29f6f7d732cf76dac01bf5d052665cdf3774d9abd"}
Nov 03 17:18:00.491289 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.491234    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9169e62e-57ab-4322-a0cc-664f54e48556","Type":"ContainerDied","Data":"2f1076bf2750c96b77091df003f21ace6b1f66ab2421ee9a735a8fbc29a894ba"}
Nov 03 17:18:00.805255 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.805225    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:18:00.846537 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.846504    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9169e62e-57ab-4322-a0cc-664f54e48556-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9169e62e-57ab-4322-a0cc-664f54e48556-alertmanager-main-db\") pod \"9169e62e-57ab-4322-a0cc-664f54e48556\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") "
Nov 03 17:18:00.846717 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.846545    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9169e62e-57ab-4322-a0cc-664f54e48556-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9169e62e-57ab-4322-a0cc-664f54e48556-metrics-client-ca\") pod \"9169e62e-57ab-4322-a0cc-664f54e48556\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") "
Nov 03 17:18:00.846717 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.846573    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-config-volume\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-config-volume\") pod \"9169e62e-57ab-4322-a0cc-664f54e48556\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") "
Nov 03 17:18:00.846717 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.846592    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9169e62e-57ab-4322-a0cc-664f54e48556-kube-api-access-5jp8z\" (UniqueName: \"kubernetes.io/projected/9169e62e-57ab-4322-a0cc-664f54e48556-kube-api-access-5jp8z\") pod \"9169e62e-57ab-4322-a0cc-664f54e48556\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") "
Nov 03 17:18:00.846717 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.846640    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-main-tls\") pod \"9169e62e-57ab-4322-a0cc-664f54e48556\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") "
Nov 03 17:18:00.846717 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.846673    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9169e62e-57ab-4322-a0cc-664f54e48556-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9169e62e-57ab-4322-a0cc-664f54e48556-alertmanager-trusted-ca-bundle\") pod \"9169e62e-57ab-4322-a0cc-664f54e48556\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") "
Nov 03 17:18:00.846717 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.846699    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy-metric\") pod \"9169e62e-57ab-4322-a0cc-664f54e48556\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") "
Nov 03 17:18:00.847750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.846735    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9169e62e-57ab-4322-a0cc-664f54e48556-tls-assets\" (UniqueName: \"kubernetes.io/projected/9169e62e-57ab-4322-a0cc-664f54e48556-tls-assets\") pod \"9169e62e-57ab-4322-a0cc-664f54e48556\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") "
Nov 03 17:18:00.847750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.846774    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-cluster-tls-config\") pod \"9169e62e-57ab-4322-a0cc-664f54e48556\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") "
Nov 03 17:18:00.847750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.846824    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy\") pod \"9169e62e-57ab-4322-a0cc-664f54e48556\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") "
Nov 03 17:18:00.847750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.846858    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-web-config\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-web-config\") pod \"9169e62e-57ab-4322-a0cc-664f54e48556\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") "
Nov 03 17:18:00.847750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.846897    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9169e62e-57ab-4322-a0cc-664f54e48556-config-out\" (UniqueName: \"kubernetes.io/empty-dir/9169e62e-57ab-4322-a0cc-664f54e48556-config-out\") pod \"9169e62e-57ab-4322-a0cc-664f54e48556\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") "
Nov 03 17:18:00.847750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.846952    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy-web\") pod \"9169e62e-57ab-4322-a0cc-664f54e48556\" (UID: \"9169e62e-57ab-4322-a0cc-664f54e48556\") "
Nov 03 17:18:00.847750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.846970    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9169e62e-57ab-4322-a0cc-664f54e48556-alertmanager-main-db" pod "9169e62e-57ab-4322-a0cc-664f54e48556" (UID: "9169e62e-57ab-4322-a0cc-664f54e48556"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:18:00.847750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.847002    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9169e62e-57ab-4322-a0cc-664f54e48556-metrics-client-ca" pod "9169e62e-57ab-4322-a0cc-664f54e48556" (UID: "9169e62e-57ab-4322-a0cc-664f54e48556"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:18:00.847750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.847241    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9169e62e-57ab-4322-a0cc-664f54e48556-alertmanager-trusted-ca-bundle" pod "9169e62e-57ab-4322-a0cc-664f54e48556" (UID: "9169e62e-57ab-4322-a0cc-664f54e48556"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:18:00.847750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.847601    6059 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9169e62e-57ab-4322-a0cc-664f54e48556-alertmanager-main-db\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:00.847750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.847626    6059 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9169e62e-57ab-4322-a0cc-664f54e48556-metrics-client-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:00.847750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.847644    6059 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9169e62e-57ab-4322-a0cc-664f54e48556-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:00.850733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.850684    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy-web" pod "9169e62e-57ab-4322-a0cc-664f54e48556" (UID: "9169e62e-57ab-4322-a0cc-664f54e48556"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:18:00.850846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.850712    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9169e62e-57ab-4322-a0cc-664f54e48556-kube-api-access-5jp8z" pod "9169e62e-57ab-4322-a0cc-664f54e48556" (UID: "9169e62e-57ab-4322-a0cc-664f54e48556"). InnerVolumeSpecName "kube-api-access-5jp8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:18:00.850994 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.850952    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy" pod "9169e62e-57ab-4322-a0cc-664f54e48556" (UID: "9169e62e-57ab-4322-a0cc-664f54e48556"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:18:00.851074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.851049    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9169e62e-57ab-4322-a0cc-664f54e48556-tls-assets" pod "9169e62e-57ab-4322-a0cc-664f54e48556" (UID: "9169e62e-57ab-4322-a0cc-664f54e48556"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:18:00.851159 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.851136    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-config-volume" pod "9169e62e-57ab-4322-a0cc-664f54e48556" (UID: "9169e62e-57ab-4322-a0cc-664f54e48556"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:18:00.851270 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.851237    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy-metric" pod "9169e62e-57ab-4322-a0cc-664f54e48556" (UID: "9169e62e-57ab-4322-a0cc-664f54e48556"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:18:00.851440 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.851390    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-main-tls" pod "9169e62e-57ab-4322-a0cc-664f54e48556" (UID: "9169e62e-57ab-4322-a0cc-664f54e48556"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:18:00.851633 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.851612    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9169e62e-57ab-4322-a0cc-664f54e48556-config-out" pod "9169e62e-57ab-4322-a0cc-664f54e48556" (UID: "9169e62e-57ab-4322-a0cc-664f54e48556"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:18:00.855422 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.855332    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-cluster-tls-config" pod "9169e62e-57ab-4322-a0cc-664f54e48556" (UID: "9169e62e-57ab-4322-a0cc-664f54e48556"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:18:00.862561 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.862492    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-web-config" pod "9169e62e-57ab-4322-a0cc-664f54e48556" (UID: "9169e62e-57ab-4322-a0cc-664f54e48556"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:18:00.948738 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.948693    6059 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:00.948738 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.948733    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jp8z\" (UniqueName: \"kubernetes.io/projected/9169e62e-57ab-4322-a0cc-664f54e48556-kube-api-access-5jp8z\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:00.948738 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.948744    6059 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-main-tls\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:00.948979 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.948754    6059 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:00.948979 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.948765    6059 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9169e62e-57ab-4322-a0cc-664f54e48556-tls-assets\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:00.948979 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.948775    6059 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-cluster-tls-config\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:00.948979 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.948785    6059 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:00.948979 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.948794    6059 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-web-config\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:00.948979 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.948804    6059 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9169e62e-57ab-4322-a0cc-664f54e48556-config-out\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:00.948979 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:00.948814    6059 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9169e62e-57ab-4322-a0cc-664f54e48556-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:01.498182 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.498136    6059 generic.go:356] "Generic (PLEG): container finished" podID="9169e62e-57ab-4322-a0cc-664f54e48556" containerID="3598a1389cbaab523929e53df155a86a6e5b2f3f4f1e4a9fcddb5a6c10b9353f" exitCode=0
Nov 03 17:18:01.498371 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.498177    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9169e62e-57ab-4322-a0cc-664f54e48556","Type":"ContainerDied","Data":"3598a1389cbaab523929e53df155a86a6e5b2f3f4f1e4a9fcddb5a6c10b9353f"}
Nov 03 17:18:01.498371 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.498213    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 17:18:01.498371 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.498258    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9169e62e-57ab-4322-a0cc-664f54e48556","Type":"ContainerDied","Data":"106da788c38a5f644bcb471f6573512cea9bc0dc229b82880ddb8dfde42b5f3e"}
Nov 03 17:18:01.498371 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.498294    6059 scope.go:122] "RemoveContainer" containerID="e439cf07cc380d37b78cd073f932ebe8f830fc79b69591e39985c762935b3a8c"
Nov 03 17:18:01.506536 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.506513    6059 scope.go:122] "RemoveContainer" containerID="c271fb7eebecdd673f7e04715d9e1b3404251b9c25d93f2885787af4c4a2024f"
Nov 03 17:18:01.514170 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.514131    6059 scope.go:122] "RemoveContainer" containerID="c10696182467d9166863ec5815f265ce29cfbac9cd90d29cde647734e12ac50c"
Nov 03 17:18:01.521335 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.521309    6059 scope.go:122] "RemoveContainer" containerID="3598a1389cbaab523929e53df155a86a6e5b2f3f4f1e4a9fcddb5a6c10b9353f"
Nov 03 17:18:01.528426 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.528404    6059 scope.go:122] "RemoveContainer" containerID="e11f53e8a8e7e9b54a01c8a29f6f7d732cf76dac01bf5d052665cdf3774d9abd"
Nov 03 17:18:01.535902 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.535878    6059 scope.go:122] "RemoveContainer" containerID="2f1076bf2750c96b77091df003f21ace6b1f66ab2421ee9a735a8fbc29a894ba"
Nov 03 17:18:01.543453 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.543433    6059 scope.go:122] "RemoveContainer" containerID="05f926711641665314245ac2f0f5f1eb8386119ada7a587f9810c9f3d363c1b9"
Nov 03 17:18:01.551107 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.551088    6059 scope.go:122] "RemoveContainer" containerID="e439cf07cc380d37b78cd073f932ebe8f830fc79b69591e39985c762935b3a8c"
Nov 03 17:18:01.551506 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:18:01.551478    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e439cf07cc380d37b78cd073f932ebe8f830fc79b69591e39985c762935b3a8c\": container with ID starting with e439cf07cc380d37b78cd073f932ebe8f830fc79b69591e39985c762935b3a8c not found: ID does not exist" containerID="e439cf07cc380d37b78cd073f932ebe8f830fc79b69591e39985c762935b3a8c"
Nov 03 17:18:01.551553 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.551518    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e439cf07cc380d37b78cd073f932ebe8f830fc79b69591e39985c762935b3a8c"} err="failed to get container status \"e439cf07cc380d37b78cd073f932ebe8f830fc79b69591e39985c762935b3a8c\": rpc error: code = NotFound desc = could not find container \"e439cf07cc380d37b78cd073f932ebe8f830fc79b69591e39985c762935b3a8c\": container with ID starting with e439cf07cc380d37b78cd073f932ebe8f830fc79b69591e39985c762935b3a8c not found: ID does not exist"
Nov 03 17:18:01.551641 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.551556    6059 scope.go:122] "RemoveContainer" containerID="c271fb7eebecdd673f7e04715d9e1b3404251b9c25d93f2885787af4c4a2024f"
Nov 03 17:18:01.551800 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:18:01.551781    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c271fb7eebecdd673f7e04715d9e1b3404251b9c25d93f2885787af4c4a2024f\": container with ID starting with c271fb7eebecdd673f7e04715d9e1b3404251b9c25d93f2885787af4c4a2024f not found: ID does not exist" containerID="c271fb7eebecdd673f7e04715d9e1b3404251b9c25d93f2885787af4c4a2024f"
Nov 03 17:18:01.551842 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.551808    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c271fb7eebecdd673f7e04715d9e1b3404251b9c25d93f2885787af4c4a2024f"} err="failed to get container status \"c271fb7eebecdd673f7e04715d9e1b3404251b9c25d93f2885787af4c4a2024f\": rpc error: code = NotFound desc = could not find container \"c271fb7eebecdd673f7e04715d9e1b3404251b9c25d93f2885787af4c4a2024f\": container with ID starting with c271fb7eebecdd673f7e04715d9e1b3404251b9c25d93f2885787af4c4a2024f not found: ID does not exist"
Nov 03 17:18:01.551842 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.551828    6059 scope.go:122] "RemoveContainer" containerID="c10696182467d9166863ec5815f265ce29cfbac9cd90d29cde647734e12ac50c"
Nov 03 17:18:01.552059 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:18:01.552041    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c10696182467d9166863ec5815f265ce29cfbac9cd90d29cde647734e12ac50c\": container with ID starting with c10696182467d9166863ec5815f265ce29cfbac9cd90d29cde647734e12ac50c not found: ID does not exist" containerID="c10696182467d9166863ec5815f265ce29cfbac9cd90d29cde647734e12ac50c"
Nov 03 17:18:01.552132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.552069    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c10696182467d9166863ec5815f265ce29cfbac9cd90d29cde647734e12ac50c"} err="failed to get container status \"c10696182467d9166863ec5815f265ce29cfbac9cd90d29cde647734e12ac50c\": rpc error: code = NotFound desc = could not find container \"c10696182467d9166863ec5815f265ce29cfbac9cd90d29cde647734e12ac50c\": container with ID starting with c10696182467d9166863ec5815f265ce29cfbac9cd90d29cde647734e12ac50c not found: ID does not exist"
Nov 03 17:18:01.552132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.552093    6059 scope.go:122] "RemoveContainer" containerID="3598a1389cbaab523929e53df155a86a6e5b2f3f4f1e4a9fcddb5a6c10b9353f"
Nov 03 17:18:01.552346 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:18:01.552329    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3598a1389cbaab523929e53df155a86a6e5b2f3f4f1e4a9fcddb5a6c10b9353f\": container with ID starting with 3598a1389cbaab523929e53df155a86a6e5b2f3f4f1e4a9fcddb5a6c10b9353f not found: ID does not exist" containerID="3598a1389cbaab523929e53df155a86a6e5b2f3f4f1e4a9fcddb5a6c10b9353f"
Nov 03 17:18:01.552428 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.552374    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3598a1389cbaab523929e53df155a86a6e5b2f3f4f1e4a9fcddb5a6c10b9353f"} err="failed to get container status \"3598a1389cbaab523929e53df155a86a6e5b2f3f4f1e4a9fcddb5a6c10b9353f\": rpc error: code = NotFound desc = could not find container \"3598a1389cbaab523929e53df155a86a6e5b2f3f4f1e4a9fcddb5a6c10b9353f\": container with ID starting with 3598a1389cbaab523929e53df155a86a6e5b2f3f4f1e4a9fcddb5a6c10b9353f not found: ID does not exist"
Nov 03 17:18:01.552428 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.552395    6059 scope.go:122] "RemoveContainer" containerID="e11f53e8a8e7e9b54a01c8a29f6f7d732cf76dac01bf5d052665cdf3774d9abd"
Nov 03 17:18:01.552655 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:18:01.552636    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e11f53e8a8e7e9b54a01c8a29f6f7d732cf76dac01bf5d052665cdf3774d9abd\": container with ID starting with e11f53e8a8e7e9b54a01c8a29f6f7d732cf76dac01bf5d052665cdf3774d9abd not found: ID does not exist" containerID="e11f53e8a8e7e9b54a01c8a29f6f7d732cf76dac01bf5d052665cdf3774d9abd"
Nov 03 17:18:01.552692 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.552663    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e11f53e8a8e7e9b54a01c8a29f6f7d732cf76dac01bf5d052665cdf3774d9abd"} err="failed to get container status \"e11f53e8a8e7e9b54a01c8a29f6f7d732cf76dac01bf5d052665cdf3774d9abd\": rpc error: code = NotFound desc = could not find container \"e11f53e8a8e7e9b54a01c8a29f6f7d732cf76dac01bf5d052665cdf3774d9abd\": container with ID starting with e11f53e8a8e7e9b54a01c8a29f6f7d732cf76dac01bf5d052665cdf3774d9abd not found: ID does not exist"
Nov 03 17:18:01.552692 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.552678    6059 scope.go:122] "RemoveContainer" containerID="2f1076bf2750c96b77091df003f21ace6b1f66ab2421ee9a735a8fbc29a894ba"
Nov 03 17:18:01.552885 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:18:01.552868    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f1076bf2750c96b77091df003f21ace6b1f66ab2421ee9a735a8fbc29a894ba\": container with ID starting with 2f1076bf2750c96b77091df003f21ace6b1f66ab2421ee9a735a8fbc29a894ba not found: ID does not exist" containerID="2f1076bf2750c96b77091df003f21ace6b1f66ab2421ee9a735a8fbc29a894ba"
Nov 03 17:18:01.552956 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.552892    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f1076bf2750c96b77091df003f21ace6b1f66ab2421ee9a735a8fbc29a894ba"} err="failed to get container status \"2f1076bf2750c96b77091df003f21ace6b1f66ab2421ee9a735a8fbc29a894ba\": rpc error: code = NotFound desc = could not find container \"2f1076bf2750c96b77091df003f21ace6b1f66ab2421ee9a735a8fbc29a894ba\": container with ID starting with 2f1076bf2750c96b77091df003f21ace6b1f66ab2421ee9a735a8fbc29a894ba not found: ID does not exist"
Nov 03 17:18:01.552956 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.552910    6059 scope.go:122] "RemoveContainer" containerID="05f926711641665314245ac2f0f5f1eb8386119ada7a587f9810c9f3d363c1b9"
Nov 03 17:18:01.553142 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:18:01.553127    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05f926711641665314245ac2f0f5f1eb8386119ada7a587f9810c9f3d363c1b9\": container with ID starting with 05f926711641665314245ac2f0f5f1eb8386119ada7a587f9810c9f3d363c1b9 not found: ID does not exist" containerID="05f926711641665314245ac2f0f5f1eb8386119ada7a587f9810c9f3d363c1b9"
Nov 03 17:18:01.553186 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.553148    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f926711641665314245ac2f0f5f1eb8386119ada7a587f9810c9f3d363c1b9"} err="failed to get container status \"05f926711641665314245ac2f0f5f1eb8386119ada7a587f9810c9f3d363c1b9\": rpc error: code = NotFound desc = could not find container \"05f926711641665314245ac2f0f5f1eb8386119ada7a587f9810c9f3d363c1b9\": container with ID starting with 05f926711641665314245ac2f0f5f1eb8386119ada7a587f9810c9f3d363c1b9 not found: ID does not exist"
Nov 03 17:18:01.705221 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.705183    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 17:18:01.752646 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:01.752574    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 17:18:02.388125 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:02.388093    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:18:02.574702 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:02.574665    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:18:03.333745 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:03.333713    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" path="/var/lib/kubelet/pods/9169e62e-57ab-4322-a0cc-664f54e48556/volumes"
Nov 03 17:18:05.374429 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.374390    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 17:18:05.374886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.374678    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="prometheus" containerID="cri-o://2422aa3f96a75eba8798b0f2d19e59a639aefbc0f2868f1381c755c865ff3ba9" gracePeriod=600
Nov 03 17:18:05.374886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.374714    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="kube-rbac-proxy" containerID="cri-o://71598ebd4a04a726ae4252ab94a67843e6d610fca75c24d900bb0310185b1726" gracePeriod=600
Nov 03 17:18:05.374886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.374760    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="kube-rbac-proxy-web" containerID="cri-o://235be22017efeb10a48b4d98ec7fc426552e2ab28c203c4df0025046fe88d1fe" gracePeriod=600
Nov 03 17:18:05.374886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.374794    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="kube-rbac-proxy-thanos" containerID="cri-o://807ab55e21372b935ec0f6b61e2a61638523471c8c0f6721ea9fc4fc77bc50c2" gracePeriod=600
Nov 03 17:18:05.374886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.374802    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="config-reloader" containerID="cri-o://b1b3f036a96309db13f2935405fe341cd62324aec92dacbdcb8265281afa00d5" gracePeriod=600
Nov 03 17:18:05.375109 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.374769    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="thanos-sidecar" containerID="cri-o://97a0cb86319a8612ce3b35d0a295dc48bd55c92e0b0ee51d9338248cb5503115" gracePeriod=600
Nov 03 17:18:05.414914 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.414871    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5d896bf64d-smcmk" podUID="00c39778-f1f8-4539-88eb-40d082abdc50" containerName="console" containerID="cri-o://5ce1bc06c7ca0db7013a0414fafca270db33f2b5d1d25f5a2d92d842e969d3c5" gracePeriod=15
Nov 03 17:18:05.415083 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.414901    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-858bfbb7dc-x6n58" podUID="8bfb6a7d-e8f9-434f-acfc-8ac3badf1350" containerName="console" containerID="cri-o://bd2c1a3533d2ff1149b8ec0fe0ee8605177b76f09ff1efc3e019fcdf9fc504af" gracePeriod=15
Nov 03 17:18:05.517009 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.516977    6059 generic.go:356] "Generic (PLEG): container finished" podID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerID="807ab55e21372b935ec0f6b61e2a61638523471c8c0f6721ea9fc4fc77bc50c2" exitCode=0
Nov 03 17:18:05.517009 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.517006    6059 generic.go:356] "Generic (PLEG): container finished" podID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerID="71598ebd4a04a726ae4252ab94a67843e6d610fca75c24d900bb0310185b1726" exitCode=0
Nov 03 17:18:05.517009 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.517012    6059 generic.go:356] "Generic (PLEG): container finished" podID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerID="235be22017efeb10a48b4d98ec7fc426552e2ab28c203c4df0025046fe88d1fe" exitCode=0
Nov 03 17:18:05.517009 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.517018    6059 generic.go:356] "Generic (PLEG): container finished" podID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerID="97a0cb86319a8612ce3b35d0a295dc48bd55c92e0b0ee51d9338248cb5503115" exitCode=0
Nov 03 17:18:05.517269 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.517026    6059 generic.go:356] "Generic (PLEG): container finished" podID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerID="b1b3f036a96309db13f2935405fe341cd62324aec92dacbdcb8265281afa00d5" exitCode=0
Nov 03 17:18:05.517269 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.517031    6059 generic.go:356] "Generic (PLEG): container finished" podID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerID="2422aa3f96a75eba8798b0f2d19e59a639aefbc0f2868f1381c755c865ff3ba9" exitCode=0
Nov 03 17:18:05.517269 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.517048    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f46ad168-4931-41e7-ac08-f6fa965721f9","Type":"ContainerDied","Data":"807ab55e21372b935ec0f6b61e2a61638523471c8c0f6721ea9fc4fc77bc50c2"}
Nov 03 17:18:05.517269 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.517111    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f46ad168-4931-41e7-ac08-f6fa965721f9","Type":"ContainerDied","Data":"71598ebd4a04a726ae4252ab94a67843e6d610fca75c24d900bb0310185b1726"}
Nov 03 17:18:05.517269 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.517170    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f46ad168-4931-41e7-ac08-f6fa965721f9","Type":"ContainerDied","Data":"235be22017efeb10a48b4d98ec7fc426552e2ab28c203c4df0025046fe88d1fe"}
Nov 03 17:18:05.517269 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.517193    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f46ad168-4931-41e7-ac08-f6fa965721f9","Type":"ContainerDied","Data":"97a0cb86319a8612ce3b35d0a295dc48bd55c92e0b0ee51d9338248cb5503115"}
Nov 03 17:18:05.517269 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.517217    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f46ad168-4931-41e7-ac08-f6fa965721f9","Type":"ContainerDied","Data":"b1b3f036a96309db13f2935405fe341cd62324aec92dacbdcb8265281afa00d5"}
Nov 03 17:18:05.517269 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.517242    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f46ad168-4931-41e7-ac08-f6fa965721f9","Type":"ContainerDied","Data":"2422aa3f96a75eba8798b0f2d19e59a639aefbc0f2868f1381c755c865ff3ba9"}
Nov 03 17:18:05.697017 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.696986    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:05.731517 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.731495    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:18:05.736771 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.736750    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:18:05.801835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.801802    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-kubelet-serving-ca-bundle\") pod \"f46ad168-4931-41e7-ac08-f6fa965721f9\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") "
Nov 03 17:18:05.802034 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.801850    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-config\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-config\") pod \"f46ad168-4931-41e7-ac08-f6fa965721f9\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") "
Nov 03 17:18:05.802034 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.801887    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert\") pod \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") "
Nov 03 17:18:05.802034 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.801923    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46ad168-4931-41e7-ac08-f6fa965721f9-tls-assets\" (UniqueName: \"kubernetes.io/projected/f46ad168-4931-41e7-ac08-f6fa965721f9-tls-assets\") pod \"f46ad168-4931-41e7-ac08-f6fa965721f9\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") "
Nov 03 17:18:05.802034 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.801955    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle\") pod \"00c39778-f1f8-4539-88eb-40d082abdc50\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") "
Nov 03 17:18:05.802034 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.801984    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f46ad168-4931-41e7-ac08-f6fa965721f9\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") "
Nov 03 17:18:05.802034 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802020    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-2\") pod \"f46ad168-4931-41e7-ac08-f6fa965721f9\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") "
Nov 03 17:18:05.802340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802054    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config\") pod \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") "
Nov 03 17:18:05.802340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802087    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca\") pod \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") "
Nov 03 17:18:05.802340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802126    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-serving-certs-ca-bundle\") pod \"f46ad168-4931-41e7-ac08-f6fa965721f9\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") "
Nov 03 17:18:05.802340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802161    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-kube-rbac-proxy\") pod \"f46ad168-4931-41e7-ac08-f6fa965721f9\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") "
Nov 03 17:18:05.802340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802197    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert\") pod \"00c39778-f1f8-4539-88eb-40d082abdc50\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") "
Nov 03 17:18:05.802340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802229    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-thanos-prometheus-http-client-file\") pod \"f46ad168-4931-41e7-ac08-f6fa965721f9\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") "
Nov 03 17:18:05.802340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802259    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj\" (UniqueName: \"kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj\") pod \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") "
Nov 03 17:18:05.802340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802289    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c39778-f1f8-4539-88eb-40d082abdc50-tmp\" (UniqueName: \"kubernetes.io/empty-dir/00c39778-f1f8-4539-88eb-40d082abdc50-tmp\") pod \"00c39778-f1f8-4539-88eb-40d082abdc50\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") "
Nov 03 17:18:05.802340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802297    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-kubelet-serving-ca-bundle" pod "f46ad168-4931-41e7-ac08-f6fa965721f9" (UID: "f46ad168-4931-41e7-ac08-f6fa965721f9"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:18:05.802340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802317    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle" pod "00c39778-f1f8-4539-88eb-40d082abdc50" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:18:05.802340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802318    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-db\") pod \"f46ad168-4931-41e7-ac08-f6fa965721f9\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") "
Nov 03 17:18:05.802888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802410    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-metrics-client-certs\") pod \"f46ad168-4931-41e7-ac08-f6fa965721f9\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") "
Nov 03 17:18:05.802888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802454    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config\") pod \"00c39778-f1f8-4539-88eb-40d082abdc50\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") "
Nov 03 17:18:05.802888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802489    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-trusted-ca-bundle\") pod \"f46ad168-4931-41e7-ac08-f6fa965721f9\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") "
Nov 03 17:18:05.802888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802649    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-web-config\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-web-config\") pod \"f46ad168-4931-41e7-ac08-f6fa965721f9\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") "
Nov 03 17:18:05.802888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802689    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert\") pod \"00c39778-f1f8-4539-88eb-40d082abdc50\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") "
Nov 03 17:18:05.802888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802727    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert\") pod \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") "
Nov 03 17:18:05.802888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802780    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config\") pod \"00c39778-f1f8-4539-88eb-40d082abdc50\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") "
Nov 03 17:18:05.802888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802813    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-tmp\") pod \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") "
Nov 03 17:18:05.802888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802851    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-0\") pod \"f46ad168-4931-41e7-ac08-f6fa965721f9\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") "
Nov 03 17:18:05.802888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802884    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46ad168-4931-41e7-ac08-f6fa965721f9-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f46ad168-4931-41e7-ac08-f6fa965721f9-config-out\") pod \"f46ad168-4931-41e7-ac08-f6fa965721f9\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") "
Nov 03 17:18:05.803394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802926    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca\") pod \"00c39778-f1f8-4539-88eb-40d082abdc50\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") "
Nov 03 17:18:05.803394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.802956    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f46ad168-4931-41e7-ac08-f6fa965721f9\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") "
Nov 03 17:18:05.803394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.803013    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config\") pod \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") "
Nov 03 17:18:05.803394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.803044    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-tls\") pod \"f46ad168-4931-41e7-ac08-f6fa965721f9\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") "
Nov 03 17:18:05.803394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.803077    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-1\") pod \"f46ad168-4931-41e7-ac08-f6fa965721f9\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") "
Nov 03 17:18:05.803394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.803113    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46ad168-4931-41e7-ac08-f6fa965721f9-kube-api-access-q4sxz\" (UniqueName: \"kubernetes.io/projected/f46ad168-4931-41e7-ac08-f6fa965721f9-kube-api-access-q4sxz\") pod \"f46ad168-4931-41e7-ac08-f6fa965721f9\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") "
Nov 03 17:18:05.803394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.803148    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b\" (UniqueName: \"kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b\") pod \"00c39778-f1f8-4539-88eb-40d082abdc50\" (UID: \"00c39778-f1f8-4539-88eb-40d082abdc50\") "
Nov 03 17:18:05.803394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.803187    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-metrics-client-ca\") pod \"f46ad168-4931-41e7-ac08-f6fa965721f9\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") "
Nov 03 17:18:05.803394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.803222    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle\") pod \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\" (UID: \"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350\") "
Nov 03 17:18:05.803394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.803263    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-grpc-tls\") pod \"f46ad168-4931-41e7-ac08-f6fa965721f9\" (UID: \"f46ad168-4931-41e7-ac08-f6fa965721f9\") "
Nov 03 17:18:05.803394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.803324    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-tmp" pod "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:18:05.803950 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.803409    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-2" pod "f46ad168-4931-41e7-ac08-f6fa965721f9" (UID: "f46ad168-4931-41e7-ac08-f6fa965721f9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:18:05.803950 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.803653    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert" pod "00c39778-f1f8-4539-88eb-40d082abdc50" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:18:05.803950 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.803678    6059 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-tmp\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.803950 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.803704    6059 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.803950 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.803721    6059 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-trusted-ca-bundle\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.803950 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.803737    6059 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.803950 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.803780    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-db" pod "f46ad168-4931-41e7-ac08-f6fa965721f9" (UID: "f46ad168-4931-41e7-ac08-f6fa965721f9"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:18:05.804335 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.804030    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert" pod "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:18:05.804335 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.804039    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c39778-f1f8-4539-88eb-40d082abdc50-tmp" pod "00c39778-f1f8-4539-88eb-40d082abdc50" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:18:05.804335 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.804264    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca" pod "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:18:05.804974 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.804943    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-serving-certs-ca-bundle" pod "f46ad168-4931-41e7-ac08-f6fa965721f9" (UID: "f46ad168-4931-41e7-ac08-f6fa965721f9"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:18:05.805094 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.805073    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-1" pod "f46ad168-4931-41e7-ac08-f6fa965721f9" (UID: "f46ad168-4931-41e7-ac08-f6fa965721f9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:18:05.805547 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.805487    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-0" pod "f46ad168-4931-41e7-ac08-f6fa965721f9" (UID: "f46ad168-4931-41e7-ac08-f6fa965721f9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:18:05.806095 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.806071    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca" pod "00c39778-f1f8-4539-88eb-40d082abdc50" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:18:05.806789 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.806742    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-trusted-ca-bundle" pod "f46ad168-4931-41e7-ac08-f6fa965721f9" (UID: "f46ad168-4931-41e7-ac08-f6fa965721f9"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:18:05.807212 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.806925    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle" pod "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:18:05.807212 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.806753    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config" pod "00c39778-f1f8-4539-88eb-40d082abdc50" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:18:05.807212 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.807143    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config" pod "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:18:05.807212 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.807186    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-metrics-client-ca" pod "f46ad168-4931-41e7-ac08-f6fa965721f9" (UID: "f46ad168-4931-41e7-ac08-f6fa965721f9"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:18:05.807876 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.807824    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config" pod "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:18:05.808123 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.807920    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-kube-rbac-proxy" pod "f46ad168-4931-41e7-ac08-f6fa965721f9" (UID: "f46ad168-4931-41e7-ac08-f6fa965721f9"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:18:05.808123 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.808044    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj" pod "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350"). InnerVolumeSpecName "kube-api-access-cjrtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:18:05.808123 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.808082    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert" pod "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350" (UID: "8bfb6a7d-e8f9-434f-acfc-8ac3badf1350"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:18:05.808386 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.808213    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46ad168-4931-41e7-ac08-f6fa965721f9-tls-assets" pod "f46ad168-4931-41e7-ac08-f6fa965721f9" (UID: "f46ad168-4931-41e7-ac08-f6fa965721f9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:18:05.808386 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.808271    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-config" pod "f46ad168-4931-41e7-ac08-f6fa965721f9" (UID: "f46ad168-4931-41e7-ac08-f6fa965721f9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:18:05.808386 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.808304    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-thanos-prometheus-http-client-file" pod "f46ad168-4931-41e7-ac08-f6fa965721f9" (UID: "f46ad168-4931-41e7-ac08-f6fa965721f9"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:18:05.809241 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.809139    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert" pod "00c39778-f1f8-4539-88eb-40d082abdc50" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:18:05.809241 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.809170    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46ad168-4931-41e7-ac08-f6fa965721f9-config-out" pod "f46ad168-4931-41e7-ac08-f6fa965721f9" (UID: "f46ad168-4931-41e7-ac08-f6fa965721f9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:18:05.809241 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.809210    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-kube-rbac-proxy-web" pod "f46ad168-4931-41e7-ac08-f6fa965721f9" (UID: "f46ad168-4931-41e7-ac08-f6fa965721f9"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:18:05.809241 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.809215    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b" pod "00c39778-f1f8-4539-88eb-40d082abdc50" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50"). InnerVolumeSpecName "kube-api-access-c986b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:18:05.809648 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.809623    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-grpc-tls" pod "f46ad168-4931-41e7-ac08-f6fa965721f9" (UID: "f46ad168-4931-41e7-ac08-f6fa965721f9"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:18:05.809744 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.809679    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config" pod "00c39778-f1f8-4539-88eb-40d082abdc50" (UID: "00c39778-f1f8-4539-88eb-40d082abdc50"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:18:05.809827 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.809799    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-thanos-sidecar-tls" pod "f46ad168-4931-41e7-ac08-f6fa965721f9" (UID: "f46ad168-4931-41e7-ac08-f6fa965721f9"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:18:05.810039 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.810020    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46ad168-4931-41e7-ac08-f6fa965721f9-kube-api-access-q4sxz" pod "f46ad168-4931-41e7-ac08-f6fa965721f9" (UID: "f46ad168-4931-41e7-ac08-f6fa965721f9"). InnerVolumeSpecName "kube-api-access-q4sxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:18:05.810111 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.810043    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-metrics-client-certs" pod "f46ad168-4931-41e7-ac08-f6fa965721f9" (UID: "f46ad168-4931-41e7-ac08-f6fa965721f9"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:18:05.810908 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.810887    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-tls" pod "f46ad168-4931-41e7-ac08-f6fa965721f9" (UID: "f46ad168-4931-41e7-ac08-f6fa965721f9"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:18:05.819258 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.819231    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-web-config" pod "f46ad168-4931-41e7-ac08-f6fa965721f9" (UID: "f46ad168-4931-41e7-ac08-f6fa965721f9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:18:05.904786 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.904746    6059 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-serving-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.904786 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.904778    6059 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f46ad168-4931-41e7-ac08-f6fa965721f9-tls-assets\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.904786 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.904793    6059 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.904809    6059 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-oauth-config\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.904821    6059 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-service-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.904834    6059 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.904845    6059 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-kube-rbac-proxy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.904857    6059 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-serving-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.904869    6059 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.904881    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjrtj\" (UniqueName: \"kubernetes.io/projected/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-kube-api-access-cjrtj\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.904893    6059 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00c39778-f1f8-4539-88eb-40d082abdc50-tmp\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.904905    6059 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-db\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.904918    6059 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-metrics-client-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.904934    6059 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-console-config\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.904948    6059 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.904963    6059 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-web-config\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.904975    6059 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-oauth-serving-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.904987    6059 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-oauth-serving-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.905001    6059 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00c39778-f1f8-4539-88eb-40d082abdc50-console-oauth-config\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.905017    6059 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.905031    6059 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f46ad168-4931-41e7-ac08-f6fa965721f9-config-out\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.905044    6059 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00c39778-f1f8-4539-88eb-40d082abdc50-service-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.905059    6059 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.905074    6059 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-console-config\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.905087    6059 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.905102    6059 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.905116    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4sxz\" (UniqueName: \"kubernetes.io/projected/f46ad168-4931-41e7-ac08-f6fa965721f9-kube-api-access-q4sxz\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.905131    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c986b\" (UniqueName: \"kubernetes.io/projected/00c39778-f1f8-4539-88eb-40d082abdc50-kube-api-access-c986b\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.905145    6059 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f46ad168-4931-41e7-ac08-f6fa965721f9-configmap-metrics-client-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.905160    6059 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350-trusted-ca-bundle\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.905175    6059 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-secret-grpc-tls\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:05.905759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:05.905190    6059 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f46ad168-4931-41e7-ac08-f6fa965721f9-config\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:06.523643 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.523605    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f46ad168-4931-41e7-ac08-f6fa965721f9","Type":"ContainerDied","Data":"1e25b5f52e685e6c8026025d53999cc07360830d7380db2ba5a109782deede16"}
Nov 03 17:18:06.523643 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.523648    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:06.524166 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.523708    6059 scope.go:122] "RemoveContainer" containerID="807ab55e21372b935ec0f6b61e2a61638523471c8c0f6721ea9fc4fc77bc50c2"
Nov 03 17:18:06.525127 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.525098    6059 generic.go:356] "Generic (PLEG): container finished" podID="8bfb6a7d-e8f9-434f-acfc-8ac3badf1350" containerID="bd2c1a3533d2ff1149b8ec0fe0ee8605177b76f09ff1efc3e019fcdf9fc504af" exitCode=0
Nov 03 17:18:06.525229 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.525156    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858bfbb7dc-x6n58"
Nov 03 17:18:06.525229 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.525196    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858bfbb7dc-x6n58" event={"ID":"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350","Type":"ContainerDied","Data":"bd2c1a3533d2ff1149b8ec0fe0ee8605177b76f09ff1efc3e019fcdf9fc504af"}
Nov 03 17:18:06.525320 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.525234    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858bfbb7dc-x6n58" event={"ID":"8bfb6a7d-e8f9-434f-acfc-8ac3badf1350","Type":"ContainerDied","Data":"ad6b0fb9aaffef859b91fc640e83dbf07b4c624dd9fe9a618643d5abc60ee82f"}
Nov 03 17:18:06.526526 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.526504    6059 generic.go:356] "Generic (PLEG): container finished" podID="00c39778-f1f8-4539-88eb-40d082abdc50" containerID="5ce1bc06c7ca0db7013a0414fafca270db33f2b5d1d25f5a2d92d842e969d3c5" exitCode=0
Nov 03 17:18:06.526617 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.526552    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d896bf64d-smcmk" event={"ID":"00c39778-f1f8-4539-88eb-40d082abdc50","Type":"ContainerDied","Data":"5ce1bc06c7ca0db7013a0414fafca270db33f2b5d1d25f5a2d92d842e969d3c5"}
Nov 03 17:18:06.526617 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.526583    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d896bf64d-smcmk" event={"ID":"00c39778-f1f8-4539-88eb-40d082abdc50","Type":"ContainerDied","Data":"9984b1a24abe3ef16d129e5956dacf85fdbde572d7642a1c35a8b2cd64cffe6d"}
Nov 03 17:18:06.526732 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.526647    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d896bf64d-smcmk"
Nov 03 17:18:06.533536 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.533469    6059 scope.go:122] "RemoveContainer" containerID="71598ebd4a04a726ae4252ab94a67843e6d610fca75c24d900bb0310185b1726"
Nov 03 17:18:06.541752 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.541735    6059 scope.go:122] "RemoveContainer" containerID="235be22017efeb10a48b4d98ec7fc426552e2ab28c203c4df0025046fe88d1fe"
Nov 03 17:18:06.548558 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.548536    6059 scope.go:122] "RemoveContainer" containerID="97a0cb86319a8612ce3b35d0a295dc48bd55c92e0b0ee51d9338248cb5503115"
Nov 03 17:18:06.555272 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.555251    6059 scope.go:122] "RemoveContainer" containerID="b1b3f036a96309db13f2935405fe341cd62324aec92dacbdcb8265281afa00d5"
Nov 03 17:18:06.562129 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.562107    6059 scope.go:122] "RemoveContainer" containerID="2422aa3f96a75eba8798b0f2d19e59a639aefbc0f2868f1381c755c865ff3ba9"
Nov 03 17:18:06.569596 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.569574    6059 scope.go:122] "RemoveContainer" containerID="ab80803ea666ba4724c980d9da8e5702b1510504878c229ff5dc64415ff94575"
Nov 03 17:18:06.578745 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.578590    6059 scope.go:122] "RemoveContainer" containerID="bd2c1a3533d2ff1149b8ec0fe0ee8605177b76f09ff1efc3e019fcdf9fc504af"
Nov 03 17:18:06.586768 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.586749    6059 scope.go:122] "RemoveContainer" containerID="bd2c1a3533d2ff1149b8ec0fe0ee8605177b76f09ff1efc3e019fcdf9fc504af"
Nov 03 17:18:06.587213 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:18:06.587194    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd2c1a3533d2ff1149b8ec0fe0ee8605177b76f09ff1efc3e019fcdf9fc504af\": container with ID starting with bd2c1a3533d2ff1149b8ec0fe0ee8605177b76f09ff1efc3e019fcdf9fc504af not found: ID does not exist" containerID="bd2c1a3533d2ff1149b8ec0fe0ee8605177b76f09ff1efc3e019fcdf9fc504af"
Nov 03 17:18:06.587292 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.587223    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd2c1a3533d2ff1149b8ec0fe0ee8605177b76f09ff1efc3e019fcdf9fc504af"} err="failed to get container status \"bd2c1a3533d2ff1149b8ec0fe0ee8605177b76f09ff1efc3e019fcdf9fc504af\": rpc error: code = NotFound desc = could not find container \"bd2c1a3533d2ff1149b8ec0fe0ee8605177b76f09ff1efc3e019fcdf9fc504af\": container with ID starting with bd2c1a3533d2ff1149b8ec0fe0ee8605177b76f09ff1efc3e019fcdf9fc504af not found: ID does not exist"
Nov 03 17:18:06.587292 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.587242    6059 scope.go:122] "RemoveContainer" containerID="5ce1bc06c7ca0db7013a0414fafca270db33f2b5d1d25f5a2d92d842e969d3c5"
Nov 03 17:18:06.594232 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.594212    6059 scope.go:122] "RemoveContainer" containerID="5ce1bc06c7ca0db7013a0414fafca270db33f2b5d1d25f5a2d92d842e969d3c5"
Nov 03 17:18:06.594517 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:18:06.594496    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ce1bc06c7ca0db7013a0414fafca270db33f2b5d1d25f5a2d92d842e969d3c5\": container with ID starting with 5ce1bc06c7ca0db7013a0414fafca270db33f2b5d1d25f5a2d92d842e969d3c5 not found: ID does not exist" containerID="5ce1bc06c7ca0db7013a0414fafca270db33f2b5d1d25f5a2d92d842e969d3c5"
Nov 03 17:18:06.594582 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.594526    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ce1bc06c7ca0db7013a0414fafca270db33f2b5d1d25f5a2d92d842e969d3c5"} err="failed to get container status \"5ce1bc06c7ca0db7013a0414fafca270db33f2b5d1d25f5a2d92d842e969d3c5\": rpc error: code = NotFound desc = could not find container \"5ce1bc06c7ca0db7013a0414fafca270db33f2b5d1d25f5a2d92d842e969d3c5\": container with ID starting with 5ce1bc06c7ca0db7013a0414fafca270db33f2b5d1d25f5a2d92d842e969d3c5 not found: ID does not exist"
Nov 03 17:18:06.731498 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.731466    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-858bfbb7dc-x6n58"]
Nov 03 17:18:06.735756 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.735724    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-858bfbb7dc-x6n58"]
Nov 03 17:18:06.843857 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.843828    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 17:18:06.861275 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:18:06.861242    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:18:06.891982 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.891946    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 17:18:06.892346 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892324    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 17:18:06.892664 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892648    6059 cpu_manager.go:414] "containerMap: removing container" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="kube-rbac-proxy-metric"
Nov 03 17:18:06.892731 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892667    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="kube-rbac-proxy-metric"
Nov 03 17:18:06.892731 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892682    6059 cpu_manager.go:414] "containerMap: removing container" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="init-config-reloader"
Nov 03 17:18:06.892731 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892690    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="init-config-reloader"
Nov 03 17:18:06.892731 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892701    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="kube-rbac-proxy-thanos"
Nov 03 17:18:06.892731 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892709    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="kube-rbac-proxy-thanos"
Nov 03 17:18:06.892731 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892724    6059 cpu_manager.go:414] "containerMap: removing container" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="config-reloader"
Nov 03 17:18:06.892731 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892731    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="config-reloader"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892747    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="kube-rbac-proxy-web"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892755    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="kube-rbac-proxy-web"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892767    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="prometheus"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892775    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="prometheus"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892785    6059 cpu_manager.go:414] "containerMap: removing container" podUID="00c39778-f1f8-4539-88eb-40d082abdc50" containerName="console"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892792    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c39778-f1f8-4539-88eb-40d082abdc50" containerName="console"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892806    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="init-config-reloader"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892814    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="init-config-reloader"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892825    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="kube-rbac-proxy"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892833    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="kube-rbac-proxy"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892843    6059 cpu_manager.go:414] "containerMap: removing container" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="alertmanager"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892850    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="alertmanager"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892861    6059 cpu_manager.go:414] "containerMap: removing container" podUID="8bfb6a7d-e8f9-434f-acfc-8ac3badf1350" containerName="console"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892868    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bfb6a7d-e8f9-434f-acfc-8ac3badf1350" containerName="console"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892880    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="config-reloader"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892887    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="config-reloader"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892895    6059 cpu_manager.go:414] "containerMap: removing container" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="prom-label-proxy"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892903    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="prom-label-proxy"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892913    6059 cpu_manager.go:414] "containerMap: removing container" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="kube-rbac-proxy-web"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892920    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="kube-rbac-proxy-web"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892933    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="thanos-sidecar"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892940    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="thanos-sidecar"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892964    6059 cpu_manager.go:414] "containerMap: removing container" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="kube-rbac-proxy"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.892972    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="kube-rbac-proxy"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.893060    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bfb6a7d-e8f9-434f-acfc-8ac3badf1350" containerName="console"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.893073    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="config-reloader"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.893083    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="prom-label-proxy"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.893092    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="kube-rbac-proxy-web"
Nov 03 17:18:06.893088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.893101    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="thanos-sidecar"
Nov 03 17:18:06.894436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.893112    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="kube-rbac-proxy"
Nov 03 17:18:06.894436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.893120    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="kube-rbac-proxy-metric"
Nov 03 17:18:06.894436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.893127    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="init-config-reloader"
Nov 03 17:18:06.894436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.893135    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="kube-rbac-proxy-thanos"
Nov 03 17:18:06.894436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.893144    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="config-reloader"
Nov 03 17:18:06.894436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.893152    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="kube-rbac-proxy-web"
Nov 03 17:18:06.894436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.893160    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="prometheus"
Nov 03 17:18:06.894436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.893169    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c39778-f1f8-4539-88eb-40d082abdc50" containerName="console"
Nov 03 17:18:06.894436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.893177    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="init-config-reloader"
Nov 03 17:18:06.894436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.893187    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" containerName="kube-rbac-proxy"
Nov 03 17:18:06.894436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.893195    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9169e62e-57ab-4322-a0cc-664f54e48556" containerName="alertmanager"
Nov 03 17:18:06.902062 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.902033    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:06.952953 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.952920    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 17:18:06.954009 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.953991    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2c6sf\""
Nov 03 17:18:06.955318 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.955302    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 17:18:06.956479 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.956462    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 17:18:06.959864 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.959848    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 17:18:06.960103 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:06.960091    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 17:18:07.004509 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.004408    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 17:18:07.004885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.004855    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d896bf64d-smcmk"]
Nov 03 17:18:07.005238 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.005226    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 17:18:07.005495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.005475    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d896bf64d-smcmk"]
Nov 03 17:18:07.012673 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.012640    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e2f223b3-bcb1-4d64-a561-96c3b1f10127-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.012804 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.012678    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.012804 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.012696    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.012804 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.012717    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.012918 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.012809    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e2f223b3-bcb1-4d64-a561-96c3b1f10127-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.012918 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.012842    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e2f223b3-bcb1-4d64-a561-96c3b1f10127-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.012918 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.012891    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2f223b3-bcb1-4d64-a561-96c3b1f10127-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.013018 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.012948    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-web-config\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.013018 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.012977    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2f223b3-bcb1-4d64-a561-96c3b1f10127-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.013018 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.012998    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e2f223b3-bcb1-4d64-a561-96c3b1f10127-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.013018 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.013014    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2f223b3-bcb1-4d64-a561-96c3b1f10127-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.013156 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.013082    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.013156 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.013101    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.013156 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.013131    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.013156 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.013149    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e2f223b3-bcb1-4d64-a561-96c3b1f10127-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.013289 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.013165    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-config\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.013289 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.013184    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrdxt\" (UniqueName: \"kubernetes.io/projected/e2f223b3-bcb1-4d64-a561-96c3b1f10127-kube-api-access-vrdxt\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.013289 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.013267    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e2f223b3-bcb1-4d64-a561-96c3b1f10127-config-out\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.013289 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.013286    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2f223b3-bcb1-4d64-a561-96c3b1f10127-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.013490 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.013347    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.053111 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.053070    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 17:18:07.053272 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.053145    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8kr56n0suls3\""
Nov 03 17:18:07.053339 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.053321    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 17:18:07.053423 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.053327    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 17:18:07.053423 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.053343    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 17:18:07.053546 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.053519    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 17:18:07.053546 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.053532    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 17:18:07.053546 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.053523    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 17:18:07.053644 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.053522    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 17:18:07.054856 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.054835    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:18:07.114711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.114673    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e2f223b3-bcb1-4d64-a561-96c3b1f10127-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.114711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.114715    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.114933 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.114734    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.114933 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.114766    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.114933 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.114797    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e2f223b3-bcb1-4d64-a561-96c3b1f10127-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.114933 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.114823    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e2f223b3-bcb1-4d64-a561-96c3b1f10127-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.114933 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.114859    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2f223b3-bcb1-4d64-a561-96c3b1f10127-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.114933 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.114884    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-web-config\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.115219 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.115204    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2f223b3-bcb1-4d64-a561-96c3b1f10127-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.115283 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.115222    6059 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 17:18:07.115283 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.115242    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e2f223b3-bcb1-4d64-a561-96c3b1f10127-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.115283 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.115210    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e2f223b3-bcb1-4d64-a561-96c3b1f10127-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.115283 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.115271    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2f223b3-bcb1-4d64-a561-96c3b1f10127-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.115562 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.115306    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.115562 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.115346    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.115562 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.115422    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.115562 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.115476    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e2f223b3-bcb1-4d64-a561-96c3b1f10127-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.115562 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.115507    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-config\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.115562 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.115544    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrdxt\" (UniqueName: \"kubernetes.io/projected/e2f223b3-bcb1-4d64-a561-96c3b1f10127-kube-api-access-vrdxt\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.115864 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.115581    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e2f223b3-bcb1-4d64-a561-96c3b1f10127-config-out\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.115864 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.115623    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2f223b3-bcb1-4d64-a561-96c3b1f10127-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.115864 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.115664    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.115864 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.115759    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e2f223b3-bcb1-4d64-a561-96c3b1f10127-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.116060 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.115997    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2f223b3-bcb1-4d64-a561-96c3b1f10127-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.116060 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.116046    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2f223b3-bcb1-4d64-a561-96c3b1f10127-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.116481 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.116453    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2f223b3-bcb1-4d64-a561-96c3b1f10127-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.116632 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.116581    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e2f223b3-bcb1-4d64-a561-96c3b1f10127-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.117910 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.117883    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2f223b3-bcb1-4d64-a561-96c3b1f10127-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.118156 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.118128    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.118554 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.118528    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.119876 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.119808    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e2f223b3-bcb1-4d64-a561-96c3b1f10127-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.120387 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.120020    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.120387 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.120029    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.120387 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.120296    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-config\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.120648 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.120620    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.120765 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.120676    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.120825 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.120783    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e2f223b3-bcb1-4d64-a561-96c3b1f10127-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.120875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.120858    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e2f223b3-bcb1-4d64-a561-96c3b1f10127-config-out\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.121082 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.121061    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.121396 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.121375    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e2f223b3-bcb1-4d64-a561-96c3b1f10127-web-config\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.176135 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.176102    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrdxt\" (UniqueName: \"kubernetes.io/projected/e2f223b3-bcb1-4d64-a561-96c3b1f10127-kube-api-access-vrdxt\") pod \"prometheus-k8s-0\" (UID: \"e2f223b3-bcb1-4d64-a561-96c3b1f10127\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.233088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.233046    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:07.237201 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.237174    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:18:07.335004 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.334967    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c39778-f1f8-4539-88eb-40d082abdc50" path="/var/lib/kubelet/pods/00c39778-f1f8-4539-88eb-40d082abdc50/volumes"
Nov 03 17:18:07.335460 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.335442    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bfb6a7d-e8f9-434f-acfc-8ac3badf1350" path="/var/lib/kubelet/pods/8bfb6a7d-e8f9-434f-acfc-8ac3badf1350/volumes"
Nov 03 17:18:07.335879 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.335863    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f46ad168-4931-41e7-ac08-f6fa965721f9" path="/var/lib/kubelet/pods/f46ad168-4931-41e7-ac08-f6fa965721f9/volumes"
Nov 03 17:18:07.699430 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:07.699316    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 17:18:07.706671 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:18:07.706634    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f223b3_bcb1_4d64_a561_96c3b1f10127.slice/crio-7a46a4a613a3b7d81d9dcbe3567818a41f9d1d05b3e423419bcda8a087c50155 WatchSource:0}: Error finding container 7a46a4a613a3b7d81d9dcbe3567818a41f9d1d05b3e423419bcda8a087c50155: Status 404 returned error can't find the container with id 7a46a4a613a3b7d81d9dcbe3567818a41f9d1d05b3e423419bcda8a087c50155
Nov 03 17:18:08.537849 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:08.537817    6059 generic.go:356] "Generic (PLEG): container finished" podID="e2f223b3-bcb1-4d64-a561-96c3b1f10127" containerID="18dfe1049731e327d5a820b9cb00645d775166f771939bd28701dd52bf6c14a9" exitCode=0
Nov 03 17:18:08.538025 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:08.537867    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e2f223b3-bcb1-4d64-a561-96c3b1f10127","Type":"ContainerDied","Data":"18dfe1049731e327d5a820b9cb00645d775166f771939bd28701dd52bf6c14a9"}
Nov 03 17:18:08.538025 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:08.537941    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e2f223b3-bcb1-4d64-a561-96c3b1f10127","Type":"ContainerStarted","Data":"7a46a4a613a3b7d81d9dcbe3567818a41f9d1d05b3e423419bcda8a087c50155"}
Nov 03 17:18:09.544781 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:09.544744    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e2f223b3-bcb1-4d64-a561-96c3b1f10127","Type":"ContainerStarted","Data":"73cbdc55423a4b448f3bd099338e9a17b796760eab0cf034fe5d3a2e0e860f35"}
Nov 03 17:18:09.545220 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:09.544825    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e2f223b3-bcb1-4d64-a561-96c3b1f10127","Type":"ContainerStarted","Data":"4f9a2924551d2bb0fb64cb7b2c190834ab23f1a0a0bdaa749f00808b9991020a"}
Nov 03 17:18:09.545220 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:09.544862    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e2f223b3-bcb1-4d64-a561-96c3b1f10127","Type":"ContainerStarted","Data":"067b5029ae055ed5894ab6abf909b51b8c0cf7e8dc162e414d528a91fcb6c74b"}
Nov 03 17:18:09.545220 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:09.544886    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e2f223b3-bcb1-4d64-a561-96c3b1f10127","Type":"ContainerStarted","Data":"e2fe2741eca41eb3b60f4fce5cd3c8dd98b8205f7ec50b262796aafbd443f596"}
Nov 03 17:18:09.545220 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:09.544908    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e2f223b3-bcb1-4d64-a561-96c3b1f10127","Type":"ContainerStarted","Data":"42b519a4ce5ac90f0075d1c42a1fb10fa352c01afb72b499e1a207d522d18dd3"}
Nov 03 17:18:09.545220 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:09.544931    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e2f223b3-bcb1-4d64-a561-96c3b1f10127","Type":"ContainerStarted","Data":"9c887151fb73e808db42de5db611f6918432888a84bf904ff44b633948be4542"}
Nov 03 17:18:09.676835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:09.675922    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.655318245 podStartE2EDuration="3.675903155s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.58491ms" isStatefulPod=false podCreationTimestamp="2026-11-03 17:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:18:09.673028328 +0000 UTC m=+30.470741172" watchObservedRunningTime="2026-11-03 17:18:09.675903155 +0000 UTC m=+30.473615936"
Nov 03 17:18:11.344715 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:11.344675    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:11.360406 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:11.360378    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:11.553098 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:11.553063    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:11.568127 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:11.568101    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:18:16.949960 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:16.949928    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-d252q"
Nov 03 17:18:27.727855 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:27.727816    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:18:30.194959 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:30.194922    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:18:30.368625 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:18:30.368590    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:18:36.860555 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:18:36.860518    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:19:06.860541 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:19:06.860505    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:19:30.562761 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:19:30.562562    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:19:36.861024 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:19:36.860984    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:19:41.557882 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:19:41.557846    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:19:54.530752 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:19:54.530713    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:20:06.860743 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:20:06.860704    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:20:30.917847 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:30.917813    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:20:32.293103 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:32.293069    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:20:32.509196 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:32.509162    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:20:33.894733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:33.894699    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:20:34.086611 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:34.086578    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:20:35.486206 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.486178    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:20:35.589578 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.589541    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn"]
Nov 03 17:20:35.593038 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.593014    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn"
Nov 03 17:20:35.690481 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.690441    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 17:20:35.690481 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.690483    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-kqh9k\""
Nov 03 17:20:35.690713 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.690482    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 17:20:35.691067 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.691047    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn"]
Nov 03 17:20:35.692311 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.692291    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt"]
Nov 03 17:20:35.695748 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.695728    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt"
Nov 03 17:20:35.717654 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.717623    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp6hp\" (UniqueName: \"kubernetes.io/projected/a4d6dda2-d650-4cf5-b89f-26b5a8233844-kube-api-access-hp6hp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn\" (UID: \"a4d6dda2-d650-4cf5-b89f-26b5a8233844\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn"
Nov 03 17:20:35.717812 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.717679    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4d6dda2-d650-4cf5-b89f-26b5a8233844-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn\" (UID: \"a4d6dda2-d650-4cf5-b89f-26b5a8233844\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn"
Nov 03 17:20:35.717812 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.717767    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4d6dda2-d650-4cf5-b89f-26b5a8233844-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn\" (UID: \"a4d6dda2-d650-4cf5-b89f-26b5a8233844\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn"
Nov 03 17:20:35.746045 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.745965    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-jlc64\""
Nov 03 17:20:35.758905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.758867    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt"]
Nov 03 17:20:35.794170 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.794127    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t"]
Nov 03 17:20:35.797725 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.797700    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t"
Nov 03 17:20:35.818548 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.818516    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp6hp\" (UniqueName: \"kubernetes.io/projected/a4d6dda2-d650-4cf5-b89f-26b5a8233844-kube-api-access-hp6hp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn\" (UID: \"a4d6dda2-d650-4cf5-b89f-26b5a8233844\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn"
Nov 03 17:20:35.818706 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.818567    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-522br\" (UniqueName: \"kubernetes.io/projected/a39b3295-6622-4055-b86f-c27840fb64b5-kube-api-access-522br\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt\" (UID: \"a39b3295-6622-4055-b86f-c27840fb64b5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt"
Nov 03 17:20:35.818706 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.818594    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a39b3295-6622-4055-b86f-c27840fb64b5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt\" (UID: \"a39b3295-6622-4055-b86f-c27840fb64b5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt"
Nov 03 17:20:35.818794 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.818717    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4d6dda2-d650-4cf5-b89f-26b5a8233844-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn\" (UID: \"a4d6dda2-d650-4cf5-b89f-26b5a8233844\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn"
Nov 03 17:20:35.818794 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.818741    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a39b3295-6622-4055-b86f-c27840fb64b5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt\" (UID: \"a39b3295-6622-4055-b86f-c27840fb64b5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt"
Nov 03 17:20:35.818794 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.818770    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4d6dda2-d650-4cf5-b89f-26b5a8233844-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn\" (UID: \"a4d6dda2-d650-4cf5-b89f-26b5a8233844\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn"
Nov 03 17:20:35.819112 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.819092    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4d6dda2-d650-4cf5-b89f-26b5a8233844-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn\" (UID: \"a4d6dda2-d650-4cf5-b89f-26b5a8233844\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn"
Nov 03 17:20:35.819185 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.819117    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4d6dda2-d650-4cf5-b89f-26b5a8233844-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn\" (UID: \"a4d6dda2-d650-4cf5-b89f-26b5a8233844\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn"
Nov 03 17:20:35.847721 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.847689    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-f59tx\""
Nov 03 17:20:35.854275 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.854247    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t"]
Nov 03 17:20:35.877272 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.877241    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp6hp\" (UniqueName: \"kubernetes.io/projected/a4d6dda2-d650-4cf5-b89f-26b5a8233844-kube-api-access-hp6hp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn\" (UID: \"a4d6dda2-d650-4cf5-b89f-26b5a8233844\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn"
Nov 03 17:20:35.898416 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.898381    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5"]
Nov 03 17:20:35.901905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.901882    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5"
Nov 03 17:20:35.905084 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.905056    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r"]
Nov 03 17:20:35.908413 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.908380    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn"
Nov 03 17:20:35.908934 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.908915    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r"
Nov 03 17:20:35.919686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.919662    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p44rx\" (UniqueName: \"kubernetes.io/projected/510707fc-6270-4156-a83f-8d4ce5fbcfa0-kube-api-access-p44rx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t\" (UID: \"510707fc-6270-4156-a83f-8d4ce5fbcfa0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t"
Nov 03 17:20:35.919777 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.919713    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/510707fc-6270-4156-a83f-8d4ce5fbcfa0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t\" (UID: \"510707fc-6270-4156-a83f-8d4ce5fbcfa0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t"
Nov 03 17:20:35.919814 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.919788    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-522br\" (UniqueName: \"kubernetes.io/projected/a39b3295-6622-4055-b86f-c27840fb64b5-kube-api-access-522br\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt\" (UID: \"a39b3295-6622-4055-b86f-c27840fb64b5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt"
Nov 03 17:20:35.919814 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.919809    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a39b3295-6622-4055-b86f-c27840fb64b5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt\" (UID: \"a39b3295-6622-4055-b86f-c27840fb64b5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt"
Nov 03 17:20:35.919890 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.919826    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/510707fc-6270-4156-a83f-8d4ce5fbcfa0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t\" (UID: \"510707fc-6270-4156-a83f-8d4ce5fbcfa0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t"
Nov 03 17:20:35.919890 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.919861    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a39b3295-6622-4055-b86f-c27840fb64b5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt\" (UID: \"a39b3295-6622-4055-b86f-c27840fb64b5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt"
Nov 03 17:20:35.920276 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.920259    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a39b3295-6622-4055-b86f-c27840fb64b5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt\" (UID: \"a39b3295-6622-4055-b86f-c27840fb64b5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt"
Nov 03 17:20:35.920313 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.920255    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a39b3295-6622-4055-b86f-c27840fb64b5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt\" (UID: \"a39b3295-6622-4055-b86f-c27840fb64b5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt"
Nov 03 17:20:35.951939 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.951904    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-d68mj\""
Nov 03 17:20:35.958500 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.958390    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5"]
Nov 03 17:20:35.964823 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.964776    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r"]
Nov 03 17:20:35.980849 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:35.980822    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-522br\" (UniqueName: \"kubernetes.io/projected/a39b3295-6622-4055-b86f-c27840fb64b5-kube-api-access-522br\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt\" (UID: \"a39b3295-6622-4055-b86f-c27840fb64b5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt"
Nov 03 17:20:36.011562 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.011470    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt"
Nov 03 17:20:36.020692 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.020670    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/510707fc-6270-4156-a83f-8d4ce5fbcfa0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t\" (UID: \"510707fc-6270-4156-a83f-8d4ce5fbcfa0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t"
Nov 03 17:20:36.020784 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.020706    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/076f5e42-e2ae-4433-aa5a-6f8d6d30f75e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r\" (UID: \"076f5e42-e2ae-4433-aa5a-6f8d6d30f75e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r"
Nov 03 17:20:36.020784 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.020740    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t98lx\" (UniqueName: \"kubernetes.io/projected/aa53fe92-f3cd-4546-a6e8-beeadeb51142-kube-api-access-t98lx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5\" (UID: \"aa53fe92-f3cd-4546-a6e8-beeadeb51142\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5"
Nov 03 17:20:36.020875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.020841    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/076f5e42-e2ae-4433-aa5a-6f8d6d30f75e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r\" (UID: \"076f5e42-e2ae-4433-aa5a-6f8d6d30f75e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r"
Nov 03 17:20:36.020924 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.020883    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa53fe92-f3cd-4546-a6e8-beeadeb51142-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5\" (UID: \"aa53fe92-f3cd-4546-a6e8-beeadeb51142\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5"
Nov 03 17:20:36.021086 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.021019    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p44rx\" (UniqueName: \"kubernetes.io/projected/510707fc-6270-4156-a83f-8d4ce5fbcfa0-kube-api-access-p44rx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t\" (UID: \"510707fc-6270-4156-a83f-8d4ce5fbcfa0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t"
Nov 03 17:20:36.021086 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.021075    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa53fe92-f3cd-4546-a6e8-beeadeb51142-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5\" (UID: \"aa53fe92-f3cd-4546-a6e8-beeadeb51142\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5"
Nov 03 17:20:36.021158 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.021124    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h58hs\" (UniqueName: \"kubernetes.io/projected/076f5e42-e2ae-4433-aa5a-6f8d6d30f75e-kube-api-access-h58hs\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r\" (UID: \"076f5e42-e2ae-4433-aa5a-6f8d6d30f75e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r"
Nov 03 17:20:36.021220 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.021180    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/510707fc-6270-4156-a83f-8d4ce5fbcfa0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t\" (UID: \"510707fc-6270-4156-a83f-8d4ce5fbcfa0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t"
Nov 03 17:20:36.021220 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.021175    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/510707fc-6270-4156-a83f-8d4ce5fbcfa0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t\" (UID: \"510707fc-6270-4156-a83f-8d4ce5fbcfa0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t"
Nov 03 17:20:36.021521 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.021505    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/510707fc-6270-4156-a83f-8d4ce5fbcfa0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t\" (UID: \"510707fc-6270-4156-a83f-8d4ce5fbcfa0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t"
Nov 03 17:20:36.052961 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.052927    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-pbzpq\""
Nov 03 17:20:36.080645 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.080614    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p44rx\" (UniqueName: \"kubernetes.io/projected/510707fc-6270-4156-a83f-8d4ce5fbcfa0-kube-api-access-p44rx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t\" (UID: \"510707fc-6270-4156-a83f-8d4ce5fbcfa0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t"
Nov 03 17:20:36.113919 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.113862    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t"
Nov 03 17:20:36.122242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.122211    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/076f5e42-e2ae-4433-aa5a-6f8d6d30f75e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r\" (UID: \"076f5e42-e2ae-4433-aa5a-6f8d6d30f75e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r"
Nov 03 17:20:36.122398 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.122263    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t98lx\" (UniqueName: \"kubernetes.io/projected/aa53fe92-f3cd-4546-a6e8-beeadeb51142-kube-api-access-t98lx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5\" (UID: \"aa53fe92-f3cd-4546-a6e8-beeadeb51142\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5"
Nov 03 17:20:36.122398 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.122289    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/076f5e42-e2ae-4433-aa5a-6f8d6d30f75e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r\" (UID: \"076f5e42-e2ae-4433-aa5a-6f8d6d30f75e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r"
Nov 03 17:20:36.122398 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.122306    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa53fe92-f3cd-4546-a6e8-beeadeb51142-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5\" (UID: \"aa53fe92-f3cd-4546-a6e8-beeadeb51142\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5"
Nov 03 17:20:36.122398 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.122343    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa53fe92-f3cd-4546-a6e8-beeadeb51142-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5\" (UID: \"aa53fe92-f3cd-4546-a6e8-beeadeb51142\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5"
Nov 03 17:20:36.122672 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.122642    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h58hs\" (UniqueName: \"kubernetes.io/projected/076f5e42-e2ae-4433-aa5a-6f8d6d30f75e-kube-api-access-h58hs\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r\" (UID: \"076f5e42-e2ae-4433-aa5a-6f8d6d30f75e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r"
Nov 03 17:20:36.122744 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.122661    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/076f5e42-e2ae-4433-aa5a-6f8d6d30f75e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r\" (UID: \"076f5e42-e2ae-4433-aa5a-6f8d6d30f75e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r"
Nov 03 17:20:36.122787 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.122661    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa53fe92-f3cd-4546-a6e8-beeadeb51142-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5\" (UID: \"aa53fe92-f3cd-4546-a6e8-beeadeb51142\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5"
Nov 03 17:20:36.122787 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.122729    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa53fe92-f3cd-4546-a6e8-beeadeb51142-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5\" (UID: \"aa53fe92-f3cd-4546-a6e8-beeadeb51142\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5"
Nov 03 17:20:36.122852 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.122788    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/076f5e42-e2ae-4433-aa5a-6f8d6d30f75e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r\" (UID: \"076f5e42-e2ae-4433-aa5a-6f8d6d30f75e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r"
Nov 03 17:20:36.183301 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.183270    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t98lx\" (UniqueName: \"kubernetes.io/projected/aa53fe92-f3cd-4546-a6e8-beeadeb51142-kube-api-access-t98lx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5\" (UID: \"aa53fe92-f3cd-4546-a6e8-beeadeb51142\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5"
Nov 03 17:20:36.183568 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.183543    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h58hs\" (UniqueName: \"kubernetes.io/projected/076f5e42-e2ae-4433-aa5a-6f8d6d30f75e-kube-api-access-h58hs\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r\" (UID: \"076f5e42-e2ae-4433-aa5a-6f8d6d30f75e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r"
Nov 03 17:20:36.220236 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.220204    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5"
Nov 03 17:20:36.239470 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.239439    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r"
Nov 03 17:20:36.331233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.331196    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn"]
Nov 03 17:20:36.338461 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:20:36.338429    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d6dda2_d650_4cf5_b89f_26b5a8233844.slice/crio-ee270ae000e1f1193f2a005ccfe32e5946458f6254bbfd651657e7daa72644f3 WatchSource:0}: Error finding container ee270ae000e1f1193f2a005ccfe32e5946458f6254bbfd651657e7daa72644f3: Status 404 returned error can't find the container with id ee270ae000e1f1193f2a005ccfe32e5946458f6254bbfd651657e7daa72644f3
Nov 03 17:20:36.340417 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.340399    6059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:20:36.386245 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.386212    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt"]
Nov 03 17:20:36.393822 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:20:36.393790    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39b3295_6622_4055_b86f_c27840fb64b5.slice/crio-47a8c4f3fe7f068cff0addc21eb38eb1beab9c1ddd8d5556392957d55c2246e7 WatchSource:0}: Error finding container 47a8c4f3fe7f068cff0addc21eb38eb1beab9c1ddd8d5556392957d55c2246e7: Status 404 returned error can't find the container with id 47a8c4f3fe7f068cff0addc21eb38eb1beab9c1ddd8d5556392957d55c2246e7
Nov 03 17:20:36.489344 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.489308    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t"]
Nov 03 17:20:36.497088 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:20:36.497062    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510707fc_6270_4156_a83f_8d4ce5fbcfa0.slice/crio-da5c3ac3201fd652ddd9b05e5d3c2fcf947afa6fdeb662ca5c5de94fb1bb9751 WatchSource:0}: Error finding container da5c3ac3201fd652ddd9b05e5d3c2fcf947afa6fdeb662ca5c5de94fb1bb9751: Status 404 returned error can't find the container with id da5c3ac3201fd652ddd9b05e5d3c2fcf947afa6fdeb662ca5c5de94fb1bb9751
Nov 03 17:20:36.593514 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.593425    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5"]
Nov 03 17:20:36.600770 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:20:36.600740    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa53fe92_f3cd_4546_a6e8_beeadeb51142.slice/crio-7e1fb43009b18de6e9acfe0223dc648867d1fb55a0a4d76a823d7e192754e3ff WatchSource:0}: Error finding container 7e1fb43009b18de6e9acfe0223dc648867d1fb55a0a4d76a823d7e192754e3ff: Status 404 returned error can't find the container with id 7e1fb43009b18de6e9acfe0223dc648867d1fb55a0a4d76a823d7e192754e3ff
Nov 03 17:20:36.626554 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:36.626518    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r"]
Nov 03 17:20:36.634994 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:20:36.634962    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076f5e42_e2ae_4433_aa5a_6f8d6d30f75e.slice/crio-dc057f545e7903334fa450da7e64a2fdf65fafd25704e7e829891c53cde1c844 WatchSource:0}: Error finding container dc057f545e7903334fa450da7e64a2fdf65fafd25704e7e829891c53cde1c844: Status 404 returned error can't find the container with id dc057f545e7903334fa450da7e64a2fdf65fafd25704e7e829891c53cde1c844
Nov 03 17:20:36.861479 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:20:36.861388    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:20:37.075241 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:37.075197    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn" event={"ID":"a4d6dda2-d650-4cf5-b89f-26b5a8233844","Type":"ContainerStarted","Data":"ee270ae000e1f1193f2a005ccfe32e5946458f6254bbfd651657e7daa72644f3"}
Nov 03 17:20:37.076330 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:37.076300    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt" event={"ID":"a39b3295-6622-4055-b86f-c27840fb64b5","Type":"ContainerStarted","Data":"47a8c4f3fe7f068cff0addc21eb38eb1beab9c1ddd8d5556392957d55c2246e7"}
Nov 03 17:20:37.077482 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:37.077457    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r" event={"ID":"076f5e42-e2ae-4433-aa5a-6f8d6d30f75e","Type":"ContainerStarted","Data":"dc057f545e7903334fa450da7e64a2fdf65fafd25704e7e829891c53cde1c844"}
Nov 03 17:20:37.078599 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:37.078577    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t" event={"ID":"510707fc-6270-4156-a83f-8d4ce5fbcfa0","Type":"ContainerStarted","Data":"da5c3ac3201fd652ddd9b05e5d3c2fcf947afa6fdeb662ca5c5de94fb1bb9751"}
Nov 03 17:20:37.079706 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:37.079684    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5" event={"ID":"aa53fe92-f3cd-4546-a6e8-beeadeb51142","Type":"ContainerStarted","Data":"7e1fb43009b18de6e9acfe0223dc648867d1fb55a0a4d76a823d7e192754e3ff"}
Nov 03 17:20:40.053673 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.053579    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:20:40.238483 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.238058    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:20:40.494909 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.494875    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"]
Nov 03 17:20:40.498909 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.498883    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.556459 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.556425    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 17:20:40.556621 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.556544    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 17:20:40.556621 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.556587    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 17:20:40.557226 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.556925    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-t4h2g\""
Nov 03 17:20:40.579455 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.579413    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c097188b-4b64-4636-822b-ab2c8817dfa0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.579630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.579462    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.579630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.579526    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.579630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.579555    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.579753 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.579723    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r95l\" (UniqueName: \"kubernetes.io/projected/c097188b-4b64-4636-822b-ab2c8817dfa0-kube-api-access-7r95l\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.579848 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.579830    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.579882 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.579871    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.579915 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.579904    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.579978 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.579961    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.593013 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.592849    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"]
Nov 03 17:20:40.618410 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.618349    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:20:40.682835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.681085    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.682835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.681143    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.682835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.681174    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.682835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.681223    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c097188b-4b64-4636-822b-ab2c8817dfa0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.682835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.681249    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.682835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.681302    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.682835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.681328    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.682835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.681394    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r95l\" (UniqueName: \"kubernetes.io/projected/c097188b-4b64-4636-822b-ab2c8817dfa0-kube-api-access-7r95l\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.682835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.681435    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.682835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.681874    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.682835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.682492    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.683538 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.682885    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.684305 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.683921    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.684305 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.684260    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c097188b-4b64-4636-822b-ab2c8817dfa0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.691061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.691028    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.693664 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.693637    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.749920 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.749844    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r95l\" (UniqueName: \"kubernetes.io/projected/c097188b-4b64-4636-822b-ab2c8817dfa0-kube-api-access-7r95l\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.753396 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.752931    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-x276d\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.805604 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.804828    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"]
Nov 03 17:20:40.817436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.815096    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.818026 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.817603    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:40.840254 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.840045    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:20:40.871658 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.871624    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-tpb4m\""
Nov 03 17:20:40.872127 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.872071    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"]
Nov 03 17:20:40.884800 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.884757    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58741805-9507-4add-ab80-651305453ee6-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.884961 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.884820    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58741805-9507-4add-ab80-651305453ee6-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.884961 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.884860    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58741805-9507-4add-ab80-651305453ee6-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.884961 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.884875    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58741805-9507-4add-ab80-651305453ee6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.884961 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.884900    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58741805-9507-4add-ab80-651305453ee6-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.884961 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.884920    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fql8l\" (UniqueName: \"kubernetes.io/projected/58741805-9507-4add-ab80-651305453ee6-kube-api-access-fql8l\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.884961 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.884948    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58741805-9507-4add-ab80-651305453ee6-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.885290 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.884970    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58741805-9507-4add-ab80-651305453ee6-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.885290 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.884990    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58741805-9507-4add-ab80-651305453ee6-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.985859 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.985819    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58741805-9507-4add-ab80-651305453ee6-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.986062 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.985873    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58741805-9507-4add-ab80-651305453ee6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.986062 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.985989    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58741805-9507-4add-ab80-651305453ee6-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.986062 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.986021    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fql8l\" (UniqueName: \"kubernetes.io/projected/58741805-9507-4add-ab80-651305453ee6-kube-api-access-fql8l\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.986242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.986066    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58741805-9507-4add-ab80-651305453ee6-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.986242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.986101    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58741805-9507-4add-ab80-651305453ee6-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.986242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.986129    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58741805-9507-4add-ab80-651305453ee6-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.986242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.986181    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58741805-9507-4add-ab80-651305453ee6-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.986242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.986225    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58741805-9507-4add-ab80-651305453ee6-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.987691 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.987660    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58741805-9507-4add-ab80-651305453ee6-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.988471 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.988445    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58741805-9507-4add-ab80-651305453ee6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.988790 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.988768    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58741805-9507-4add-ab80-651305453ee6-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.989058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.989037    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58741805-9507-4add-ab80-651305453ee6-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:40.989324 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:40.989300    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58741805-9507-4add-ab80-651305453ee6-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:41.003013 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:41.002857    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58741805-9507-4add-ab80-651305453ee6-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:41.003013 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:41.002865    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58741805-9507-4add-ab80-651305453ee6-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:41.035983 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:41.035885    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:20:41.048920 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:41.048855    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58741805-9507-4add-ab80-651305453ee6-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:41.057218 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:41.057164    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fql8l\" (UniqueName: \"kubernetes.io/projected/58741805-9507-4add-ab80-651305453ee6-kube-api-access-fql8l\") pod \"openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w\" (UID: \"58741805-9507-4add-ab80-651305453ee6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:41.152331 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:41.152290    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:41.231143 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:41.231102    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"]
Nov 03 17:20:41.564713 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:41.564628    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"]
Nov 03 17:20:41.575018 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:20:41.574906    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58741805_9507_4add_ab80_651305453ee6.slice/crio-62868b5a24f48a468b1fe2717d666d97fa763fdf52dabb66798da6b8c5d67c54 WatchSource:0}: Error finding container 62868b5a24f48a468b1fe2717d666d97fa763fdf52dabb66798da6b8c5d67c54: Status 404 returned error can't find the container with id 62868b5a24f48a468b1fe2717d666d97fa763fdf52dabb66798da6b8c5d67c54
Nov 03 17:20:42.125373 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:42.125325    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d" event={"ID":"c097188b-4b64-4636-822b-ab2c8817dfa0","Type":"ContainerStarted","Data":"630a0dc32c2dfc1973fed8972d1f81cbb8b3b9e8aafcf30cfd8334f00a79938f"}
Nov 03 17:20:42.127871 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:42.127835    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w" event={"ID":"58741805-9507-4add-ab80-651305453ee6","Type":"ContainerStarted","Data":"62868b5a24f48a468b1fe2717d666d97fa763fdf52dabb66798da6b8c5d67c54"}
Nov 03 17:20:42.943315 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:42.943281    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh"]
Nov 03 17:20:42.949895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:42.949861    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh"
Nov 03 17:20:43.009009 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:43.008968    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh"]
Nov 03 17:20:43.019994 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:43.019951    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggz88\" (UniqueName: \"kubernetes.io/projected/16640fa1-49c9-401a-9e45-c52df8a84a57-kube-api-access-ggz88\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh\" (UID: \"16640fa1-49c9-401a-9e45-c52df8a84a57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh"
Nov 03 17:20:43.020181 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:43.020006    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16640fa1-49c9-401a-9e45-c52df8a84a57-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh\" (UID: \"16640fa1-49c9-401a-9e45-c52df8a84a57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh"
Nov 03 17:20:43.020181 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:43.020046    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16640fa1-49c9-401a-9e45-c52df8a84a57-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh\" (UID: \"16640fa1-49c9-401a-9e45-c52df8a84a57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh"
Nov 03 17:20:43.121043 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:43.120997    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggz88\" (UniqueName: \"kubernetes.io/projected/16640fa1-49c9-401a-9e45-c52df8a84a57-kube-api-access-ggz88\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh\" (UID: \"16640fa1-49c9-401a-9e45-c52df8a84a57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh"
Nov 03 17:20:43.121206 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:43.121064    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16640fa1-49c9-401a-9e45-c52df8a84a57-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh\" (UID: \"16640fa1-49c9-401a-9e45-c52df8a84a57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh"
Nov 03 17:20:43.121206 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:43.121112    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16640fa1-49c9-401a-9e45-c52df8a84a57-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh\" (UID: \"16640fa1-49c9-401a-9e45-c52df8a84a57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh"
Nov 03 17:20:43.121852 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:43.121830    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16640fa1-49c9-401a-9e45-c52df8a84a57-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh\" (UID: \"16640fa1-49c9-401a-9e45-c52df8a84a57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh"
Nov 03 17:20:43.122655 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:43.122558    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16640fa1-49c9-401a-9e45-c52df8a84a57-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh\" (UID: \"16640fa1-49c9-401a-9e45-c52df8a84a57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh"
Nov 03 17:20:43.194266 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:43.194191    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggz88\" (UniqueName: \"kubernetes.io/projected/16640fa1-49c9-401a-9e45-c52df8a84a57-kube-api-access-ggz88\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh\" (UID: \"16640fa1-49c9-401a-9e45-c52df8a84a57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh"
Nov 03 17:20:43.274708 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:43.274672    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh"
Nov 03 17:20:43.703559 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:43.703519    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh"]
Nov 03 17:20:43.717119 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:20:43.717073    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16640fa1_49c9_401a_9e45_c52df8a84a57.slice/crio-21b32b8defc7f2ed8fbc7adae03cd4166b2600125d3cc033cc74cf1c0db68a39 WatchSource:0}: Error finding container 21b32b8defc7f2ed8fbc7adae03cd4166b2600125d3cc033cc74cf1c0db68a39: Status 404 returned error can't find the container with id 21b32b8defc7f2ed8fbc7adae03cd4166b2600125d3cc033cc74cf1c0db68a39
Nov 03 17:20:44.146435 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:44.146374    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh" event={"ID":"16640fa1-49c9-401a-9e45-c52df8a84a57","Type":"ContainerStarted","Data":"21b32b8defc7f2ed8fbc7adae03cd4166b2600125d3cc033cc74cf1c0db68a39"}
Nov 03 17:20:44.889246 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:44.889212    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:20:50.198825 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:50.198728    6059 generic.go:356] "Generic (PLEG): container finished" podID="a4d6dda2-d650-4cf5-b89f-26b5a8233844" containerID="e01f42df7c85b9b4d1456869eb4ecbb0d792f9af14574e7657b4b6a7307060b9" exitCode=0
Nov 03 17:20:50.198825 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:50.198809    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn" event={"ID":"a4d6dda2-d650-4cf5-b89f-26b5a8233844","Type":"ContainerDied","Data":"e01f42df7c85b9b4d1456869eb4ecbb0d792f9af14574e7657b4b6a7307060b9"}
Nov 03 17:20:50.611884 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:50.611834    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:20:50.612068 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:50.611927    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:20:50.612068 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:50.611973    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:20:50.641314 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:50.641281    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:20:50.641451 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:50.641351    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:20:50.641451 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:50.641429    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:20:51.205818 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:51.205730    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w" event={"ID":"58741805-9507-4add-ab80-651305453ee6","Type":"ContainerStarted","Data":"e3ccfd4045441133cb0a00c1430f4a6fce6c285cca13afd41706c0fbc15c505c"}
Nov 03 17:20:51.207692 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:51.207660    6059 generic.go:356] "Generic (PLEG): container finished" podID="a39b3295-6622-4055-b86f-c27840fb64b5" containerID="06423e60584b073dcdb1bc0fc84f19b4a692e81ad84533f36634f179625c7386" exitCode=0
Nov 03 17:20:51.207812 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:51.207754    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt" event={"ID":"a39b3295-6622-4055-b86f-c27840fb64b5","Type":"ContainerDied","Data":"06423e60584b073dcdb1bc0fc84f19b4a692e81ad84533f36634f179625c7386"}
Nov 03 17:20:51.209653 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:51.209588    6059 generic.go:356] "Generic (PLEG): container finished" podID="510707fc-6270-4156-a83f-8d4ce5fbcfa0" containerID="12c917a86202d4afa431ba5e9dd18987492defa57b8fb0bb30e8e7a1352a119f" exitCode=0
Nov 03 17:20:51.209802 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:51.209667    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t" event={"ID":"510707fc-6270-4156-a83f-8d4ce5fbcfa0","Type":"ContainerDied","Data":"12c917a86202d4afa431ba5e9dd18987492defa57b8fb0bb30e8e7a1352a119f"}
Nov 03 17:20:51.211710 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:51.211688    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d" event={"ID":"c097188b-4b64-4636-822b-ab2c8817dfa0","Type":"ContainerStarted","Data":"c982d01afb4ddc77ddff2d184fbae564c62dada763de0a7204ca4dc30b39c590"}
Nov 03 17:20:51.322777 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:51.322716    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w" podStartSLOduration=2.2902778870000002 podStartE2EDuration="11.322695762s" totalImagesPullingTime="9.032417875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:20:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:20:51.321286472 +0000 UTC m=+192.118999253" watchObservedRunningTime="2026-11-03 17:20:51.322695762 +0000 UTC m=+192.120408543"
Nov 03 17:20:51.663293 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:51.663236    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d" podStartSLOduration=2.273185042 podStartE2EDuration="11.66322232s" totalImagesPullingTime="9.390037278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:20:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:20:51.662167259 +0000 UTC m=+192.459880040" watchObservedRunningTime="2026-11-03 17:20:51.66322232 +0000 UTC m=+192.460935100"
Nov 03 17:20:51.818456 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:51.818323    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:51.823349 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:51.823318    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:52.152944 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:52.152865    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:52.157844 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:52.157816    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:52.217697 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:52.217665    6059 generic.go:356] "Generic (PLEG): container finished" podID="076f5e42-e2ae-4433-aa5a-6f8d6d30f75e" containerID="6729c08c379e0a5ff3dc0688874571389ba83a175d1848972cd7af3846a141b1" exitCode=0
Nov 03 17:20:52.218071 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:52.217744    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r" event={"ID":"076f5e42-e2ae-4433-aa5a-6f8d6d30f75e","Type":"ContainerDied","Data":"6729c08c379e0a5ff3dc0688874571389ba83a175d1848972cd7af3846a141b1"}
Nov 03 17:20:52.219732 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:52.219550    6059 generic.go:356] "Generic (PLEG): container finished" podID="aa53fe92-f3cd-4546-a6e8-beeadeb51142" containerID="44a3ea73964f76f78a50d64a7989958cb80975caa55eb5a6cae664b7760dc575" exitCode=0
Nov 03 17:20:52.219732 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:52.219667    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5" event={"ID":"aa53fe92-f3cd-4546-a6e8-beeadeb51142","Type":"ContainerDied","Data":"44a3ea73964f76f78a50d64a7989958cb80975caa55eb5a6cae664b7760dc575"}
Nov 03 17:20:52.220386 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:52.220000    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:52.220917 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:52.220886    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:52.221488 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:52.221470    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w"
Nov 03 17:20:52.221569 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:52.221531    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:20:53.228329 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:53.228303    6059 generic.go:356] "Generic (PLEG): container finished" podID="a4d6dda2-d650-4cf5-b89f-26b5a8233844" containerID="b3f52adb317373d9b525ad395f5b221f42b225458afdb889a01952a8ceebec22" exitCode=0
Nov 03 17:20:53.228734 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:53.228401    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn" event={"ID":"a4d6dda2-d650-4cf5-b89f-26b5a8233844","Type":"ContainerDied","Data":"b3f52adb317373d9b525ad395f5b221f42b225458afdb889a01952a8ceebec22"}
Nov 03 17:20:53.230080 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:53.230060    6059 generic.go:356] "Generic (PLEG): container finished" podID="16640fa1-49c9-401a-9e45-c52df8a84a57" containerID="714f8763eb8bc5e576cc5691e62c75ef989d55420361a9ffe0bbfb9974e39b7b" exitCode=0
Nov 03 17:20:53.230180 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:53.230139    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh" event={"ID":"16640fa1-49c9-401a-9e45-c52df8a84a57","Type":"ContainerDied","Data":"714f8763eb8bc5e576cc5691e62c75ef989d55420361a9ffe0bbfb9974e39b7b"}
Nov 03 17:20:53.233291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:53.232192    6059 generic.go:356] "Generic (PLEG): container finished" podID="a39b3295-6622-4055-b86f-c27840fb64b5" containerID="cec4e92206ab418ef7a290bc6bd76c263b343f0237294ca7372944b463f07fb4" exitCode=0
Nov 03 17:20:53.233291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:53.232392    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt" event={"ID":"a39b3295-6622-4055-b86f-c27840fb64b5","Type":"ContainerDied","Data":"cec4e92206ab418ef7a290bc6bd76c263b343f0237294ca7372944b463f07fb4"}
Nov 03 17:20:54.239645 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:54.239608    6059 generic.go:356] "Generic (PLEG): container finished" podID="16640fa1-49c9-401a-9e45-c52df8a84a57" containerID="7230fa230f81d0f1a1785592d65c9e0001c0cb4ad187f9877e816207cd23ee31" exitCode=0
Nov 03 17:20:54.240099 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:54.239754    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh" event={"ID":"16640fa1-49c9-401a-9e45-c52df8a84a57","Type":"ContainerDied","Data":"7230fa230f81d0f1a1785592d65c9e0001c0cb4ad187f9877e816207cd23ee31"}
Nov 03 17:20:54.242674 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:54.242645    6059 generic.go:356] "Generic (PLEG): container finished" podID="076f5e42-e2ae-4433-aa5a-6f8d6d30f75e" containerID="1370fcbe3bd57be3dfb217a4e2edae10abbe1bc6e2bb48b1300b6614b6baef51" exitCode=0
Nov 03 17:20:54.242792 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:54.242732    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r" event={"ID":"076f5e42-e2ae-4433-aa5a-6f8d6d30f75e","Type":"ContainerDied","Data":"1370fcbe3bd57be3dfb217a4e2edae10abbe1bc6e2bb48b1300b6614b6baef51"}
Nov 03 17:20:54.245505 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:54.245407    6059 generic.go:356] "Generic (PLEG): container finished" podID="510707fc-6270-4156-a83f-8d4ce5fbcfa0" containerID="66ae398928f614001db3815c2552b5b774bd0ccaa67e8550f8634bdb98005867" exitCode=0
Nov 03 17:20:54.245505 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:54.245487    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t" event={"ID":"510707fc-6270-4156-a83f-8d4ce5fbcfa0","Type":"ContainerDied","Data":"66ae398928f614001db3815c2552b5b774bd0ccaa67e8550f8634bdb98005867"}
Nov 03 17:20:54.248027 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:54.247997    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5" event={"ID":"aa53fe92-f3cd-4546-a6e8-beeadeb51142","Type":"ContainerStarted","Data":"6c9455036211c9862dbaa28d692ebaf24425a79b3ef924e67813ad48ac89881c"}
Nov 03 17:20:55.256154 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:55.256114    6059 generic.go:356] "Generic (PLEG): container finished" podID="aa53fe92-f3cd-4546-a6e8-beeadeb51142" containerID="6c9455036211c9862dbaa28d692ebaf24425a79b3ef924e67813ad48ac89881c" exitCode=0
Nov 03 17:20:55.256610 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:55.256178    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5" event={"ID":"aa53fe92-f3cd-4546-a6e8-beeadeb51142","Type":"ContainerDied","Data":"6c9455036211c9862dbaa28d692ebaf24425a79b3ef924e67813ad48ac89881c"}
Nov 03 17:20:55.926080 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:55.926050    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:20:56.228196 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:56.228028    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:20:57.294225 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:57.294190    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:20:57.997951 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:57.997911    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-672zj"]
Nov 03 17:20:58.003509 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:58.003471    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-672zj"
Nov 03 17:20:58.050120 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:58.050029    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-672zj"]
Nov 03 17:20:58.079315 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:58.079272    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91e37bb3-4d7d-4b8d-8eb4-6e206f6cc00d-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-672zj\" (UID: \"91e37bb3-4d7d-4b8d-8eb4-6e206f6cc00d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-672zj"
Nov 03 17:20:58.079533 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:58.079385    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9dtd\" (UniqueName: \"kubernetes.io/projected/91e37bb3-4d7d-4b8d-8eb4-6e206f6cc00d-kube-api-access-q9dtd\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-672zj\" (UID: \"91e37bb3-4d7d-4b8d-8eb4-6e206f6cc00d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-672zj"
Nov 03 17:20:58.096067 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:58.096004    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-ttvzz\""
Nov 03 17:20:58.096254 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:58.096125    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 03 17:20:58.096460 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:58.096430    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:20:58.181974 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:58.181494    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91e37bb3-4d7d-4b8d-8eb4-6e206f6cc00d-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-672zj\" (UID: \"91e37bb3-4d7d-4b8d-8eb4-6e206f6cc00d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-672zj"
Nov 03 17:20:58.181974 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:58.181921    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91e37bb3-4d7d-4b8d-8eb4-6e206f6cc00d-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-672zj\" (UID: \"91e37bb3-4d7d-4b8d-8eb4-6e206f6cc00d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-672zj"
Nov 03 17:20:58.182536 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:58.182473    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9dtd\" (UniqueName: \"kubernetes.io/projected/91e37bb3-4d7d-4b8d-8eb4-6e206f6cc00d-kube-api-access-q9dtd\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-672zj\" (UID: \"91e37bb3-4d7d-4b8d-8eb4-6e206f6cc00d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-672zj"
Nov 03 17:20:58.246500 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:58.246461    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9dtd\" (UniqueName: \"kubernetes.io/projected/91e37bb3-4d7d-4b8d-8eb4-6e206f6cc00d-kube-api-access-q9dtd\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-672zj\" (UID: \"91e37bb3-4d7d-4b8d-8eb4-6e206f6cc00d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-672zj"
Nov 03 17:20:58.322369 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:58.322270    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-672zj"
Nov 03 17:20:58.788421 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:58.788343    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-672zj"]
Nov 03 17:20:58.813397 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:20:58.813341    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e37bb3_4d7d_4b8d_8eb4_6e206f6cc00d.slice/crio-346aa59b005e30b09e79a5df22e40d9a3a4c977c3212641055bf62c035f52ad3 WatchSource:0}: Error finding container 346aa59b005e30b09e79a5df22e40d9a3a4c977c3212641055bf62c035f52ad3: Status 404 returned error can't find the container with id 346aa59b005e30b09e79a5df22e40d9a3a4c977c3212641055bf62c035f52ad3
Nov 03 17:20:59.288632 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:59.288545    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-672zj" event={"ID":"91e37bb3-4d7d-4b8d-8eb4-6e206f6cc00d","Type":"ContainerStarted","Data":"346aa59b005e30b09e79a5df22e40d9a3a4c977c3212641055bf62c035f52ad3"}
Nov 03 17:20:59.923389 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:20:59.923330    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:00.110097 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:00.110047    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:04.887029 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:04.886988    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:05.071382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:05.071316    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:06.882396 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:21:06.882282    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:21:07.378045 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:07.378003    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-672zj" event={"ID":"91e37bb3-4d7d-4b8d-8eb4-6e206f6cc00d","Type":"ContainerStarted","Data":"f2812e76ab2f4f26a4d7e6818f0a9762371740ed45e6f8878683f5614298225a"}
Nov 03 17:21:07.597813 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:07.597133    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-672zj" podStartSLOduration=2.925582913 podStartE2EDuration="10.597113993s" totalImagesPullingTime="7.67153108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:20:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:21:07.549012813 +0000 UTC m=+208.346725598" watchObservedRunningTime="2026-11-03 17:21:07.597113993 +0000 UTC m=+208.394826775"
Nov 03 17:21:08.547898 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:08.547828    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:08.867711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:08.867595    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:09.031511 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:09.031469    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-rsqmx"]
Nov 03 17:21:09.045121 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:09.045078    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rsqmx"
Nov 03 17:21:09.093296 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:09.093258    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-rsqmx"]
Nov 03 17:21:09.139079 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:09.139050    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 17:21:09.139079 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:09.139068    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 17:21:09.206890 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:09.206853    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7e466b7-1d0e-43ff-b180-7cf585679a8f-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-rsqmx\" (UID: \"a7e466b7-1d0e-43ff-b180-7cf585679a8f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rsqmx"
Nov 03 17:21:09.207052 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:09.207028    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sskt\" (UniqueName: \"kubernetes.io/projected/a7e466b7-1d0e-43ff-b180-7cf585679a8f-kube-api-access-2sskt\") pod \"cert-manager-webhook-5dbbbf75cc-rsqmx\" (UID: \"a7e466b7-1d0e-43ff-b180-7cf585679a8f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rsqmx"
Nov 03 17:21:09.308134 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:09.307903    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7e466b7-1d0e-43ff-b180-7cf585679a8f-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-rsqmx\" (UID: \"a7e466b7-1d0e-43ff-b180-7cf585679a8f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rsqmx"
Nov 03 17:21:09.308134 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:09.308042    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sskt\" (UniqueName: \"kubernetes.io/projected/a7e466b7-1d0e-43ff-b180-7cf585679a8f-kube-api-access-2sskt\") pod \"cert-manager-webhook-5dbbbf75cc-rsqmx\" (UID: \"a7e466b7-1d0e-43ff-b180-7cf585679a8f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rsqmx"
Nov 03 17:21:09.380011 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:09.379978    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7e466b7-1d0e-43ff-b180-7cf585679a8f-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-rsqmx\" (UID: \"a7e466b7-1d0e-43ff-b180-7cf585679a8f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rsqmx"
Nov 03 17:21:09.380184 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:09.380074    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sskt\" (UniqueName: \"kubernetes.io/projected/a7e466b7-1d0e-43ff-b180-7cf585679a8f-kube-api-access-2sskt\") pod \"cert-manager-webhook-5dbbbf75cc-rsqmx\" (UID: \"a7e466b7-1d0e-43ff-b180-7cf585679a8f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rsqmx"
Nov 03 17:21:09.393203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:09.393170    6059 generic.go:356] "Generic (PLEG): container finished" podID="a4d6dda2-d650-4cf5-b89f-26b5a8233844" containerID="33f9b0ab1afe2160a98e1a51e7dc8a892408cf5ff58feabdd4b313e09baf5ada" exitCode=0
Nov 03 17:21:09.393339 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:09.393262    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn" event={"ID":"a4d6dda2-d650-4cf5-b89f-26b5a8233844","Type":"ContainerDied","Data":"33f9b0ab1afe2160a98e1a51e7dc8a892408cf5ff58feabdd4b313e09baf5ada"}
Nov 03 17:21:09.393519 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:09.393497    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rsqmx"
Nov 03 17:21:09.396261 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:09.396189    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt" event={"ID":"a39b3295-6622-4055-b86f-c27840fb64b5","Type":"ContainerStarted","Data":"ece0bb308f4af14e783992f348cd338cb9aee9c06d3989245a80a59d386cf779"}
Nov 03 17:21:09.624896 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:09.624831    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt" podStartSLOduration=2.863307046 podStartE2EDuration="34.624814888s" totalImagesPullingTime="31.761507842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:20:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:21:09.624277086 +0000 UTC m=+210.421989878" watchObservedRunningTime="2026-11-03 17:21:09.624814888 +0000 UTC m=+210.422527648"
Nov 03 17:21:09.799414 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:09.799370    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-rsqmx"]
Nov 03 17:21:09.806883 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:21:09.806851    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7e466b7_1d0e_43ff_b180_7cf585679a8f.slice/crio-7de69a8227a1812b677a97392d967f692b8d395dcf65050aba63f91990290e00 WatchSource:0}: Error finding container 7de69a8227a1812b677a97392d967f692b8d395dcf65050aba63f91990290e00: Status 404 returned error can't find the container with id 7de69a8227a1812b677a97392d967f692b8d395dcf65050aba63f91990290e00
Nov 03 17:21:10.408412 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.408377    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rsqmx" event={"ID":"a7e466b7-1d0e-43ff-b180-7cf585679a8f","Type":"ContainerStarted","Data":"7de69a8227a1812b677a97392d967f692b8d395dcf65050aba63f91990290e00"}
Nov 03 17:21:10.410835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.410804    6059 generic.go:356] "Generic (PLEG): container finished" podID="16640fa1-49c9-401a-9e45-c52df8a84a57" containerID="3246710b44e6f776497d5e622ac1c652e5d751149e2f460262b7ca41029bf1dd" exitCode=0
Nov 03 17:21:10.410979 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.410889    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh" event={"ID":"16640fa1-49c9-401a-9e45-c52df8a84a57","Type":"ContainerDied","Data":"3246710b44e6f776497d5e622ac1c652e5d751149e2f460262b7ca41029bf1dd"}
Nov 03 17:21:10.412872 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.412849    6059 generic.go:356] "Generic (PLEG): container finished" podID="a39b3295-6622-4055-b86f-c27840fb64b5" containerID="ece0bb308f4af14e783992f348cd338cb9aee9c06d3989245a80a59d386cf779" exitCode=0
Nov 03 17:21:10.412996 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.412922    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt" event={"ID":"a39b3295-6622-4055-b86f-c27840fb64b5","Type":"ContainerDied","Data":"ece0bb308f4af14e783992f348cd338cb9aee9c06d3989245a80a59d386cf779"}
Nov 03 17:21:10.415169 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.415151    6059 generic.go:356] "Generic (PLEG): container finished" podID="076f5e42-e2ae-4433-aa5a-6f8d6d30f75e" containerID="62dcbaaba9d732494e207fc704a52d9aa876da5ebdc7369ab690d9103b023032" exitCode=0
Nov 03 17:21:10.415265 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.415217    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r" event={"ID":"076f5e42-e2ae-4433-aa5a-6f8d6d30f75e","Type":"ContainerDied","Data":"62dcbaaba9d732494e207fc704a52d9aa876da5ebdc7369ab690d9103b023032"}
Nov 03 17:21:10.416800 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.416782    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 17:21:10.417413 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.417393    6059 generic.go:356] "Generic (PLEG): container finished" podID="510707fc-6270-4156-a83f-8d4ce5fbcfa0" containerID="2092d21181f3b00c065e0b133015a7f89c47a5e0b992b93018d296de23a82746" exitCode=1
Nov 03 17:21:10.417499 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.417467    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t" event={"ID":"510707fc-6270-4156-a83f-8d4ce5fbcfa0","Type":"ContainerDied","Data":"2092d21181f3b00c065e0b133015a7f89c47a5e0b992b93018d296de23a82746"}
Nov 03 17:21:10.419549 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.419528    6059 generic.go:356] "Generic (PLEG): container finished" podID="aa53fe92-f3cd-4546-a6e8-beeadeb51142" containerID="f2fff514e738977d314926ad2cda1a2de83def65b92ede4e48c1188b7180b290" exitCode=0
Nov 03 17:21:10.419640 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.419612    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5" event={"ID":"aa53fe92-f3cd-4546-a6e8-beeadeb51142","Type":"ContainerDied","Data":"f2fff514e738977d314926ad2cda1a2de83def65b92ede4e48c1188b7180b290"}
Nov 03 17:21:10.549742 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.549710    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn"
Nov 03 17:21:10.629777 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.629692    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:10.718580 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.718540    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d6dda2-d650-4cf5-b89f-26b5a8233844-kube-api-access-hp6hp\" (UniqueName: \"kubernetes.io/projected/a4d6dda2-d650-4cf5-b89f-26b5a8233844-kube-api-access-hp6hp\") pod \"a4d6dda2-d650-4cf5-b89f-26b5a8233844\" (UID: \"a4d6dda2-d650-4cf5-b89f-26b5a8233844\") "
Nov 03 17:21:10.718796 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.718595    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d6dda2-d650-4cf5-b89f-26b5a8233844-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4d6dda2-d650-4cf5-b89f-26b5a8233844-bundle\") pod \"a4d6dda2-d650-4cf5-b89f-26b5a8233844\" (UID: \"a4d6dda2-d650-4cf5-b89f-26b5a8233844\") "
Nov 03 17:21:10.718796 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.718750    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d6dda2-d650-4cf5-b89f-26b5a8233844-util\" (UniqueName: \"kubernetes.io/empty-dir/a4d6dda2-d650-4cf5-b89f-26b5a8233844-util\") pod \"a4d6dda2-d650-4cf5-b89f-26b5a8233844\" (UID: \"a4d6dda2-d650-4cf5-b89f-26b5a8233844\") "
Nov 03 17:21:10.719123 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.719093    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d6dda2-d650-4cf5-b89f-26b5a8233844-bundle" pod "a4d6dda2-d650-4cf5-b89f-26b5a8233844" (UID: "a4d6dda2-d650-4cf5-b89f-26b5a8233844"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:21:10.720899 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.720873    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d6dda2-d650-4cf5-b89f-26b5a8233844-kube-api-access-hp6hp" pod "a4d6dda2-d650-4cf5-b89f-26b5a8233844" (UID: "a4d6dda2-d650-4cf5-b89f-26b5a8233844"). InnerVolumeSpecName "kube-api-access-hp6hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:21:10.723230 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.723203    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d6dda2-d650-4cf5-b89f-26b5a8233844-util" pod "a4d6dda2-d650-4cf5-b89f-26b5a8233844" (UID: "a4d6dda2-d650-4cf5-b89f-26b5a8233844"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:21:10.820515 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.820480    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp6hp\" (UniqueName: \"kubernetes.io/projected/a4d6dda2-d650-4cf5-b89f-26b5a8233844-kube-api-access-hp6hp\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:10.820515 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.820511    6059 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4d6dda2-d650-4cf5-b89f-26b5a8233844-bundle\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:10.820515 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:10.820522    6059 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4d6dda2-d650-4cf5-b89f-26b5a8233844-util\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:11.347136 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.347103    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:11.426663 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.426578    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn" event={"ID":"a4d6dda2-d650-4cf5-b89f-26b5a8233844","Type":"ContainerDied","Data":"ee270ae000e1f1193f2a005ccfe32e5946458f6254bbfd651657e7daa72644f3"}
Nov 03 17:21:11.426663 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.426636    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee270ae000e1f1193f2a005ccfe32e5946458f6254bbfd651657e7daa72644f3"
Nov 03 17:21:11.428250 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.428197    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hx8fn"
Nov 03 17:21:11.625749 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.625723    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt"
Nov 03 17:21:11.629931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.629731    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5"
Nov 03 17:21:11.638333 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.638310    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 17:21:11.641801 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.641778    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t"
Nov 03 17:21:11.652900 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.652876    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r"
Nov 03 17:21:11.655813 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.655796    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh"
Nov 03 17:21:11.672081 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.672051    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:11.726809 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.726774    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510707fc-6270-4156-a83f-8d4ce5fbcfa0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/510707fc-6270-4156-a83f-8d4ce5fbcfa0-bundle\") pod \"510707fc-6270-4156-a83f-8d4ce5fbcfa0\" (UID: \"510707fc-6270-4156-a83f-8d4ce5fbcfa0\") "
Nov 03 17:21:11.726990 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.726831    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39b3295-6622-4055-b86f-c27840fb64b5-kube-api-access-522br\" (UniqueName: \"kubernetes.io/projected/a39b3295-6622-4055-b86f-c27840fb64b5-kube-api-access-522br\") pod \"a39b3295-6622-4055-b86f-c27840fb64b5\" (UID: \"a39b3295-6622-4055-b86f-c27840fb64b5\") "
Nov 03 17:21:11.726990 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.726881    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39b3295-6622-4055-b86f-c27840fb64b5-util\" (UniqueName: \"kubernetes.io/empty-dir/a39b3295-6622-4055-b86f-c27840fb64b5-util\") pod \"a39b3295-6622-4055-b86f-c27840fb64b5\" (UID: \"a39b3295-6622-4055-b86f-c27840fb64b5\") "
Nov 03 17:21:11.726990 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.726947    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa53fe92-f3cd-4546-a6e8-beeadeb51142-util\" (UniqueName: \"kubernetes.io/empty-dir/aa53fe92-f3cd-4546-a6e8-beeadeb51142-util\") pod \"aa53fe92-f3cd-4546-a6e8-beeadeb51142\" (UID: \"aa53fe92-f3cd-4546-a6e8-beeadeb51142\") "
Nov 03 17:21:11.726990 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.726979    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39b3295-6622-4055-b86f-c27840fb64b5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a39b3295-6622-4055-b86f-c27840fb64b5-bundle\") pod \"a39b3295-6622-4055-b86f-c27840fb64b5\" (UID: \"a39b3295-6622-4055-b86f-c27840fb64b5\") "
Nov 03 17:21:11.727203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.727002    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa53fe92-f3cd-4546-a6e8-beeadeb51142-kube-api-access-t98lx\" (UniqueName: \"kubernetes.io/projected/aa53fe92-f3cd-4546-a6e8-beeadeb51142-kube-api-access-t98lx\") pod \"aa53fe92-f3cd-4546-a6e8-beeadeb51142\" (UID: \"aa53fe92-f3cd-4546-a6e8-beeadeb51142\") "
Nov 03 17:21:11.727203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.727029    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510707fc-6270-4156-a83f-8d4ce5fbcfa0-util\" (UniqueName: \"kubernetes.io/empty-dir/510707fc-6270-4156-a83f-8d4ce5fbcfa0-util\") pod \"510707fc-6270-4156-a83f-8d4ce5fbcfa0\" (UID: \"510707fc-6270-4156-a83f-8d4ce5fbcfa0\") "
Nov 03 17:21:11.727203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.727059    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa53fe92-f3cd-4546-a6e8-beeadeb51142-bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa53fe92-f3cd-4546-a6e8-beeadeb51142-bundle\") pod \"aa53fe92-f3cd-4546-a6e8-beeadeb51142\" (UID: \"aa53fe92-f3cd-4546-a6e8-beeadeb51142\") "
Nov 03 17:21:11.727203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.727104    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510707fc-6270-4156-a83f-8d4ce5fbcfa0-kube-api-access-p44rx\" (UniqueName: \"kubernetes.io/projected/510707fc-6270-4156-a83f-8d4ce5fbcfa0-kube-api-access-p44rx\") pod \"510707fc-6270-4156-a83f-8d4ce5fbcfa0\" (UID: \"510707fc-6270-4156-a83f-8d4ce5fbcfa0\") "
Nov 03 17:21:11.727759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.727724    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39b3295-6622-4055-b86f-c27840fb64b5-bundle" pod "a39b3295-6622-4055-b86f-c27840fb64b5" (UID: "a39b3295-6622-4055-b86f-c27840fb64b5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:21:11.728107 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.728081    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510707fc-6270-4156-a83f-8d4ce5fbcfa0-bundle" pod "510707fc-6270-4156-a83f-8d4ce5fbcfa0" (UID: "510707fc-6270-4156-a83f-8d4ce5fbcfa0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:21:11.728383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.728331    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa53fe92-f3cd-4546-a6e8-beeadeb51142-bundle" pod "aa53fe92-f3cd-4546-a6e8-beeadeb51142" (UID: "aa53fe92-f3cd-4546-a6e8-beeadeb51142"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:21:11.729632 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.729602    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510707fc-6270-4156-a83f-8d4ce5fbcfa0-kube-api-access-p44rx" pod "510707fc-6270-4156-a83f-8d4ce5fbcfa0" (UID: "510707fc-6270-4156-a83f-8d4ce5fbcfa0"). InnerVolumeSpecName "kube-api-access-p44rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:21:11.729722 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.729688    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39b3295-6622-4055-b86f-c27840fb64b5-kube-api-access-522br" pod "a39b3295-6622-4055-b86f-c27840fb64b5" (UID: "a39b3295-6622-4055-b86f-c27840fb64b5"). InnerVolumeSpecName "kube-api-access-522br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:21:11.729811 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.729790    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa53fe92-f3cd-4546-a6e8-beeadeb51142-kube-api-access-t98lx" pod "aa53fe92-f3cd-4546-a6e8-beeadeb51142" (UID: "aa53fe92-f3cd-4546-a6e8-beeadeb51142"). InnerVolumeSpecName "kube-api-access-t98lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:21:11.731973 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.731952    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39b3295-6622-4055-b86f-c27840fb64b5-util" pod "a39b3295-6622-4055-b86f-c27840fb64b5" (UID: "a39b3295-6622-4055-b86f-c27840fb64b5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:21:11.734392 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.734367    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510707fc-6270-4156-a83f-8d4ce5fbcfa0-util" pod "510707fc-6270-4156-a83f-8d4ce5fbcfa0" (UID: "510707fc-6270-4156-a83f-8d4ce5fbcfa0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:21:11.734479 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.734421    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa53fe92-f3cd-4546-a6e8-beeadeb51142-util" pod "aa53fe92-f3cd-4546-a6e8-beeadeb51142" (UID: "aa53fe92-f3cd-4546-a6e8-beeadeb51142"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:21:11.828691 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.828650    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076f5e42-e2ae-4433-aa5a-6f8d6d30f75e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/076f5e42-e2ae-4433-aa5a-6f8d6d30f75e-bundle\") pod \"076f5e42-e2ae-4433-aa5a-6f8d6d30f75e\" (UID: \"076f5e42-e2ae-4433-aa5a-6f8d6d30f75e\") "
Nov 03 17:21:11.828886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.828759    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076f5e42-e2ae-4433-aa5a-6f8d6d30f75e-util\" (UniqueName: \"kubernetes.io/empty-dir/076f5e42-e2ae-4433-aa5a-6f8d6d30f75e-util\") pod \"076f5e42-e2ae-4433-aa5a-6f8d6d30f75e\" (UID: \"076f5e42-e2ae-4433-aa5a-6f8d6d30f75e\") "
Nov 03 17:21:11.828886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.828797    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16640fa1-49c9-401a-9e45-c52df8a84a57-bundle\" (UniqueName: \"kubernetes.io/empty-dir/16640fa1-49c9-401a-9e45-c52df8a84a57-bundle\") pod \"16640fa1-49c9-401a-9e45-c52df8a84a57\" (UID: \"16640fa1-49c9-401a-9e45-c52df8a84a57\") "
Nov 03 17:21:11.828886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.828846    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16640fa1-49c9-401a-9e45-c52df8a84a57-kube-api-access-ggz88\" (UniqueName: \"kubernetes.io/projected/16640fa1-49c9-401a-9e45-c52df8a84a57-kube-api-access-ggz88\") pod \"16640fa1-49c9-401a-9e45-c52df8a84a57\" (UID: \"16640fa1-49c9-401a-9e45-c52df8a84a57\") "
Nov 03 17:21:11.829057 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.828917    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/076f5e42-e2ae-4433-aa5a-6f8d6d30f75e-kube-api-access-h58hs\" (UniqueName: \"kubernetes.io/projected/076f5e42-e2ae-4433-aa5a-6f8d6d30f75e-kube-api-access-h58hs\") pod \"076f5e42-e2ae-4433-aa5a-6f8d6d30f75e\" (UID: \"076f5e42-e2ae-4433-aa5a-6f8d6d30f75e\") "
Nov 03 17:21:11.829057 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.828968    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16640fa1-49c9-401a-9e45-c52df8a84a57-util\" (UniqueName: \"kubernetes.io/empty-dir/16640fa1-49c9-401a-9e45-c52df8a84a57-util\") pod \"16640fa1-49c9-401a-9e45-c52df8a84a57\" (UID: \"16640fa1-49c9-401a-9e45-c52df8a84a57\") "
Nov 03 17:21:11.829171 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.829144    6059 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/510707fc-6270-4156-a83f-8d4ce5fbcfa0-bundle\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:11.829171 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.829158    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-522br\" (UniqueName: \"kubernetes.io/projected/a39b3295-6622-4055-b86f-c27840fb64b5-kube-api-access-522br\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:11.829171 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.829169    6059 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a39b3295-6622-4055-b86f-c27840fb64b5-util\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:11.829321 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.829177    6059 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa53fe92-f3cd-4546-a6e8-beeadeb51142-util\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:11.829321 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.829185    6059 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a39b3295-6622-4055-b86f-c27840fb64b5-bundle\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:11.829321 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.829193    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t98lx\" (UniqueName: \"kubernetes.io/projected/aa53fe92-f3cd-4546-a6e8-beeadeb51142-kube-api-access-t98lx\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:11.829321 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.829201    6059 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/510707fc-6270-4156-a83f-8d4ce5fbcfa0-util\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:11.829321 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.829209    6059 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa53fe92-f3cd-4546-a6e8-beeadeb51142-bundle\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:11.829321 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.829217    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p44rx\" (UniqueName: \"kubernetes.io/projected/510707fc-6270-4156-a83f-8d4ce5fbcfa0-kube-api-access-p44rx\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:11.829845 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.829815    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076f5e42-e2ae-4433-aa5a-6f8d6d30f75e-bundle" pod "076f5e42-e2ae-4433-aa5a-6f8d6d30f75e" (UID: "076f5e42-e2ae-4433-aa5a-6f8d6d30f75e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:21:11.829960 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.829932    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16640fa1-49c9-401a-9e45-c52df8a84a57-bundle" pod "16640fa1-49c9-401a-9e45-c52df8a84a57" (UID: "16640fa1-49c9-401a-9e45-c52df8a84a57"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:21:11.831269 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.831245    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16640fa1-49c9-401a-9e45-c52df8a84a57-kube-api-access-ggz88" pod "16640fa1-49c9-401a-9e45-c52df8a84a57" (UID: "16640fa1-49c9-401a-9e45-c52df8a84a57"). InnerVolumeSpecName "kube-api-access-ggz88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:21:11.831682 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.831660    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076f5e42-e2ae-4433-aa5a-6f8d6d30f75e-kube-api-access-h58hs" pod "076f5e42-e2ae-4433-aa5a-6f8d6d30f75e" (UID: "076f5e42-e2ae-4433-aa5a-6f8d6d30f75e"). InnerVolumeSpecName "kube-api-access-h58hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:21:11.834403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.834374    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16640fa1-49c9-401a-9e45-c52df8a84a57-util" pod "16640fa1-49c9-401a-9e45-c52df8a84a57" (UID: "16640fa1-49c9-401a-9e45-c52df8a84a57"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:21:11.835684 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.835661    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076f5e42-e2ae-4433-aa5a-6f8d6d30f75e-util" pod "076f5e42-e2ae-4433-aa5a-6f8d6d30f75e" (UID: "076f5e42-e2ae-4433-aa5a-6f8d6d30f75e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:21:11.861781 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.861704    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:11.930231 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.930194    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h58hs\" (UniqueName: \"kubernetes.io/projected/076f5e42-e2ae-4433-aa5a-6f8d6d30f75e-kube-api-access-h58hs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:11.930231 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.930223    6059 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16640fa1-49c9-401a-9e45-c52df8a84a57-util\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:11.930231 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.930234    6059 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/076f5e42-e2ae-4433-aa5a-6f8d6d30f75e-bundle\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:11.930231 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.930242    6059 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/076f5e42-e2ae-4433-aa5a-6f8d6d30f75e-util\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:11.930560 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.930250    6059 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16640fa1-49c9-401a-9e45-c52df8a84a57-bundle\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:11.930560 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:11.930260    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggz88\" (UniqueName: \"kubernetes.io/projected/16640fa1-49c9-401a-9e45-c52df8a84a57-kube-api-access-ggz88\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:12.434116 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:12.434085    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r"
Nov 03 17:21:12.434339 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:12.434078    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bsp4r" event={"ID":"076f5e42-e2ae-4433-aa5a-6f8d6d30f75e","Type":"ContainerDied","Data":"dc057f545e7903334fa450da7e64a2fdf65fafd25704e7e829891c53cde1c844"}
Nov 03 17:21:12.434339 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:12.434222    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc057f545e7903334fa450da7e64a2fdf65fafd25704e7e829891c53cde1c844"
Nov 03 17:21:12.436235 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:12.436203    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 17:21:12.437440 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:12.437421    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t"
Nov 03 17:21:12.437543 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:12.437413    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t" event={"ID":"510707fc-6270-4156-a83f-8d4ce5fbcfa0","Type":"ContainerDied","Data":"da5c3ac3201fd652ddd9b05e5d3c2fcf947afa6fdeb662ca5c5de94fb1bb9751"}
Nov 03 17:21:12.437611 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:12.437546    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da5c3ac3201fd652ddd9b05e5d3c2fcf947afa6fdeb662ca5c5de94fb1bb9751"
Nov 03 17:21:12.440569 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:12.440521    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5"
Nov 03 17:21:12.440671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:12.440566    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxfng5" event={"ID":"aa53fe92-f3cd-4546-a6e8-beeadeb51142","Type":"ContainerDied","Data":"7e1fb43009b18de6e9acfe0223dc648867d1fb55a0a4d76a823d7e192754e3ff"}
Nov 03 17:21:12.440671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:12.440610    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e1fb43009b18de6e9acfe0223dc648867d1fb55a0a4d76a823d7e192754e3ff"
Nov 03 17:21:12.450065 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:12.450034    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh" event={"ID":"16640fa1-49c9-401a-9e45-c52df8a84a57","Type":"ContainerDied","Data":"21b32b8defc7f2ed8fbc7adae03cd4166b2600125d3cc033cc74cf1c0db68a39"}
Nov 03 17:21:12.450185 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:12.450089    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21b32b8defc7f2ed8fbc7adae03cd4166b2600125d3cc033cc74cf1c0db68a39"
Nov 03 17:21:12.450185 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:12.450043    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akxvqh"
Nov 03 17:21:12.453828 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:12.453797    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt" event={"ID":"a39b3295-6622-4055-b86f-c27840fb64b5","Type":"ContainerDied","Data":"47a8c4f3fe7f068cff0addc21eb38eb1beab9c1ddd8d5556392957d55c2246e7"}
Nov 03 17:21:12.453931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:12.453838    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47a8c4f3fe7f068cff0addc21eb38eb1beab9c1ddd8d5556392957d55c2246e7"
Nov 03 17:21:12.453931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:12.453852    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487xt"
Nov 03 17:21:14.465090 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:14.465049    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rsqmx" event={"ID":"a7e466b7-1d0e-43ff-b180-7cf585679a8f","Type":"ContainerStarted","Data":"d3f4754f8aa664ce65f32af37b89b02d7970de2d066aa99bd09cc0d99091aaf7"}
Nov 03 17:21:14.465533 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:14.465185    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rsqmx"
Nov 03 17:21:14.590126 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:14.590093    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:14.634574 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:14.634524    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rsqmx" podStartSLOduration=2.6848491230000002 podStartE2EDuration="6.634508319s" totalImagesPullingTime="3.949659196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:21:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:21:14.634348748 +0000 UTC m=+215.432061529" watchObservedRunningTime="2026-11-03 17:21:14.634508319 +0000 UTC m=+215.432221101"
Nov 03 17:21:14.776031 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:14.775946    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:16.873733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:16.873703    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:17.066272 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:17.066237    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:17.257872 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:17.257843    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:18.203195 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.203164    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:18.434293 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434257    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"]
Nov 03 17:21:18.434626 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434610    6059 cpu_manager.go:414] "containerMap: removing container" podUID="510707fc-6270-4156-a83f-8d4ce5fbcfa0" containerName="pull"
Nov 03 17:21:18.434680 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434628    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510707fc-6270-4156-a83f-8d4ce5fbcfa0" containerName="pull"
Nov 03 17:21:18.434680 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434645    6059 cpu_manager.go:414] "containerMap: removing container" podUID="a4d6dda2-d650-4cf5-b89f-26b5a8233844" containerName="util"
Nov 03 17:21:18.434680 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434652    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4d6dda2-d650-4cf5-b89f-26b5a8233844" containerName="util"
Nov 03 17:21:18.434680 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434663    6059 cpu_manager.go:414] "containerMap: removing container" podUID="510707fc-6270-4156-a83f-8d4ce5fbcfa0" containerName="util"
Nov 03 17:21:18.434680 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434672    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510707fc-6270-4156-a83f-8d4ce5fbcfa0" containerName="util"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434685    6059 cpu_manager.go:414] "containerMap: removing container" podUID="a4d6dda2-d650-4cf5-b89f-26b5a8233844" containerName="extract"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434693    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4d6dda2-d650-4cf5-b89f-26b5a8233844" containerName="extract"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434704    6059 cpu_manager.go:414] "containerMap: removing container" podUID="510707fc-6270-4156-a83f-8d4ce5fbcfa0" containerName="extract"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434708    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510707fc-6270-4156-a83f-8d4ce5fbcfa0" containerName="extract"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434717    6059 cpu_manager.go:414] "containerMap: removing container" podUID="076f5e42-e2ae-4433-aa5a-6f8d6d30f75e" containerName="pull"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434722    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="076f5e42-e2ae-4433-aa5a-6f8d6d30f75e" containerName="pull"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434730    6059 cpu_manager.go:414] "containerMap: removing container" podUID="aa53fe92-f3cd-4546-a6e8-beeadeb51142" containerName="pull"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434736    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa53fe92-f3cd-4546-a6e8-beeadeb51142" containerName="pull"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434748    6059 cpu_manager.go:414] "containerMap: removing container" podUID="16640fa1-49c9-401a-9e45-c52df8a84a57" containerName="util"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434755    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16640fa1-49c9-401a-9e45-c52df8a84a57" containerName="util"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434766    6059 cpu_manager.go:414] "containerMap: removing container" podUID="a39b3295-6622-4055-b86f-c27840fb64b5" containerName="extract"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434773    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39b3295-6622-4055-b86f-c27840fb64b5" containerName="extract"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434781    6059 cpu_manager.go:414] "containerMap: removing container" podUID="aa53fe92-f3cd-4546-a6e8-beeadeb51142" containerName="extract"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434785    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa53fe92-f3cd-4546-a6e8-beeadeb51142" containerName="extract"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434792    6059 cpu_manager.go:414] "containerMap: removing container" podUID="076f5e42-e2ae-4433-aa5a-6f8d6d30f75e" containerName="extract"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434797    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="076f5e42-e2ae-4433-aa5a-6f8d6d30f75e" containerName="extract"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434809    6059 cpu_manager.go:414] "containerMap: removing container" podUID="16640fa1-49c9-401a-9e45-c52df8a84a57" containerName="pull"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434813    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16640fa1-49c9-401a-9e45-c52df8a84a57" containerName="pull"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434821    6059 cpu_manager.go:414] "containerMap: removing container" podUID="a39b3295-6622-4055-b86f-c27840fb64b5" containerName="util"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434826    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39b3295-6622-4055-b86f-c27840fb64b5" containerName="util"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434835    6059 cpu_manager.go:414] "containerMap: removing container" podUID="a39b3295-6622-4055-b86f-c27840fb64b5" containerName="pull"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434840    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39b3295-6622-4055-b86f-c27840fb64b5" containerName="pull"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434848    6059 cpu_manager.go:414] "containerMap: removing container" podUID="16640fa1-49c9-401a-9e45-c52df8a84a57" containerName="extract"
Nov 03 17:21:18.434846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434854    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16640fa1-49c9-401a-9e45-c52df8a84a57" containerName="extract"
Nov 03 17:21:18.435880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434865    6059 cpu_manager.go:414] "containerMap: removing container" podUID="aa53fe92-f3cd-4546-a6e8-beeadeb51142" containerName="util"
Nov 03 17:21:18.435880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434872    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa53fe92-f3cd-4546-a6e8-beeadeb51142" containerName="util"
Nov 03 17:21:18.435880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434888    6059 cpu_manager.go:414] "containerMap: removing container" podUID="076f5e42-e2ae-4433-aa5a-6f8d6d30f75e" containerName="util"
Nov 03 17:21:18.435880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434895    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="076f5e42-e2ae-4433-aa5a-6f8d6d30f75e" containerName="util"
Nov 03 17:21:18.435880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434905    6059 cpu_manager.go:414] "containerMap: removing container" podUID="a4d6dda2-d650-4cf5-b89f-26b5a8233844" containerName="pull"
Nov 03 17:21:18.435880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434912    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4d6dda2-d650-4cf5-b89f-26b5a8233844" containerName="pull"
Nov 03 17:21:18.435880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.434992    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa53fe92-f3cd-4546-a6e8-beeadeb51142" containerName="extract"
Nov 03 17:21:18.435880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.435003    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="076f5e42-e2ae-4433-aa5a-6f8d6d30f75e" containerName="extract"
Nov 03 17:21:18.435880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.435017    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="16640fa1-49c9-401a-9e45-c52df8a84a57" containerName="extract"
Nov 03 17:21:18.435880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.435030    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4d6dda2-d650-4cf5-b89f-26b5a8233844" containerName="extract"
Nov 03 17:21:18.435880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.435039    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="510707fc-6270-4156-a83f-8d4ce5fbcfa0" containerName="extract"
Nov 03 17:21:18.435880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.435049    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a39b3295-6622-4055-b86f-c27840fb64b5" containerName="extract"
Nov 03 17:21:18.453157 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.453126    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"
Nov 03 17:21:18.509504 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.509469    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:18.567523 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.567491    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 17:21:18.567780 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.567767    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 17:21:18.568154 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.568127    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 17:21:18.568198 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.568173    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 17:21:18.568293 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.568278    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-n7pls\""
Nov 03 17:21:18.568329 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.568292    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 17:21:18.580598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.580564    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"]
Nov 03 17:21:18.596035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.596004    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13ef6872-e8f8-46c0-800f-c73e2a355c6f-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cf8664c87-xdl6k\" (UID: \"13ef6872-e8f8-46c0-800f-c73e2a355c6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"
Nov 03 17:21:18.596187 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.596046    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13ef6872-e8f8-46c0-800f-c73e2a355c6f-tmp\") pod \"opendatahub-operator-controller-manager-5cf8664c87-xdl6k\" (UID: \"13ef6872-e8f8-46c0-800f-c73e2a355c6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"
Nov 03 17:21:18.596187 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.596071    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/13ef6872-e8f8-46c0-800f-c73e2a355c6f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cf8664c87-xdl6k\" (UID: \"13ef6872-e8f8-46c0-800f-c73e2a355c6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"
Nov 03 17:21:18.596187 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.596133    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/13ef6872-e8f8-46c0-800f-c73e2a355c6f-manifests\") pod \"opendatahub-operator-controller-manager-5cf8664c87-xdl6k\" (UID: \"13ef6872-e8f8-46c0-800f-c73e2a355c6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"
Nov 03 17:21:18.596187 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.596173    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/13ef6872-e8f8-46c0-800f-c73e2a355c6f-metrics-cert\") pod \"opendatahub-operator-controller-manager-5cf8664c87-xdl6k\" (UID: \"13ef6872-e8f8-46c0-800f-c73e2a355c6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"
Nov 03 17:21:18.596316 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.596253    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd4xh\" (UniqueName: \"kubernetes.io/projected/13ef6872-e8f8-46c0-800f-c73e2a355c6f-kube-api-access-bd4xh\") pod \"opendatahub-operator-controller-manager-5cf8664c87-xdl6k\" (UID: \"13ef6872-e8f8-46c0-800f-c73e2a355c6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"
Nov 03 17:21:18.697528 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.697486    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13ef6872-e8f8-46c0-800f-c73e2a355c6f-tmp\") pod \"opendatahub-operator-controller-manager-5cf8664c87-xdl6k\" (UID: \"13ef6872-e8f8-46c0-800f-c73e2a355c6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"
Nov 03 17:21:18.697713 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.697532    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/13ef6872-e8f8-46c0-800f-c73e2a355c6f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cf8664c87-xdl6k\" (UID: \"13ef6872-e8f8-46c0-800f-c73e2a355c6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"
Nov 03 17:21:18.697713 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.697562    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/13ef6872-e8f8-46c0-800f-c73e2a355c6f-manifests\") pod \"opendatahub-operator-controller-manager-5cf8664c87-xdl6k\" (UID: \"13ef6872-e8f8-46c0-800f-c73e2a355c6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"
Nov 03 17:21:18.697713 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.697593    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/13ef6872-e8f8-46c0-800f-c73e2a355c6f-metrics-cert\") pod \"opendatahub-operator-controller-manager-5cf8664c87-xdl6k\" (UID: \"13ef6872-e8f8-46c0-800f-c73e2a355c6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"
Nov 03 17:21:18.697713 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.697655    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd4xh\" (UniqueName: \"kubernetes.io/projected/13ef6872-e8f8-46c0-800f-c73e2a355c6f-kube-api-access-bd4xh\") pod \"opendatahub-operator-controller-manager-5cf8664c87-xdl6k\" (UID: \"13ef6872-e8f8-46c0-800f-c73e2a355c6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"
Nov 03 17:21:18.697933 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.697715    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13ef6872-e8f8-46c0-800f-c73e2a355c6f-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cf8664c87-xdl6k\" (UID: \"13ef6872-e8f8-46c0-800f-c73e2a355c6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"
Nov 03 17:21:18.698075 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.698040    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/13ef6872-e8f8-46c0-800f-c73e2a355c6f-manifests\") pod \"opendatahub-operator-controller-manager-5cf8664c87-xdl6k\" (UID: \"13ef6872-e8f8-46c0-800f-c73e2a355c6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"
Nov 03 17:21:18.700092 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.700070    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13ef6872-e8f8-46c0-800f-c73e2a355c6f-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cf8664c87-xdl6k\" (UID: \"13ef6872-e8f8-46c0-800f-c73e2a355c6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"
Nov 03 17:21:18.700220 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.700189    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13ef6872-e8f8-46c0-800f-c73e2a355c6f-tmp\") pod \"opendatahub-operator-controller-manager-5cf8664c87-xdl6k\" (UID: \"13ef6872-e8f8-46c0-800f-c73e2a355c6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"
Nov 03 17:21:18.700385 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.700368    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/13ef6872-e8f8-46c0-800f-c73e2a355c6f-metrics-cert\") pod \"opendatahub-operator-controller-manager-5cf8664c87-xdl6k\" (UID: \"13ef6872-e8f8-46c0-800f-c73e2a355c6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"
Nov 03 17:21:18.700448 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.700411    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/13ef6872-e8f8-46c0-800f-c73e2a355c6f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cf8664c87-xdl6k\" (UID: \"13ef6872-e8f8-46c0-800f-c73e2a355c6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"
Nov 03 17:21:18.756831 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.756755    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd4xh\" (UniqueName: \"kubernetes.io/projected/13ef6872-e8f8-46c0-800f-c73e2a355c6f-kube-api-access-bd4xh\") pod \"opendatahub-operator-controller-manager-5cf8664c87-xdl6k\" (UID: \"13ef6872-e8f8-46c0-800f-c73e2a355c6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"
Nov 03 17:21:18.775101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.775070    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:18.802282 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.802253    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"
Nov 03 17:21:18.955877 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:18.955847    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:19.090625 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:19.090528    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-98kvh"]
Nov 03 17:21:19.110087 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:19.110051    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-98kvh"
Nov 03 17:21:19.155918 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:19.155884    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-98kvh"]
Nov 03 17:21:19.165343 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:19.165225    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-b45b2\""
Nov 03 17:21:19.279707 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:19.279678    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:19.303078 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:19.302993    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6smfd\" (UniqueName: \"kubernetes.io/projected/e2be9059-ad01-4f90-9573-4e08bd2a9a48-kube-api-access-6smfd\") pod \"cert-manager-597b95cb84-98kvh\" (UID: \"e2be9059-ad01-4f90-9573-4e08bd2a9a48\") " pod="cert-manager/cert-manager-597b95cb84-98kvh"
Nov 03 17:21:19.303265 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:19.303154    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2be9059-ad01-4f90-9573-4e08bd2a9a48-bound-sa-token\") pod \"cert-manager-597b95cb84-98kvh\" (UID: \"e2be9059-ad01-4f90-9573-4e08bd2a9a48\") " pod="cert-manager/cert-manager-597b95cb84-98kvh"
Nov 03 17:21:19.317839 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:19.317804    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"]
Nov 03 17:21:19.324297 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:21:19.324266    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ef6872_e8f8_46c0_800f_c73e2a355c6f.slice/crio-fe580a202b33864ca29da186dbb4d7d84d6385f452849b3f08ebafe906e93386 WatchSource:0}: Error finding container fe580a202b33864ca29da186dbb4d7d84d6385f452849b3f08ebafe906e93386: Status 404 returned error can't find the container with id fe580a202b33864ca29da186dbb4d7d84d6385f452849b3f08ebafe906e93386
Nov 03 17:21:19.404537 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:19.404435    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6smfd\" (UniqueName: \"kubernetes.io/projected/e2be9059-ad01-4f90-9573-4e08bd2a9a48-kube-api-access-6smfd\") pod \"cert-manager-597b95cb84-98kvh\" (UID: \"e2be9059-ad01-4f90-9573-4e08bd2a9a48\") " pod="cert-manager/cert-manager-597b95cb84-98kvh"
Nov 03 17:21:19.404537 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:19.404525    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2be9059-ad01-4f90-9573-4e08bd2a9a48-bound-sa-token\") pod \"cert-manager-597b95cb84-98kvh\" (UID: \"e2be9059-ad01-4f90-9573-4e08bd2a9a48\") " pod="cert-manager/cert-manager-597b95cb84-98kvh"
Nov 03 17:21:19.467086 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:19.467052    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2be9059-ad01-4f90-9573-4e08bd2a9a48-bound-sa-token\") pod \"cert-manager-597b95cb84-98kvh\" (UID: \"e2be9059-ad01-4f90-9573-4e08bd2a9a48\") " pod="cert-manager/cert-manager-597b95cb84-98kvh"
Nov 03 17:21:19.467635 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:19.467617    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6smfd\" (UniqueName: \"kubernetes.io/projected/e2be9059-ad01-4f90-9573-4e08bd2a9a48-kube-api-access-6smfd\") pod \"cert-manager-597b95cb84-98kvh\" (UID: \"e2be9059-ad01-4f90-9573-4e08bd2a9a48\") " pod="cert-manager/cert-manager-597b95cb84-98kvh"
Nov 03 17:21:19.490728 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:19.490689    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k" event={"ID":"13ef6872-e8f8-46c0-800f-c73e2a355c6f","Type":"ContainerStarted","Data":"fe580a202b33864ca29da186dbb4d7d84d6385f452849b3f08ebafe906e93386"}
Nov 03 17:21:19.734848 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:19.734811    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-98kvh"
Nov 03 17:21:20.124214 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:20.124125    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-98kvh"]
Nov 03 17:21:20.131988 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:21:20.131954    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2be9059_ad01_4f90_9573_4e08bd2a9a48.slice/crio-5b187ed0ed8467240b6d60f676ff6fe99a571fe9f89bb2c840b2bed9a7194ec2 WatchSource:0}: Error finding container 5b187ed0ed8467240b6d60f676ff6fe99a571fe9f89bb2c840b2bed9a7194ec2: Status 404 returned error can't find the container with id 5b187ed0ed8467240b6d60f676ff6fe99a571fe9f89bb2c840b2bed9a7194ec2
Nov 03 17:21:20.361204 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:20.361169    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:20.472180 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:20.472150    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-rsqmx"
Nov 03 17:21:20.498784 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:20.498742    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-98kvh" event={"ID":"e2be9059-ad01-4f90-9573-4e08bd2a9a48","Type":"ContainerStarted","Data":"267ca6e91661421095df01bfe5e845a31abb05b95e9f6ba239531b2638ff4853"}
Nov 03 17:21:20.498784 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:20.498792    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-98kvh" event={"ID":"e2be9059-ad01-4f90-9573-4e08bd2a9a48","Type":"ContainerStarted","Data":"5b187ed0ed8467240b6d60f676ff6fe99a571fe9f89bb2c840b2bed9a7194ec2"}
Nov 03 17:21:20.703639 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:20.703580    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-98kvh" podStartSLOduration=1.703564785 podStartE2EDuration="1.703564785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:21:20.701908781 +0000 UTC m=+221.499621562" watchObservedRunningTime="2026-11-03 17:21:20.703564785 +0000 UTC m=+221.501277568"
Nov 03 17:21:21.384958 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:21.384925    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:22.360999 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:22.360956    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:22.681814 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:22.681718    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:23.516399 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:23.516339    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k" event={"ID":"13ef6872-e8f8-46c0-800f-c73e2a355c6f","Type":"ContainerStarted","Data":"a217d7e3188677352822fbe696612e6d88857bb7c329bb91f060724720f37756"}
Nov 03 17:21:24.435744 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:24.435707    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:24.522017 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:24.521975    6059 generic.go:356] "Generic (PLEG): container finished" podID="13ef6872-e8f8-46c0-800f-c73e2a355c6f" containerID="a217d7e3188677352822fbe696612e6d88857bb7c329bb91f060724720f37756" exitCode=0
Nov 03 17:21:24.522207 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:24.522055    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k" event={"ID":"13ef6872-e8f8-46c0-800f-c73e2a355c6f","Type":"ContainerDied","Data":"a217d7e3188677352822fbe696612e6d88857bb7c329bb91f060724720f37756"}
Nov 03 17:21:24.655868 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:24.655819    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:26.533463 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:26.533421    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k" event={"ID":"13ef6872-e8f8-46c0-800f-c73e2a355c6f","Type":"ContainerStarted","Data":"94bfd5a7a37aa2d0b64360b371732318a4a45730b9d7a4eaf36a591134b4f173"}
Nov 03 17:21:26.534014 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:26.533621    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"
Nov 03 17:21:26.825183 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:26.825065    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k" podStartSLOduration=3.609005747 podStartE2EDuration="8.825047522s" totalImagesPullingTime="4.786173084s" totalInitContainerRuntime="429.868691ms" isStatefulPod=false podCreationTimestamp="2026-11-03 17:21:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:21:26.823560434 +0000 UTC m=+227.621273215" watchObservedRunningTime="2026-11-03 17:21:26.825047522 +0000 UTC m=+227.622760303"
Nov 03 17:21:27.732441 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:27.732403    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:27.934210 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:27.934175    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:29.293888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:29.293849    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:29.502294 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:29.502262    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:31.277094 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:31.277062    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:31.520813 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:31.520773    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:31.912631 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:31.912598    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:32.183727 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:32.183613    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:32.726818 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:32.726783    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:35.108482 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:35.108427    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:35.484736 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:35.484697    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:35.965867 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:35.965798    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:36.862091 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:21:36.862049    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:21:37.540008 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:37.539969    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-xdl6k"
Nov 03 17:21:37.998787 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:37.998754    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:38.235145 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:38.235111    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:40.181325 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:40.181291    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-n4k4d"]
Nov 03 17:21:40.188519 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:40.188488    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n4k4d"
Nov 03 17:21:40.208723 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:40.208692    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:40.293082 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:40.293048    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 03 17:21:40.293304 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:40.293116    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 17:21:40.293304 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:40.293176    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-grtmb\""
Nov 03 17:21:40.293304 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:40.293182    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 03 17:21:40.293304 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:40.293200    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 17:21:40.294409 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:40.294386    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-n4k4d"]
Nov 03 17:21:40.296735 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:40.296700    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d8bf1633-07f3-4660-9268-cd96bfee740a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-n4k4d\" (UID: \"d8bf1633-07f3-4660-9268-cd96bfee740a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n4k4d"
Nov 03 17:21:40.296919 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:40.296762    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8bf1633-07f3-4660-9268-cd96bfee740a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-n4k4d\" (UID: \"d8bf1633-07f3-4660-9268-cd96bfee740a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n4k4d"
Nov 03 17:21:40.296919 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:40.296836    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbm5z\" (UniqueName: \"kubernetes.io/projected/d8bf1633-07f3-4660-9268-cd96bfee740a-kube-api-access-jbm5z\") pod \"kuadrant-console-plugin-66c4fb575d-n4k4d\" (UID: \"d8bf1633-07f3-4660-9268-cd96bfee740a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n4k4d"
Nov 03 17:21:40.398024 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:40.397978    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d8bf1633-07f3-4660-9268-cd96bfee740a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-n4k4d\" (UID: \"d8bf1633-07f3-4660-9268-cd96bfee740a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n4k4d"
Nov 03 17:21:40.398222 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:40.398118    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8bf1633-07f3-4660-9268-cd96bfee740a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-n4k4d\" (UID: \"d8bf1633-07f3-4660-9268-cd96bfee740a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n4k4d"
Nov 03 17:21:40.398222 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:40.398172    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbm5z\" (UniqueName: \"kubernetes.io/projected/d8bf1633-07f3-4660-9268-cd96bfee740a-kube-api-access-jbm5z\") pod \"kuadrant-console-plugin-66c4fb575d-n4k4d\" (UID: \"d8bf1633-07f3-4660-9268-cd96bfee740a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n4k4d"
Nov 03 17:21:40.398859 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:40.398807    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d8bf1633-07f3-4660-9268-cd96bfee740a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-n4k4d\" (UID: \"d8bf1633-07f3-4660-9268-cd96bfee740a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n4k4d"
Nov 03 17:21:40.400943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:40.400919    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8bf1633-07f3-4660-9268-cd96bfee740a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-n4k4d\" (UID: \"d8bf1633-07f3-4660-9268-cd96bfee740a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n4k4d"
Nov 03 17:21:40.436786 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:40.436689    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:40.463997 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:40.463964    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbm5z\" (UniqueName: \"kubernetes.io/projected/d8bf1633-07f3-4660-9268-cd96bfee740a-kube-api-access-jbm5z\") pod \"kuadrant-console-plugin-66c4fb575d-n4k4d\" (UID: \"d8bf1633-07f3-4660-9268-cd96bfee740a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n4k4d"
Nov 03 17:21:40.500940 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:40.500892    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n4k4d"
Nov 03 17:21:40.731486 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:40.731445    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:40.891314 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:40.891278    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-n4k4d"]
Nov 03 17:21:40.898274 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:21:40.898234    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8bf1633_07f3_4660_9268_cd96bfee740a.slice/crio-2ffb7032f09fe30ef2ab4741ea5e3be5bc53ec424dc984d3ea1f38e02670d88f WatchSource:0}: Error finding container 2ffb7032f09fe30ef2ab4741ea5e3be5bc53ec424dc984d3ea1f38e02670d88f: Status 404 returned error can't find the container with id 2ffb7032f09fe30ef2ab4741ea5e3be5bc53ec424dc984d3ea1f38e02670d88f
Nov 03 17:21:41.017882 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:41.017788    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:41.609573 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:41.609530    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n4k4d" event={"ID":"d8bf1633-07f3-4660-9268-cd96bfee740a","Type":"ContainerStarted","Data":"2ffb7032f09fe30ef2ab4741ea5e3be5bc53ec424dc984d3ea1f38e02670d88f"}
Nov 03 17:21:41.713989 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:41.713957    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:41.955312 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:41.955221    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:42.193716 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:42.193681    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:42.402919 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:42.402882    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:47.641043 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:47.640953    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n4k4d" event={"ID":"d8bf1633-07f3-4660-9268-cd96bfee740a","Type":"ContainerStarted","Data":"6099214a0d4eba20f165d240459374565e92dff5b2a4c613865294f0e982f964"}
Nov 03 17:21:47.780571 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:47.780515    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n4k4d" podStartSLOduration=1.466234471 podStartE2EDuration="7.780501805s" totalImagesPullingTime="6.314267334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:21:47.778572888 +0000 UTC m=+248.576285669" watchObservedRunningTime="2026-11-03 17:21:47.780501805 +0000 UTC m=+248.578214586"
Nov 03 17:21:48.621425 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:48.621388    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:48.856740 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:48.856703    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:53.272024 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:53.271986    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:53.529612 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:53.529525    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:53.766766 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:53.766729    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:53.960721 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:53.960691    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:56.114264 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:56.114224    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:56.358090 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:56.358055    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:56.620865 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:56.620832    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:58.048002 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:58.047962    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:58.374265 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:58.374191    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5fc78d544-x9ljt"]
Nov 03 17:21:58.382943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:58.382918    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5fc78d544-x9ljt"
Nov 03 17:21:58.467183 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:58.467127    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5fc78d544-x9ljt"]
Nov 03 17:21:58.469667 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:58.469635    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/079406c1-7060-4a19-b62f-0f8c25e4aea7-tls-certs\") pod \"kube-auth-proxy-5fc78d544-x9ljt\" (UID: \"079406c1-7060-4a19-b62f-0f8c25e4aea7\") " pod="openshift-ingress/kube-auth-proxy-5fc78d544-x9ljt"
Nov 03 17:21:58.469835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:58.469812    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/079406c1-7060-4a19-b62f-0f8c25e4aea7-tmp\") pod \"kube-auth-proxy-5fc78d544-x9ljt\" (UID: \"079406c1-7060-4a19-b62f-0f8c25e4aea7\") " pod="openshift-ingress/kube-auth-proxy-5fc78d544-x9ljt"
Nov 03 17:21:58.469886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:58.469855    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvzxn\" (UniqueName: \"kubernetes.io/projected/079406c1-7060-4a19-b62f-0f8c25e4aea7-kube-api-access-hvzxn\") pod \"kube-auth-proxy-5fc78d544-x9ljt\" (UID: \"079406c1-7060-4a19-b62f-0f8c25e4aea7\") " pod="openshift-ingress/kube-auth-proxy-5fc78d544-x9ljt"
Nov 03 17:21:58.511258 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:58.511224    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 17:21:58.512467 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:58.512431    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-glcnq\""
Nov 03 17:21:58.512467 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:58.512435    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 17:21:58.570732 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:58.570683    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/079406c1-7060-4a19-b62f-0f8c25e4aea7-tmp\") pod \"kube-auth-proxy-5fc78d544-x9ljt\" (UID: \"079406c1-7060-4a19-b62f-0f8c25e4aea7\") " pod="openshift-ingress/kube-auth-proxy-5fc78d544-x9ljt"
Nov 03 17:21:58.570732 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:58.570736    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvzxn\" (UniqueName: \"kubernetes.io/projected/079406c1-7060-4a19-b62f-0f8c25e4aea7-kube-api-access-hvzxn\") pod \"kube-auth-proxy-5fc78d544-x9ljt\" (UID: \"079406c1-7060-4a19-b62f-0f8c25e4aea7\") " pod="openshift-ingress/kube-auth-proxy-5fc78d544-x9ljt"
Nov 03 17:21:58.570926 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:58.570763    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/079406c1-7060-4a19-b62f-0f8c25e4aea7-tls-certs\") pod \"kube-auth-proxy-5fc78d544-x9ljt\" (UID: \"079406c1-7060-4a19-b62f-0f8c25e4aea7\") " pod="openshift-ingress/kube-auth-proxy-5fc78d544-x9ljt"
Nov 03 17:21:58.573166 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:58.573133    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/079406c1-7060-4a19-b62f-0f8c25e4aea7-tmp\") pod \"kube-auth-proxy-5fc78d544-x9ljt\" (UID: \"079406c1-7060-4a19-b62f-0f8c25e4aea7\") " pod="openshift-ingress/kube-auth-proxy-5fc78d544-x9ljt"
Nov 03 17:21:58.573352 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:58.573334    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/079406c1-7060-4a19-b62f-0f8c25e4aea7-tls-certs\") pod \"kube-auth-proxy-5fc78d544-x9ljt\" (UID: \"079406c1-7060-4a19-b62f-0f8c25e4aea7\") " pod="openshift-ingress/kube-auth-proxy-5fc78d544-x9ljt"
Nov 03 17:21:58.635851 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:58.635777    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvzxn\" (UniqueName: \"kubernetes.io/projected/079406c1-7060-4a19-b62f-0f8c25e4aea7-kube-api-access-hvzxn\") pod \"kube-auth-proxy-5fc78d544-x9ljt\" (UID: \"079406c1-7060-4a19-b62f-0f8c25e4aea7\") " pod="openshift-ingress/kube-auth-proxy-5fc78d544-x9ljt"
Nov 03 17:21:58.695649 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:58.695616    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5fc78d544-x9ljt"
Nov 03 17:21:59.009424 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.009388    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"]
Nov 03 17:21:59.014103 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.014079    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.072576 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.072544    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-2d79q\""
Nov 03 17:21:59.086045 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.086003    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"]
Nov 03 17:21:59.147557 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.147518    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5fc78d544-x9ljt"]
Nov 03 17:21:59.154762 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:21:59.154726    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079406c1_7060_4a19_b62f_0f8c25e4aea7.slice/crio-a68682f0703ae43afb7df5761578f7e3651ac8fe6f650f58958f193497c04c99 WatchSource:0}: Error finding container a68682f0703ae43afb7df5761578f7e3651ac8fe6f650f58958f193497c04c99: Status 404 returned error can't find the container with id a68682f0703ae43afb7df5761578f7e3651ac8fe6f650f58958f193497c04c99
Nov 03 17:21:59.175884 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.175846    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twnzf\" (UniqueName: \"kubernetes.io/projected/96f2949d-40c0-46ec-9cb6-235e005b0736-kube-api-access-twnzf\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.176078 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.175901    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f2949d-40c0-46ec-9cb6-235e005b0736-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.176078 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.175926    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f2949d-40c0-46ec-9cb6-235e005b0736-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.176078 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.176018    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f2949d-40c0-46ec-9cb6-235e005b0736-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.176078 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.176051    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f2949d-40c0-46ec-9cb6-235e005b0736-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.176078 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.176076    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f2949d-40c0-46ec-9cb6-235e005b0736-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.176325 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.176198    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f2949d-40c0-46ec-9cb6-235e005b0736-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.176325 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.176243    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f2949d-40c0-46ec-9cb6-235e005b0736-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.176476 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.176389    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f2949d-40c0-46ec-9cb6-235e005b0736-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.277073 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.276976    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f2949d-40c0-46ec-9cb6-235e005b0736-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.277073 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.277031    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f2949d-40c0-46ec-9cb6-235e005b0736-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.277073 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.277066    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f2949d-40c0-46ec-9cb6-235e005b0736-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.277341 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.277094    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f2949d-40c0-46ec-9cb6-235e005b0736-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.277341 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.277122    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twnzf\" (UniqueName: \"kubernetes.io/projected/96f2949d-40c0-46ec-9cb6-235e005b0736-kube-api-access-twnzf\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.277341 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.277159    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f2949d-40c0-46ec-9cb6-235e005b0736-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.277640 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.277186    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f2949d-40c0-46ec-9cb6-235e005b0736-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.277712 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.277677    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f2949d-40c0-46ec-9cb6-235e005b0736-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.277770 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.277733    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f2949d-40c0-46ec-9cb6-235e005b0736-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.284720 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.277851    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f2949d-40c0-46ec-9cb6-235e005b0736-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.284720 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.277961    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f2949d-40c0-46ec-9cb6-235e005b0736-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.284720 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.278026    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f2949d-40c0-46ec-9cb6-235e005b0736-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.284720 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.278222    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f2949d-40c0-46ec-9cb6-235e005b0736-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.284720 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.278570    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f2949d-40c0-46ec-9cb6-235e005b0736-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.284720 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.280230    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f2949d-40c0-46ec-9cb6-235e005b0736-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.286687 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.286660    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f2949d-40c0-46ec-9cb6-235e005b0736-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.298422 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.298382    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:59.348177 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.348142    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f2949d-40c0-46ec-9cb6-235e005b0736-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.348470 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.348449    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twnzf\" (UniqueName: \"kubernetes.io/projected/96f2949d-40c0-46ec-9cb6-235e005b0736-kube-api-access-twnzf\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp\" (UID: \"96f2949d-40c0-46ec-9cb6-235e005b0736\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.502294 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.502250    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:21:59.626729 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.626630    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:21:59.698403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:21:59.698338    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5fc78d544-x9ljt" event={"ID":"079406c1-7060-4a19-b62f-0f8c25e4aea7","Type":"ContainerStarted","Data":"a68682f0703ae43afb7df5761578f7e3651ac8fe6f650f58958f193497c04c99"}
Nov 03 17:22:00.110637 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:00.110591    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"]
Nov 03 17:22:00.150230 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:22:00.150199    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f2949d_40c0_46ec_9cb6_235e005b0736.slice/crio-22bd5f47d33ffcd836a1ed3058add887f33aee0b5e8f464a1467760e37fbb909 WatchSource:0}: Error finding container 22bd5f47d33ffcd836a1ed3058add887f33aee0b5e8f464a1467760e37fbb909: Status 404 returned error can't find the container with id 22bd5f47d33ffcd836a1ed3058add887f33aee0b5e8f464a1467760e37fbb909
Nov 03 17:22:00.152525 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:00.152487    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:22:00.152632 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:00.152562    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:22:00.152632 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:00.152590    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:22:00.567261 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:00.567227    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:00.708668 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:00.707923    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp" event={"ID":"96f2949d-40c0-46ec-9cb6-235e005b0736","Type":"ContainerStarted","Data":"c20feef650ab5206675370a0d0657a0e4b7bd8f570539d37dbba318be52b6a96"}
Nov 03 17:22:00.708668 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:00.707996    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp" event={"ID":"96f2949d-40c0-46ec-9cb6-235e005b0736","Type":"ContainerStarted","Data":"22bd5f47d33ffcd836a1ed3058add887f33aee0b5e8f464a1467760e37fbb909"}
Nov 03 17:22:00.915249 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:00.915111    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp" podStartSLOduration=2.915090099 podStartE2EDuration="2.915090099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:22:00.913815161 +0000 UTC m=+261.711527941" watchObservedRunningTime="2026-11-03 17:22:00.915090099 +0000 UTC m=+261.712803082"
Nov 03 17:22:01.627370 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:01.627316    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:22:01.633157 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:01.633125    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:22:01.713745 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:01.713656    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:22:01.714573 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:01.714547    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp"
Nov 03 17:22:02.006209 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.006154    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7968b79986-hqntz"]
Nov 03 17:22:02.010650 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.010622    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7968b79986-hqntz"
Nov 03 17:22:02.081812 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.081775    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:22:02.082017 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.081975    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-nf6dd\""
Nov 03 17:22:02.082097 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.082071    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 03 17:22:02.082097 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.082080    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 17:22:02.082516 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.082214    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 03 17:22:02.083633 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.083612    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 03 17:22:02.084271 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.084249    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7968b79986-hqntz"]
Nov 03 17:22:02.106663 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.106632    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a706eef4-927c-48a8-bcf0-600f23c4126c-metrics-certs\") pod \"jobset-controller-manager-7968b79986-hqntz\" (UID: \"a706eef4-927c-48a8-bcf0-600f23c4126c\") " pod="openshift-jobset-operator/jobset-controller-manager-7968b79986-hqntz"
Nov 03 17:22:02.106859 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.106684    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a706eef4-927c-48a8-bcf0-600f23c4126c-cert\") pod \"jobset-controller-manager-7968b79986-hqntz\" (UID: \"a706eef4-927c-48a8-bcf0-600f23c4126c\") " pod="openshift-jobset-operator/jobset-controller-manager-7968b79986-hqntz"
Nov 03 17:22:02.106859 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.106826    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a706eef4-927c-48a8-bcf0-600f23c4126c-manager-config\") pod \"jobset-controller-manager-7968b79986-hqntz\" (UID: \"a706eef4-927c-48a8-bcf0-600f23c4126c\") " pod="openshift-jobset-operator/jobset-controller-manager-7968b79986-hqntz"
Nov 03 17:22:02.106987 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.106917    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jftjn\" (UniqueName: \"kubernetes.io/projected/a706eef4-927c-48a8-bcf0-600f23c4126c-kube-api-access-jftjn\") pod \"jobset-controller-manager-7968b79986-hqntz\" (UID: \"a706eef4-927c-48a8-bcf0-600f23c4126c\") " pod="openshift-jobset-operator/jobset-controller-manager-7968b79986-hqntz"
Nov 03 17:22:02.208126 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.208087    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a706eef4-927c-48a8-bcf0-600f23c4126c-cert\") pod \"jobset-controller-manager-7968b79986-hqntz\" (UID: \"a706eef4-927c-48a8-bcf0-600f23c4126c\") " pod="openshift-jobset-operator/jobset-controller-manager-7968b79986-hqntz"
Nov 03 17:22:02.208271 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.208158    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a706eef4-927c-48a8-bcf0-600f23c4126c-manager-config\") pod \"jobset-controller-manager-7968b79986-hqntz\" (UID: \"a706eef4-927c-48a8-bcf0-600f23c4126c\") " pod="openshift-jobset-operator/jobset-controller-manager-7968b79986-hqntz"
Nov 03 17:22:02.208271 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.208241    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jftjn\" (UniqueName: \"kubernetes.io/projected/a706eef4-927c-48a8-bcf0-600f23c4126c-kube-api-access-jftjn\") pod \"jobset-controller-manager-7968b79986-hqntz\" (UID: \"a706eef4-927c-48a8-bcf0-600f23c4126c\") " pod="openshift-jobset-operator/jobset-controller-manager-7968b79986-hqntz"
Nov 03 17:22:02.208399 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.208298    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a706eef4-927c-48a8-bcf0-600f23c4126c-metrics-certs\") pod \"jobset-controller-manager-7968b79986-hqntz\" (UID: \"a706eef4-927c-48a8-bcf0-600f23c4126c\") " pod="openshift-jobset-operator/jobset-controller-manager-7968b79986-hqntz"
Nov 03 17:22:02.209007 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.208983    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a706eef4-927c-48a8-bcf0-600f23c4126c-manager-config\") pod \"jobset-controller-manager-7968b79986-hqntz\" (UID: \"a706eef4-927c-48a8-bcf0-600f23c4126c\") " pod="openshift-jobset-operator/jobset-controller-manager-7968b79986-hqntz"
Nov 03 17:22:02.210890 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.210822    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a706eef4-927c-48a8-bcf0-600f23c4126c-metrics-certs\") pod \"jobset-controller-manager-7968b79986-hqntz\" (UID: \"a706eef4-927c-48a8-bcf0-600f23c4126c\") " pod="openshift-jobset-operator/jobset-controller-manager-7968b79986-hqntz"
Nov 03 17:22:02.210995 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.210855    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a706eef4-927c-48a8-bcf0-600f23c4126c-cert\") pod \"jobset-controller-manager-7968b79986-hqntz\" (UID: \"a706eef4-927c-48a8-bcf0-600f23c4126c\") " pod="openshift-jobset-operator/jobset-controller-manager-7968b79986-hqntz"
Nov 03 17:22:02.277163 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.277089    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jftjn\" (UniqueName: \"kubernetes.io/projected/a706eef4-927c-48a8-bcf0-600f23c4126c-kube-api-access-jftjn\") pod \"jobset-controller-manager-7968b79986-hqntz\" (UID: \"a706eef4-927c-48a8-bcf0-600f23c4126c\") " pod="openshift-jobset-operator/jobset-controller-manager-7968b79986-hqntz"
Nov 03 17:22:02.327619 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.327583    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7968b79986-hqntz"
Nov 03 17:22:02.704877 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.704781    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7968b79986-hqntz"]
Nov 03 17:22:02.711583 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:22:02.711549    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda706eef4_927c_48a8_bcf0_600f23c4126c.slice/crio-5af48f7564ad68dc3a17dabac35d21ff3e9650c2502a1abd08e3d7c05c5dac81 WatchSource:0}: Error finding container 5af48f7564ad68dc3a17dabac35d21ff3e9650c2502a1abd08e3d7c05c5dac81: Status 404 returned error can't find the container with id 5af48f7564ad68dc3a17dabac35d21ff3e9650c2502a1abd08e3d7c05c5dac81
Nov 03 17:22:02.719042 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.719004    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5fc78d544-x9ljt" event={"ID":"079406c1-7060-4a19-b62f-0f8c25e4aea7","Type":"ContainerStarted","Data":"d1de5d4cbea471b0611bfd4853254eed99a0337b3eabecc74ca9c3dc6ea9d652"}
Nov 03 17:22:02.720218 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.720192    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7968b79986-hqntz" event={"ID":"a706eef4-927c-48a8-bcf0-600f23c4126c","Type":"ContainerStarted","Data":"5af48f7564ad68dc3a17dabac35d21ff3e9650c2502a1abd08e3d7c05c5dac81"}
Nov 03 17:22:02.832334 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:02.832273    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5fc78d544-x9ljt" podStartSLOduration=1.8143289820000001 podStartE2EDuration="4.832257038s" totalImagesPullingTime="3.017928056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:21:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:22:02.831187406 +0000 UTC m=+263.628900187" watchObservedRunningTime="2026-11-03 17:22:02.832257038 +0000 UTC m=+263.629969838"
Nov 03 17:22:03.277502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:03.277469    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:03.491625 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:03.491589    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:06.685301 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:06.685270    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:06.868085 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:22:06.868042    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:22:07.748215 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:07.748178    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7968b79986-hqntz" event={"ID":"a706eef4-927c-48a8-bcf0-600f23c4126c","Type":"ContainerStarted","Data":"29a7223e9784fad22e2dda080e135da2b42ec17f2f63585c305d616a0dcdf3cc"}
Nov 03 17:22:07.748645 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:07.748342    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-7968b79986-hqntz"
Nov 03 17:22:07.866734 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:07.866677    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-7968b79986-hqntz" podStartSLOduration=2.179708976 podStartE2EDuration="6.866660918s" totalImagesPullingTime="4.686951942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:22:07.864842494 +0000 UTC m=+268.662555287" watchObservedRunningTime="2026-11-03 17:22:07.866660918 +0000 UTC m=+268.664373699"
Nov 03 17:22:09.781686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:09.781653    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:09.969039 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:09.969003    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:14.540237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:14.540201    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:14.785144 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:14.785108    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:18.758784 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:18.758710    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-7968b79986-hqntz"
Nov 03 17:22:19.383837 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:19.383798    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:19.709747 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:19.709712    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:22.068414 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:22.068369    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:22.318576 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:22.318486    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:23.422604 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:23.422569    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:23.644151 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:23.644117    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:24.542460 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:24.542419    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:24.762030 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:24.761996    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:24.993231 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:24.993194    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:25.236813 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:25.236777    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:31.252236 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:31.252200    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:31.447063 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:31.447030    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:31.643922 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:31.643807    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:36.003832 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:36.003796    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:36.242470 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:36.242393    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:36.861286 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:22:36.861244    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:22:39.354599 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:39.354562    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 17:22:39.357556 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:39.357530    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 17:22:39.371940 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:39.371912    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:22:39.375019 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:39.374995    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:22:39.573644 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:39.573610    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:39.790859 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:39.790821    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:40.308539 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:40.308507    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:40.760750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:40.760700    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:22:40.940122 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:22:40.940086    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:04.518050 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:04.518012    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:04.966920 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:04.966889    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:05.261488 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:05.261378    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:06.859620 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:23:06.859582    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:23:12.772021 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:12.771958    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:12.962576 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:12.962525    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:16.972818 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:16.972784    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:17.198599 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:17.198564    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:18.877858 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:18.877824    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:19.063663 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:19.063632    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:19.257132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:19.257096    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:19.447342 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:19.447298    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:19.667731 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:19.667650    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:19.854679 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:19.854646    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:20.067133 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:20.067104    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:21.252493 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:21.252454    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:26.563913 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:26.563877    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:26.749281 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:26.749250    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:26.948617 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:26.948538    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:27.125117 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:27.125083    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:27.319554 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:27.319500    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:27.498327 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:27.498291    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:27.716157 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:27.716126    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:27.906163 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:27.906132    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:31.983900 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:31.983864    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:36.860985 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:23:36.860940    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:23:46.392497 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:46.392459    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:46.658188 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:46.658089    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:48.491416 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:48.491378    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:48.674328 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:48.674282    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:48.858060 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:48.857984    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:23:49.047778 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:23:49.047748    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:24:03.759210 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:03.759169    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:24:04.021222 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:04.021141    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:24:04.304130 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:04.304034    6059 generic.go:356] "Generic (PLEG): container finished" podID="c097188b-4b64-4636-822b-ab2c8817dfa0" containerID="c982d01afb4ddc77ddff2d184fbae564c62dada763de0a7204ca4dc30b39c590" exitCode=0
Nov 03 17:24:04.304130 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:04.304105    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d" event={"ID":"c097188b-4b64-4636-822b-ab2c8817dfa0","Type":"ContainerDied","Data":"c982d01afb4ddc77ddff2d184fbae564c62dada763de0a7204ca4dc30b39c590"}
Nov 03 17:24:04.304521 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:04.304506    6059 scope.go:122] "RemoveContainer" containerID="c982d01afb4ddc77ddff2d184fbae564c62dada763de0a7204ca4dc30b39c590"
Nov 03 17:24:04.305700 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:04.305675    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:24:04.305768 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:04.305740    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:24:04.305804 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:04.305771    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:24:05.310870 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:05.310833    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d" event={"ID":"c097188b-4b64-4636-822b-ab2c8817dfa0","Type":"ContainerStarted","Data":"58f02de98c6f04c8694d8c7f340a484d92974d0e41245245afbb5c44e810e122"}
Nov 03 17:24:05.818705 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:05.818668    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:24:05.819838 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:05.819808    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-x276d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Nov 03 17:24:05.819954 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:05.819870    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d" podUID="c097188b-4b64-4636-822b-ab2c8817dfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Nov 03 17:24:06.818345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:06.818308    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-x276d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Nov 03 17:24:06.818756 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:06.818387    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d" podUID="c097188b-4b64-4636-822b-ab2c8817dfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Nov 03 17:24:06.881776 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:24:06.881736    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:24:07.220809 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:07.220775    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:24:07.818337 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:07.818304    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-x276d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Nov 03 17:24:07.818539 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:07.818384    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d" podUID="c097188b-4b64-4636-822b-ab2c8817dfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Nov 03 17:24:08.818638 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:08.818603    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-x276d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Nov 03 17:24:08.819051 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:08.818669    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d" podUID="c097188b-4b64-4636-822b-ab2c8817dfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Nov 03 17:24:09.818058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:09.818018    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-x276d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Nov 03 17:24:09.818205 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:09.818083    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d" podUID="c097188b-4b64-4636-822b-ab2c8817dfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Nov 03 17:24:10.822146 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:10.822115    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:24:10.823137 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:10.823113    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:24:11.399635 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:11.399605    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:24:32.340734 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:32.340696    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:24:36.860196 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:24:36.860153    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:24:49.469790 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:49.469750    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:24:49.664083 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:24:49.664049    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:25:06.859757 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:25:06.859722    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:25:07.729300 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:07.729260    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-4vjv9"]
Nov 03 17:25:07.735014 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:07.734989    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-4vjv9"
Nov 03 17:25:07.756814 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:07.756787    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:25:07.793486 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:07.793464    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-4vjv9"]
Nov 03 17:25:07.814701 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:07.814676    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsmc7\" (UniqueName: \"kubernetes.io/projected/a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c-kube-api-access-tsmc7\") pod \"ai-gateway-controller-5d8dddb676-4vjv9\" (UID: \"a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-4vjv9"
Nov 03 17:25:07.814820 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:07.814801    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-4vjv9\" (UID: \"a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-4vjv9"
Nov 03 17:25:07.843241 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:07.843215    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 03 17:25:07.843350 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:07.843257    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 17:25:07.843350 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:07.843277    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-nx74j\""
Nov 03 17:25:07.843350 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:07.843306    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 17:25:07.843533 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:07.843430    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 03 17:25:07.915981 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:07.915945    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-4vjv9\" (UID: \"a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-4vjv9"
Nov 03 17:25:07.916386 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:07.916005    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsmc7\" (UniqueName: \"kubernetes.io/projected/a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c-kube-api-access-tsmc7\") pod \"ai-gateway-controller-5d8dddb676-4vjv9\" (UID: \"a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-4vjv9"
Nov 03 17:25:07.918518 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:07.918498    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-4vjv9\" (UID: \"a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-4vjv9"
Nov 03 17:25:07.944220 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:07.944192    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:25:07.980147 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:07.980087    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsmc7\" (UniqueName: \"kubernetes.io/projected/a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c-kube-api-access-tsmc7\") pod \"ai-gateway-controller-5d8dddb676-4vjv9\" (UID: \"a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-4vjv9"
Nov 03 17:25:08.047864 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:08.047840    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-4vjv9"
Nov 03 17:25:08.421884 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:08.421855    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-4vjv9"]
Nov 03 17:25:08.628696 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:08.628661    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-4vjv9" event={"ID":"a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c","Type":"ContainerStarted","Data":"7cf5d7c6d4f34c64f563ddb20bfd510feb39aebfd939fdf58768b3671055a88e"}
Nov 03 17:25:14.672906 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:14.672817    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-4vjv9" event={"ID":"a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c","Type":"ContainerStarted","Data":"ad10d7bf3ddbcd878326c3182642dea611512fe6ec6f57bf242a672388a8296d"}
Nov 03 17:25:14.673245 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:14.672918    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-4vjv9"
Nov 03 17:25:14.793593 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:14.793550    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-4vjv9" podStartSLOduration=1.913344406 podStartE2EDuration="7.793538922s" totalImagesPullingTime="5.880194516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:25:14.792452022 +0000 UTC m=+455.590164806" watchObservedRunningTime="2026-11-03 17:25:14.793538922 +0000 UTC m=+455.591251702"
Nov 03 17:25:17.144624 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:17.144563    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:25:17.361306 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:17.361219    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:25:17.561153 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:17.561118    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7"]
Nov 03 17:25:17.564734 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:17.564716    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7"
Nov 03 17:25:17.670717 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:17.670687    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9x8h5\""
Nov 03 17:25:17.670873 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:17.670735    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 17:25:17.672209 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:17.672186    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7"]
Nov 03 17:25:17.688008 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:17.687979    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-pm2wr"]
Nov 03 17:25:17.691583 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:17.691566    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-pm2wr"
Nov 03 17:25:17.718954 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:17.718926    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41eba073-3e2a-4b3e-acba-d68aad192e43-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-n4xz7\" (UID: \"41eba073-3e2a-4b3e-acba-d68aad192e43\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7"
Nov 03 17:25:17.719082 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:17.718965    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbcnx\" (UniqueName: \"kubernetes.io/projected/41eba073-3e2a-4b3e-acba-d68aad192e43-kube-api-access-qbcnx\") pod \"payload-pre-processing-7b98bb678c-n4xz7\" (UID: \"41eba073-3e2a-4b3e-acba-d68aad192e43\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7"
Nov 03 17:25:17.751297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:17.751273    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-pm2wr"]
Nov 03 17:25:17.820492 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:17.820391    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41eba073-3e2a-4b3e-acba-d68aad192e43-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-n4xz7\" (UID: \"41eba073-3e2a-4b3e-acba-d68aad192e43\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7"
Nov 03 17:25:17.820492 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:17.820440    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbcnx\" (UniqueName: \"kubernetes.io/projected/41eba073-3e2a-4b3e-acba-d68aad192e43-kube-api-access-qbcnx\") pod \"payload-pre-processing-7b98bb678c-n4xz7\" (UID: \"41eba073-3e2a-4b3e-acba-d68aad192e43\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7"
Nov 03 17:25:17.820492 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:17.820465    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqshg\" (UniqueName: \"kubernetes.io/projected/927401f9-153a-476e-8b41-7364a03d3cc8-kube-api-access-tqshg\") pod \"payload-processing-55fbbbf9dc-pm2wr\" (UID: \"927401f9-153a-476e-8b41-7364a03d3cc8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pm2wr"
Nov 03 17:25:17.820492 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:17.820486    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/927401f9-153a-476e-8b41-7364a03d3cc8-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-pm2wr\" (UID: \"927401f9-153a-476e-8b41-7364a03d3cc8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pm2wr"
Nov 03 17:25:17.821067 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:17.821044    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41eba073-3e2a-4b3e-acba-d68aad192e43-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-n4xz7\" (UID: \"41eba073-3e2a-4b3e-acba-d68aad192e43\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7"
Nov 03 17:25:17.884596 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:17.884555    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbcnx\" (UniqueName: \"kubernetes.io/projected/41eba073-3e2a-4b3e-acba-d68aad192e43-kube-api-access-qbcnx\") pod \"payload-pre-processing-7b98bb678c-n4xz7\" (UID: \"41eba073-3e2a-4b3e-acba-d68aad192e43\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7"
Nov 03 17:25:17.921512 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:17.921484    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqshg\" (UniqueName: \"kubernetes.io/projected/927401f9-153a-476e-8b41-7364a03d3cc8-kube-api-access-tqshg\") pod \"payload-processing-55fbbbf9dc-pm2wr\" (UID: \"927401f9-153a-476e-8b41-7364a03d3cc8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pm2wr"
Nov 03 17:25:17.921649 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:17.921519    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/927401f9-153a-476e-8b41-7364a03d3cc8-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-pm2wr\" (UID: \"927401f9-153a-476e-8b41-7364a03d3cc8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pm2wr"
Nov 03 17:25:17.922244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:17.922222    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/927401f9-153a-476e-8b41-7364a03d3cc8-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-pm2wr\" (UID: \"927401f9-153a-476e-8b41-7364a03d3cc8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pm2wr"
Nov 03 17:25:17.984970 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:17.984944    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqshg\" (UniqueName: \"kubernetes.io/projected/927401f9-153a-476e-8b41-7364a03d3cc8-kube-api-access-tqshg\") pod \"payload-processing-55fbbbf9dc-pm2wr\" (UID: \"927401f9-153a-476e-8b41-7364a03d3cc8\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pm2wr"
Nov 03 17:25:18.005398 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:18.005372    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-pm2wr"
Nov 03 17:25:18.177601 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:18.177517    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7"
Nov 03 17:25:18.375009 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:18.374969    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-pm2wr"]
Nov 03 17:25:18.549255 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:25:18.549221    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41eba073_3e2a_4b3e_acba_d68aad192e43.slice/crio-6ca4a2b39ac1a0361e93c0ca374a0be0bceed779783f95188418d90f1b4ef18d WatchSource:0}: Error finding container 6ca4a2b39ac1a0361e93c0ca374a0be0bceed779783f95188418d90f1b4ef18d: Status 404 returned error can't find the container with id 6ca4a2b39ac1a0361e93c0ca374a0be0bceed779783f95188418d90f1b4ef18d
Nov 03 17:25:18.551147 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:18.551126    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7"]
Nov 03 17:25:18.692536 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:18.692499    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7" event={"ID":"41eba073-3e2a-4b3e-acba-d68aad192e43","Type":"ContainerStarted","Data":"6ca4a2b39ac1a0361e93c0ca374a0be0bceed779783f95188418d90f1b4ef18d"}
Nov 03 17:25:18.693644 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:18.693622    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-pm2wr" event={"ID":"927401f9-153a-476e-8b41-7364a03d3cc8","Type":"ContainerStarted","Data":"dd66e90c2435dc75873147c3b7f74812722054be31e088a20ec59db039507149"}
Nov 03 17:25:21.712507 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:21.712476    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7" event={"ID":"41eba073-3e2a-4b3e-acba-d68aad192e43","Type":"ContainerStarted","Data":"11a873048266de01db216e89377defabe2bbb57e9ed3793756cdc1a953741184"}
Nov 03 17:25:21.712817 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:21.712585    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7"
Nov 03 17:25:21.834745 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:21.834692    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7" podStartSLOduration=1.905881408 podStartE2EDuration="4.83468068s" totalImagesPullingTime="2.928799272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:25:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:25:21.833243439 +0000 UTC m=+462.630956221" watchObservedRunningTime="2026-11-03 17:25:21.83468068 +0000 UTC m=+462.632393461"
Nov 03 17:25:22.719334 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:22.719294    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-pm2wr" event={"ID":"927401f9-153a-476e-8b41-7364a03d3cc8","Type":"ContainerStarted","Data":"9e752319effd8d398968c4bff83a645f8b203027373e5f81b89f9eb6dcc2bd19"}
Nov 03 17:25:22.719774 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:22.719463    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-pm2wr"
Nov 03 17:25:22.841089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:22.841029    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-pm2wr" podStartSLOduration=2.527650983 podStartE2EDuration="5.841015254s" totalImagesPullingTime="3.313364271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:25:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:25:22.839188408 +0000 UTC m=+463.636901191" watchObservedRunningTime="2026-11-03 17:25:22.841015254 +0000 UTC m=+463.638728057"
Nov 03 17:25:25.679571 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:25.679539    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-4vjv9"
Nov 03 17:25:26.091255 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:26.091222    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5fdd44f589-9c6qf"]
Nov 03 17:25:26.094889 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:26.094864    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5fdd44f589-9c6qf"
Nov 03 17:25:26.155471 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:26.155442    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5fdd44f589-9c6qf"]
Nov 03 17:25:26.195754 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:26.195717    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9f4c7ae-adf2-4c9a-a9be-a8b5829b69d2-metrics-certs\") pod \"ai-gateway-controller-5fdd44f589-9c6qf\" (UID: \"e9f4c7ae-adf2-4c9a-a9be-a8b5829b69d2\") " pod="opendatahub/ai-gateway-controller-5fdd44f589-9c6qf"
Nov 03 17:25:26.195915 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:26.195787    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpdwz\" (UniqueName: \"kubernetes.io/projected/e9f4c7ae-adf2-4c9a-a9be-a8b5829b69d2-kube-api-access-tpdwz\") pod \"ai-gateway-controller-5fdd44f589-9c6qf\" (UID: \"e9f4c7ae-adf2-4c9a-a9be-a8b5829b69d2\") " pod="opendatahub/ai-gateway-controller-5fdd44f589-9c6qf"
Nov 03 17:25:26.296590 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:26.296560    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9f4c7ae-adf2-4c9a-a9be-a8b5829b69d2-metrics-certs\") pod \"ai-gateway-controller-5fdd44f589-9c6qf\" (UID: \"e9f4c7ae-adf2-4c9a-a9be-a8b5829b69d2\") " pod="opendatahub/ai-gateway-controller-5fdd44f589-9c6qf"
Nov 03 17:25:26.296729 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:26.296615    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpdwz\" (UniqueName: \"kubernetes.io/projected/e9f4c7ae-adf2-4c9a-a9be-a8b5829b69d2-kube-api-access-tpdwz\") pod \"ai-gateway-controller-5fdd44f589-9c6qf\" (UID: \"e9f4c7ae-adf2-4c9a-a9be-a8b5829b69d2\") " pod="opendatahub/ai-gateway-controller-5fdd44f589-9c6qf"
Nov 03 17:25:26.299104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:26.299069    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9f4c7ae-adf2-4c9a-a9be-a8b5829b69d2-metrics-certs\") pod \"ai-gateway-controller-5fdd44f589-9c6qf\" (UID: \"e9f4c7ae-adf2-4c9a-a9be-a8b5829b69d2\") " pod="opendatahub/ai-gateway-controller-5fdd44f589-9c6qf"
Nov 03 17:25:26.360114 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:26.360044    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpdwz\" (UniqueName: \"kubernetes.io/projected/e9f4c7ae-adf2-4c9a-a9be-a8b5829b69d2-kube-api-access-tpdwz\") pod \"ai-gateway-controller-5fdd44f589-9c6qf\" (UID: \"e9f4c7ae-adf2-4c9a-a9be-a8b5829b69d2\") " pod="opendatahub/ai-gateway-controller-5fdd44f589-9c6qf"
Nov 03 17:25:26.408032 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:26.407997    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5fdd44f589-9c6qf"
Nov 03 17:25:26.781566 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:25:26.781531    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f4c7ae_adf2_4c9a_a9be_a8b5829b69d2.slice/crio-03317163bbd682cbfe91777613d44425965faac572cf306f291d94e2e8d50faa WatchSource:0}: Error finding container 03317163bbd682cbfe91777613d44425965faac572cf306f291d94e2e8d50faa: Status 404 returned error can't find the container with id 03317163bbd682cbfe91777613d44425965faac572cf306f291d94e2e8d50faa
Nov 03 17:25:26.783140 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:26.783116    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5fdd44f589-9c6qf"]
Nov 03 17:25:27.744110 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:27.744083    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5fdd44f589-9c6qf" event={"ID":"e9f4c7ae-adf2-4c9a-a9be-a8b5829b69d2","Type":"ContainerStarted","Data":"03317163bbd682cbfe91777613d44425965faac572cf306f291d94e2e8d50faa"}
Nov 03 17:25:28.750464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:28.750419    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5fdd44f589-9c6qf" event={"ID":"e9f4c7ae-adf2-4c9a-a9be-a8b5829b69d2","Type":"ContainerStarted","Data":"59a3978a1e7d94d0d64c488b85a576783a6491c8584efb9658b996313558a0a4"}
Nov 03 17:25:28.750883 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:28.750497    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5fdd44f589-9c6qf"
Nov 03 17:25:28.872760 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:28.872703    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5fdd44f589-9c6qf" podStartSLOduration=1.960012627 podStartE2EDuration="2.872690514s" totalImagesPullingTime="912.677887ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:25:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:25:28.870972158 +0000 UTC m=+469.668684938" watchObservedRunningTime="2026-11-03 17:25:28.872690514 +0000 UTC m=+469.670403317"
Nov 03 17:25:32.698613 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:32.698575    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:25:32.723159 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:32.723135    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7"
Nov 03 17:25:33.727521 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:33.727482    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-pm2wr"
Nov 03 17:25:36.860605 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:25:36.860567    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:25:39.758235 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:39.758204    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5fdd44f589-9c6qf"
Nov 03 17:25:39.932260 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:39.932230    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-4vjv9"]
Nov 03 17:25:39.932504 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:39.932462    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-4vjv9" podUID="a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c" containerName="manager" containerID="cri-o://ad10d7bf3ddbcd878326c3182642dea611512fe6ec6f57bf242a672388a8296d" gracePeriod=30
Nov 03 17:25:40.182438 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:40.182412    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-4vjv9"
Nov 03 17:25:40.220080 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:40.220055    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c-metrics-certs\") pod \"a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c\" (UID: \"a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c\") "
Nov 03 17:25:40.220191 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:40.220151    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c-kube-api-access-tsmc7\" (UniqueName: \"kubernetes.io/projected/a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c-kube-api-access-tsmc7\") pod \"a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c\" (UID: \"a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c\") "
Nov 03 17:25:40.222417 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:40.222392    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c-metrics-certs" pod "a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c" (UID: "a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:25:40.222532 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:40.222513    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c-kube-api-access-tsmc7" pod "a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c" (UID: "a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c"). InnerVolumeSpecName "kube-api-access-tsmc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:25:40.321117 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:40.321044    6059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c-metrics-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:25:40.321117 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:40.321068    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsmc7\" (UniqueName: \"kubernetes.io/projected/a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c-kube-api-access-tsmc7\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:25:40.808243 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:40.808204    6059 generic.go:356] "Generic (PLEG): container finished" podID="a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c" containerID="ad10d7bf3ddbcd878326c3182642dea611512fe6ec6f57bf242a672388a8296d" exitCode=0
Nov 03 17:25:40.808684 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:40.808274    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-4vjv9"
Nov 03 17:25:40.808684 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:40.808285    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-4vjv9" event={"ID":"a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c","Type":"ContainerDied","Data":"ad10d7bf3ddbcd878326c3182642dea611512fe6ec6f57bf242a672388a8296d"}
Nov 03 17:25:40.808684 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:40.808373    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-4vjv9" event={"ID":"a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c","Type":"ContainerDied","Data":"7cf5d7c6d4f34c64f563ddb20bfd510feb39aebfd939fdf58768b3671055a88e"}
Nov 03 17:25:40.808684 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:40.808401    6059 scope.go:122] "RemoveContainer" containerID="ad10d7bf3ddbcd878326c3182642dea611512fe6ec6f57bf242a672388a8296d"
Nov 03 17:25:40.817308 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:40.817292    6059 scope.go:122] "RemoveContainer" containerID="ad10d7bf3ddbcd878326c3182642dea611512fe6ec6f57bf242a672388a8296d"
Nov 03 17:25:40.817603 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:25:40.817585    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad10d7bf3ddbcd878326c3182642dea611512fe6ec6f57bf242a672388a8296d\": container with ID starting with ad10d7bf3ddbcd878326c3182642dea611512fe6ec6f57bf242a672388a8296d not found: ID does not exist" containerID="ad10d7bf3ddbcd878326c3182642dea611512fe6ec6f57bf242a672388a8296d"
Nov 03 17:25:40.817680 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:40.817617    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad10d7bf3ddbcd878326c3182642dea611512fe6ec6f57bf242a672388a8296d"} err="failed to get container status \"ad10d7bf3ddbcd878326c3182642dea611512fe6ec6f57bf242a672388a8296d\": rpc error: code = NotFound desc = could not find container \"ad10d7bf3ddbcd878326c3182642dea611512fe6ec6f57bf242a672388a8296d\": container with ID starting with ad10d7bf3ddbcd878326c3182642dea611512fe6ec6f57bf242a672388a8296d not found: ID does not exist"
Nov 03 17:25:40.985039 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:40.985008    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-4vjv9"]
Nov 03 17:25:40.989342 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:40.989309    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-4vjv9"]
Nov 03 17:25:41.333563 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:41.333532    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c" path="/var/lib/kubelet/pods/a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c/volumes"
Nov 03 17:25:41.680898 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:41.680817    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:25:42.194967 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:42.194934    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:25:43.505938 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:43.505901    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:25:43.685336 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:43.685302    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:25:50.352981 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:50.352946    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:25:55.911505 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:55.911469    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:25:59.372068 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:25:59.372006    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:26:01.369967 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:01.369932    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:26:01.559853 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:01.559825    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:26:02.878649 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:02.878612    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"]
Nov 03 17:26:02.879032 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:02.879011    6059 cpu_manager.go:414] "containerMap: removing container" podUID="a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c" containerName="manager"
Nov 03 17:26:02.879032 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:02.879023    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c" containerName="manager"
Nov 03 17:26:02.879168 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:02.879148    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b3c6d8-0336-4c4d-9ebf-b1bd4e21608c" containerName="manager"
Nov 03 17:26:02.882624 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:02.882603    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:02.988652 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:02.988620    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-chc7v\""
Nov 03 17:26:02.988820 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:02.988761    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 17:26:02.988820 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:02.988770    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 03 17:26:02.988820 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:02.988762    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 17:26:02.990471 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:02.990452    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"]
Nov 03 17:26:02.998275 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:02.998256    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e687910a-3808-430e-a834-08c248fe5855-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zkrg5\" (UID: \"e687910a-3808-430e-a834-08c248fe5855\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:02.998391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:02.998330    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e687910a-3808-430e-a834-08c248fe5855-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zkrg5\" (UID: \"e687910a-3808-430e-a834-08c248fe5855\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:02.998435 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:02.998393    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbzjj\" (UniqueName: \"kubernetes.io/projected/e687910a-3808-430e-a834-08c248fe5855-kube-api-access-gbzjj\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zkrg5\" (UID: \"e687910a-3808-430e-a834-08c248fe5855\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:02.998503 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:02.998487    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e687910a-3808-430e-a834-08c248fe5855-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zkrg5\" (UID: \"e687910a-3808-430e-a834-08c248fe5855\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:02.998538 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:02.998524    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e687910a-3808-430e-a834-08c248fe5855-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zkrg5\" (UID: \"e687910a-3808-430e-a834-08c248fe5855\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:02.998611 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:02.998596    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e687910a-3808-430e-a834-08c248fe5855-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zkrg5\" (UID: \"e687910a-3808-430e-a834-08c248fe5855\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:03.002400 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:03.002376    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:26:03.099543 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:03.099510    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e687910a-3808-430e-a834-08c248fe5855-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zkrg5\" (UID: \"e687910a-3808-430e-a834-08c248fe5855\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:03.099543 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:03.099547    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e687910a-3808-430e-a834-08c248fe5855-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zkrg5\" (UID: \"e687910a-3808-430e-a834-08c248fe5855\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:03.099763 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:03.099594    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e687910a-3808-430e-a834-08c248fe5855-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zkrg5\" (UID: \"e687910a-3808-430e-a834-08c248fe5855\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:03.099763 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:03.099681    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e687910a-3808-430e-a834-08c248fe5855-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zkrg5\" (UID: \"e687910a-3808-430e-a834-08c248fe5855\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:03.099763 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:03.099709    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbzjj\" (UniqueName: \"kubernetes.io/projected/e687910a-3808-430e-a834-08c248fe5855-kube-api-access-gbzjj\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zkrg5\" (UID: \"e687910a-3808-430e-a834-08c248fe5855\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:03.099929 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:03.099856    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e687910a-3808-430e-a834-08c248fe5855-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zkrg5\" (UID: \"e687910a-3808-430e-a834-08c248fe5855\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:03.100079 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:03.100053    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e687910a-3808-430e-a834-08c248fe5855-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zkrg5\" (UID: \"e687910a-3808-430e-a834-08c248fe5855\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:03.100208 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:03.100107    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e687910a-3808-430e-a834-08c248fe5855-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zkrg5\" (UID: \"e687910a-3808-430e-a834-08c248fe5855\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:03.100208 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:03.100188    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e687910a-3808-430e-a834-08c248fe5855-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zkrg5\" (UID: \"e687910a-3808-430e-a834-08c248fe5855\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:03.101754 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:03.101734    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e687910a-3808-430e-a834-08c248fe5855-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zkrg5\" (UID: \"e687910a-3808-430e-a834-08c248fe5855\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:03.102158 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:03.102136    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e687910a-3808-430e-a834-08c248fe5855-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zkrg5\" (UID: \"e687910a-3808-430e-a834-08c248fe5855\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:03.162846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:03.162780    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbzjj\" (UniqueName: \"kubernetes.io/projected/e687910a-3808-430e-a834-08c248fe5855-kube-api-access-gbzjj\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zkrg5\" (UID: \"e687910a-3808-430e-a834-08c248fe5855\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:03.194799 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:03.194773    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:03.573630 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:26:03.573593    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode687910a_3808_430e_a834_08c248fe5855.slice/crio-ab27da97995bd8f3fa941bdfe552b183e6532e4196d44b6f0f4d3ab0c566a446 WatchSource:0}: Error finding container ab27da97995bd8f3fa941bdfe552b183e6532e4196d44b6f0f4d3ab0c566a446: Status 404 returned error can't find the container with id ab27da97995bd8f3fa941bdfe552b183e6532e4196d44b6f0f4d3ab0c566a446
Nov 03 17:26:03.574324 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:03.574297    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"]
Nov 03 17:26:03.574936 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:03.574920    6059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:26:03.928548 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:03.928469    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5" event={"ID":"e687910a-3808-430e-a834-08c248fe5855","Type":"ContainerStarted","Data":"ab27da97995bd8f3fa941bdfe552b183e6532e4196d44b6f0f4d3ab0c566a446"}
Nov 03 17:26:05.939205 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:05.939166    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5" event={"ID":"e687910a-3808-430e-a834-08c248fe5855","Type":"ContainerStarted","Data":"5357dbbab828454c6a770b5f9231b0961a428eff28766ea4bc4013c56eb270d6"}
Nov 03 17:26:06.072708 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.072637    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5" podStartSLOduration=2.55240116 podStartE2EDuration="4.072618601s" totalImagesPullingTime="1.520217441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:26:06.071620938 +0000 UTC m=+506.869333724" watchObservedRunningTime="2026-11-03 17:26:06.072618601 +0000 UTC m=+506.870331382"
Nov 03 17:26:06.195135 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.195051    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:06.709755 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.709716    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"]
Nov 03 17:26:06.713450 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.713430    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:06.769083 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.769047    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 03 17:26:06.821899 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.821862    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"]
Nov 03 17:26:06.838650 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.838607    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k\" (UID: \"3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:06.838820 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.838752    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k\" (UID: \"3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:06.838820 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.838809    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k\" (UID: \"3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:06.838952 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.838886    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpnzv\" (UniqueName: \"kubernetes.io/projected/3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2-kube-api-access-rpnzv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k\" (UID: \"3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:06.839024 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.838948    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k\" (UID: \"3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:06.839110 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.839083    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k\" (UID: \"3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:06.861795 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:26:06.861758    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:26:06.863291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.863267    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:26:06.940156 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.940123    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k\" (UID: \"3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:06.940630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.940177    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k\" (UID: \"3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:06.940630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.940213    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k\" (UID: \"3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:06.940630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.940242    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k\" (UID: \"3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:06.940630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.940335    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpnzv\" (UniqueName: \"kubernetes.io/projected/3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2-kube-api-access-rpnzv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k\" (UID: \"3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:06.940630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.940424    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k\" (UID: \"3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:06.940630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.940596    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k\" (UID: \"3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:06.940913 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.940767    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k\" (UID: \"3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:06.940913 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.940893    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k\" (UID: \"3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:06.943060 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.943039    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k\" (UID: \"3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:06.943301 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:06.943283    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k\" (UID: \"3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:07.004826 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:07.004737    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpnzv\" (UniqueName: \"kubernetes.io/projected/3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2-kube-api-access-rpnzv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k\" (UID: \"3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:07.025682 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:07.025656    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:07.408057 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:26:07.408021    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9c7d34_2338_4cc0_81c6_7e70e16c8cf2.slice/crio-eb1ca55f91bff9d5412e93f259b5cb78ac3c5c86c9dc664bb67ed6b70c2f7cc9 WatchSource:0}: Error finding container eb1ca55f91bff9d5412e93f259b5cb78ac3c5c86c9dc664bb67ed6b70c2f7cc9: Status 404 returned error can't find the container with id eb1ca55f91bff9d5412e93f259b5cb78ac3c5c86c9dc664bb67ed6b70c2f7cc9
Nov 03 17:26:07.409568 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:07.409546    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"]
Nov 03 17:26:07.951840 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:07.951807    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k" event={"ID":"3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2","Type":"ContainerStarted","Data":"eb1ca55f91bff9d5412e93f259b5cb78ac3c5c86c9dc664bb67ed6b70c2f7cc9"}
Nov 03 17:26:08.961071 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:08.961031    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k" event={"ID":"3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2","Type":"ContainerStarted","Data":"1d91b637222ee6e4c7429d10004aa24fb911158d70ec33a1788d83cae5c644f4"}
Nov 03 17:26:09.026468 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:09.026439    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:09.084735 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:09.084681    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k" podStartSLOduration=2.486536881 podStartE2EDuration="3.084665948s" totalImagesPullingTime="598.129067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:26:09.082570191 +0000 UTC m=+509.880283010" watchObservedRunningTime="2026-11-03 17:26:09.084665948 +0000 UTC m=+509.882378729"
Nov 03 17:26:09.895374 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:09.895324    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-4t6c8"]
Nov 03 17:26:09.899039 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:09.899022    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:09.967682 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:09.967654    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078a0cf0-420d-4a24-a609-55e00cc4bc09-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-4t6c8\" (UID: \"078a0cf0-420d-4a24-a609-55e00cc4bc09\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:09.968044 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:09.967701    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/078a0cf0-420d-4a24-a609-55e00cc4bc09-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-4t6c8\" (UID: \"078a0cf0-420d-4a24-a609-55e00cc4bc09\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:09.968044 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:09.967721    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/078a0cf0-420d-4a24-a609-55e00cc4bc09-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-4t6c8\" (UID: \"078a0cf0-420d-4a24-a609-55e00cc4bc09\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:09.968044 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:09.967894    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078a0cf0-420d-4a24-a609-55e00cc4bc09-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-4t6c8\" (UID: \"078a0cf0-420d-4a24-a609-55e00cc4bc09\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:09.968044 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:09.967930    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/078a0cf0-420d-4a24-a609-55e00cc4bc09-home\") pod \"echo-simulator-kserve-64b6f8f96c-4t6c8\" (UID: \"078a0cf0-420d-4a24-a609-55e00cc4bc09\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:09.968044 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:09.968011    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qbdq\" (UniqueName: \"kubernetes.io/projected/078a0cf0-420d-4a24-a609-55e00cc4bc09-kube-api-access-5qbdq\") pod \"echo-simulator-kserve-64b6f8f96c-4t6c8\" (UID: \"078a0cf0-420d-4a24-a609-55e00cc4bc09\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:10.004990 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:10.004965    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 03 17:26:10.006861 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:10.006837    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-4t6c8"]
Nov 03 17:26:10.045378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:10.045323    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:26:10.069501 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:10.069476    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qbdq\" (UniqueName: \"kubernetes.io/projected/078a0cf0-420d-4a24-a609-55e00cc4bc09-kube-api-access-5qbdq\") pod \"echo-simulator-kserve-64b6f8f96c-4t6c8\" (UID: \"078a0cf0-420d-4a24-a609-55e00cc4bc09\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:10.069597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:10.069534    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078a0cf0-420d-4a24-a609-55e00cc4bc09-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-4t6c8\" (UID: \"078a0cf0-420d-4a24-a609-55e00cc4bc09\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:10.069597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:10.069567    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/078a0cf0-420d-4a24-a609-55e00cc4bc09-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-4t6c8\" (UID: \"078a0cf0-420d-4a24-a609-55e00cc4bc09\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:10.069597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:10.069582    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/078a0cf0-420d-4a24-a609-55e00cc4bc09-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-4t6c8\" (UID: \"078a0cf0-420d-4a24-a609-55e00cc4bc09\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:10.069723 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:10.069655    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078a0cf0-420d-4a24-a609-55e00cc4bc09-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-4t6c8\" (UID: \"078a0cf0-420d-4a24-a609-55e00cc4bc09\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:10.069723 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:10.069686    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/078a0cf0-420d-4a24-a609-55e00cc4bc09-home\") pod \"echo-simulator-kserve-64b6f8f96c-4t6c8\" (UID: \"078a0cf0-420d-4a24-a609-55e00cc4bc09\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:10.069993 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:10.069972    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078a0cf0-420d-4a24-a609-55e00cc4bc09-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-4t6c8\" (UID: \"078a0cf0-420d-4a24-a609-55e00cc4bc09\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:10.070083 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:10.070065    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/078a0cf0-420d-4a24-a609-55e00cc4bc09-home\") pod \"echo-simulator-kserve-64b6f8f96c-4t6c8\" (UID: \"078a0cf0-420d-4a24-a609-55e00cc4bc09\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:10.070138 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:10.070081    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078a0cf0-420d-4a24-a609-55e00cc4bc09-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-4t6c8\" (UID: \"078a0cf0-420d-4a24-a609-55e00cc4bc09\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:10.071882 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:10.071859    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/078a0cf0-420d-4a24-a609-55e00cc4bc09-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-4t6c8\" (UID: \"078a0cf0-420d-4a24-a609-55e00cc4bc09\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:10.072091 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:10.072070    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/078a0cf0-420d-4a24-a609-55e00cc4bc09-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-4t6c8\" (UID: \"078a0cf0-420d-4a24-a609-55e00cc4bc09\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:10.132808 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:10.132782    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qbdq\" (UniqueName: \"kubernetes.io/projected/078a0cf0-420d-4a24-a609-55e00cc4bc09-kube-api-access-5qbdq\") pod \"echo-simulator-kserve-64b6f8f96c-4t6c8\" (UID: \"078a0cf0-420d-4a24-a609-55e00cc4bc09\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:10.211311 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:10.211287    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:10.591038 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:26:10.591010    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078a0cf0_420d_4a24_a609_55e00cc4bc09.slice/crio-4a36c24a4adda04d12ae2c1d44582bc7d111ddae15f037ffcab30ede2c7d7688 WatchSource:0}: Error finding container 4a36c24a4adda04d12ae2c1d44582bc7d111ddae15f037ffcab30ede2c7d7688: Status 404 returned error can't find the container with id 4a36c24a4adda04d12ae2c1d44582bc7d111ddae15f037ffcab30ede2c7d7688
Nov 03 17:26:10.592244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:10.592220    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-4t6c8"]
Nov 03 17:26:10.972003 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:10.971971    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8" event={"ID":"078a0cf0-420d-4a24-a609-55e00cc4bc09","Type":"ContainerStarted","Data":"4a36c24a4adda04d12ae2c1d44582bc7d111ddae15f037ffcab30ede2c7d7688"}
Nov 03 17:26:13.195743 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:13.195705    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:13.211589 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:13.211559    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:13.915390 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:13.915345    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:26:13.992825 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:13.992786    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8" event={"ID":"078a0cf0-420d-4a24-a609-55e00cc4bc09","Type":"ContainerStarted","Data":"42b6f246b755a85522889bd010593c38a7b0af32575955ea5cc6eb3c52c4fa59"}
Nov 03 17:26:14.005295 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:14.005270    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zkrg5"
Nov 03 17:26:14.115688 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:14.115630    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8" podStartSLOduration=2.388733679 podStartE2EDuration="5.11561411s" totalImagesPullingTime="2.726880431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:26:14.114735211 +0000 UTC m=+514.912447992" watchObservedRunningTime="2026-11-03 17:26:14.11561411 +0000 UTC m=+514.913326920"
Nov 03 17:26:14.211959 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:14.211930    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:16.612423 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.612387    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"]
Nov 03 17:26:16.616071 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.616055    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:16.721993 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.721960    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 03 17:26:16.723830 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.723809    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"]
Nov 03 17:26:16.732622 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.732596    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c194a4be-0499-419f-a4a1-910dbaafd303-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl\" (UID: \"c194a4be-0499-419f-a4a1-910dbaafd303\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:16.732747 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.732671    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c194a4be-0499-419f-a4a1-910dbaafd303-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl\" (UID: \"c194a4be-0499-419f-a4a1-910dbaafd303\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:16.732747 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.732725    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c194a4be-0499-419f-a4a1-910dbaafd303-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl\" (UID: \"c194a4be-0499-419f-a4a1-910dbaafd303\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:16.732859 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.732821    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sjx4\" (UniqueName: \"kubernetes.io/projected/c194a4be-0499-419f-a4a1-910dbaafd303-kube-api-access-2sjx4\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl\" (UID: \"c194a4be-0499-419f-a4a1-910dbaafd303\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:16.732914 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.732890    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c194a4be-0499-419f-a4a1-910dbaafd303-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl\" (UID: \"c194a4be-0499-419f-a4a1-910dbaafd303\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:16.732956 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.732931    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c194a4be-0499-419f-a4a1-910dbaafd303-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl\" (UID: \"c194a4be-0499-419f-a4a1-910dbaafd303\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:16.780143 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.780107    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:26:16.833955 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.833926    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c194a4be-0499-419f-a4a1-910dbaafd303-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl\" (UID: \"c194a4be-0499-419f-a4a1-910dbaafd303\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:16.833955 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.833956    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c194a4be-0499-419f-a4a1-910dbaafd303-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl\" (UID: \"c194a4be-0499-419f-a4a1-910dbaafd303\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:16.834132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.833987    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sjx4\" (UniqueName: \"kubernetes.io/projected/c194a4be-0499-419f-a4a1-910dbaafd303-kube-api-access-2sjx4\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl\" (UID: \"c194a4be-0499-419f-a4a1-910dbaafd303\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:16.834132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.834036    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c194a4be-0499-419f-a4a1-910dbaafd303-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl\" (UID: \"c194a4be-0499-419f-a4a1-910dbaafd303\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:16.834132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.834072    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c194a4be-0499-419f-a4a1-910dbaafd303-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl\" (UID: \"c194a4be-0499-419f-a4a1-910dbaafd303\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:16.834279 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.834215    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c194a4be-0499-419f-a4a1-910dbaafd303-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl\" (UID: \"c194a4be-0499-419f-a4a1-910dbaafd303\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:16.834459 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.834440    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c194a4be-0499-419f-a4a1-910dbaafd303-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl\" (UID: \"c194a4be-0499-419f-a4a1-910dbaafd303\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:16.834532 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.834472    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c194a4be-0499-419f-a4a1-910dbaafd303-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl\" (UID: \"c194a4be-0499-419f-a4a1-910dbaafd303\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:16.834586 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.834554    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c194a4be-0499-419f-a4a1-910dbaafd303-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl\" (UID: \"c194a4be-0499-419f-a4a1-910dbaafd303\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:16.836245 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.836224    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c194a4be-0499-419f-a4a1-910dbaafd303-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl\" (UID: \"c194a4be-0499-419f-a4a1-910dbaafd303\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:16.836570 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.836551    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c194a4be-0499-419f-a4a1-910dbaafd303-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl\" (UID: \"c194a4be-0499-419f-a4a1-910dbaafd303\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:16.896940 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.896886    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sjx4\" (UniqueName: \"kubernetes.io/projected/c194a4be-0499-419f-a4a1-910dbaafd303-kube-api-access-2sjx4\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl\" (UID: \"c194a4be-0499-419f-a4a1-910dbaafd303\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:16.927728 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:16.927705    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:17.026628 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:17.026594    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:17.038799 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:17.038763    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:17.301330 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:26:17.301293    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc194a4be_0499_419f_a4a1_910dbaafd303.slice/crio-96f27ca346cc0b37f1cddf83aaba989d85de55fe538571f7b3c65a575eb0d98a WatchSource:0}: Error finding container 96f27ca346cc0b37f1cddf83aaba989d85de55fe538571f7b3c65a575eb0d98a: Status 404 returned error can't find the container with id 96f27ca346cc0b37f1cddf83aaba989d85de55fe538571f7b3c65a575eb0d98a
Nov 03 17:26:17.303642 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:17.303621    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"]
Nov 03 17:26:18.016441 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:18.016399    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl" event={"ID":"c194a4be-0499-419f-a4a1-910dbaafd303","Type":"ContainerStarted","Data":"d343ea0627f959f7f8589d08533822dc80e4f9b2d3983b8c79a30b5a3abd798f"}
Nov 03 17:26:18.016915 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:18.016465    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl" event={"ID":"c194a4be-0499-419f-a4a1-910dbaafd303","Type":"ContainerStarted","Data":"96f27ca346cc0b37f1cddf83aaba989d85de55fe538571f7b3c65a575eb0d98a"}
Nov 03 17:26:18.027941 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:18.027919    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k"
Nov 03 17:26:18.137638 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:18.137586    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl" podStartSLOduration=1.506792248 podStartE2EDuration="2.137568805s" totalImagesPullingTime="630.776557ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:26:18.137516447 +0000 UTC m=+518.935229231" watchObservedRunningTime="2026-11-03 17:26:18.137568805 +0000 UTC m=+518.935281587"
Nov 03 17:26:18.928182 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:18.928154    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:20.212141 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:20.212109    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:20.224880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:20.224851    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:21.043248 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:21.043219    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-4t6c8"
Nov 03 17:26:26.927940 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:26.927901    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:26.940127 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:26.940101    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:27.077746 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:27.077715    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:26:28.384112 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.384075    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"]
Nov 03 17:26:28.387765 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.387744    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.546826 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.546787    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.546826 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.546829    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.547027 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.546848    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1db59e38-c479-48cd-b641-48c6e44a0b23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.547027 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.546919    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.547027 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.547023    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.547181 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.547047    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.547181 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.547144    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.547181 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.547175    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.547287 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.547193    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmj8t\" (UniqueName: \"kubernetes.io/projected/1db59e38-c479-48cd-b641-48c6e44a0b23-kube-api-access-dmj8t\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.607213 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.607176    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"]
Nov 03 17:26:28.648509 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.648434    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.648509 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.648477    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.648509 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.648496    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmj8t\" (UniqueName: \"kubernetes.io/projected/1db59e38-c479-48cd-b641-48c6e44a0b23-kube-api-access-dmj8t\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.648758 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.648554    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.648758 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.648590    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.648758 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.648616    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1db59e38-c479-48cd-b641-48c6e44a0b23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.648758 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.648642    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.648758 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.648695    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.648758 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.648727    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.648997 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.648842    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.649344 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.649310    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.649501 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.649409    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.649501 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.649419    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1db59e38-c479-48cd-b641-48c6e44a0b23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.649501 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.649405    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.651133 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.651112    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.651248 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.651238    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.712143 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.712109    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:28.712306 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:28.712216    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmj8t\" (UniqueName: \"kubernetes.io/projected/1db59e38-c479-48cd-b641-48c6e44a0b23-kube-api-access-dmj8t\") pod \"maas-default-gateway-openshift-default-58c6585447-679z9\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:29.000443 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:29.000412    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:29.426380 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:26:29.426321    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db59e38_c479_48cd_b641_48c6e44a0b23.slice/crio-a38a017659010860eea7940ff07d8f594738a730bef8f2ad8c48f6a1b1f844e8 WatchSource:0}: Error finding container a38a017659010860eea7940ff07d8f594738a730bef8f2ad8c48f6a1b1f844e8: Status 404 returned error can't find the container with id a38a017659010860eea7940ff07d8f594738a730bef8f2ad8c48f6a1b1f844e8
Nov 03 17:26:29.428383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:29.428307    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"]
Nov 03 17:26:29.428973 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:29.428943    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:26:29.429051 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:29.429004    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:26:29.429051 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:29.429032    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:26:30.087589 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:30.087551    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9" event={"ID":"1db59e38-c479-48cd-b641-48c6e44a0b23","Type":"ContainerStarted","Data":"d6217027333c556c73a797c90cdd7e0f1aa6801a16eab7c1dd7e17c853582af5"}
Nov 03 17:26:30.087805 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:30.087612    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9" event={"ID":"1db59e38-c479-48cd-b641-48c6e44a0b23","Type":"ContainerStarted","Data":"a38a017659010860eea7940ff07d8f594738a730bef8f2ad8c48f6a1b1f844e8"}
Nov 03 17:26:30.325373 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:30.325298    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9" podStartSLOduration=2.325285208 podStartE2EDuration="2.325285208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:26:30.272584288 +0000 UTC m=+531.070297070" watchObservedRunningTime="2026-11-03 17:26:30.325285208 +0000 UTC m=+531.122997979"
Nov 03 17:26:31.000903 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:31.000862    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:31.002500 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:31.002473    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-679z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 17:26:31.002627 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:31.002519    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9" podUID="1db59e38-c479-48cd-b641-48c6e44a0b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 17:26:32.001237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:32.001199    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-679z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 17:26:32.001660 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:32.001264    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9" podUID="1db59e38-c479-48cd-b641-48c6e44a0b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 17:26:33.001534 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:33.001495    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-679z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 17:26:33.002007 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:33.001552    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9" podUID="1db59e38-c479-48cd-b641-48c6e44a0b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 17:26:33.056145 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:33.056086    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:26:34.001556 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:34.001522    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-679z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 17:26:34.002040 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:34.001580    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9" podUID="1db59e38-c479-48cd-b641-48c6e44a0b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 17:26:35.001109 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:35.001071    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-679z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 17:26:35.001311 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:35.001140    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9" podUID="1db59e38-c479-48cd-b641-48c6e44a0b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 17:26:35.078433 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:35.078402    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:26:36.001190 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:36.001154    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-679z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 17:26:36.001416 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:36.001206    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9" podUID="1db59e38-c479-48cd-b641-48c6e44a0b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 17:26:36.861514 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:26:36.861471    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:26:37.001476 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:37.001433    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-679z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 17:26:37.001630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:37.001500    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9" podUID="1db59e38-c479-48cd-b641-48c6e44a0b23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 17:26:38.840607 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:38.840558    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:38.841172 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:38.841147    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:38.842052 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:38.842033    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:26:43.507215 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.507180    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"]
Nov 03 17:26:43.514122 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.514099    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.597832 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.597790    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.598025 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.597853    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.598025 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.597930    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.598158 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.598023    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z6jm\" (UniqueName: \"kubernetes.io/projected/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-kube-api-access-9z6jm\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.598158 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.598101    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.598158 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.598140    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.598318 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.598179    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.598318 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.598216    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.598454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.598334    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.699831 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.699785    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.700041 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.699880    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.700041 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.699906    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.700041 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.699925    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.700041 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.699962    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z6jm\" (UniqueName: \"kubernetes.io/projected/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-kube-api-access-9z6jm\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.700041 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.700008    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.700041 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.700033    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.700432 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.700068    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.700432 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.700211    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.700432 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.700266    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.700432 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.700339    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.700570 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.700466    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.700701 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.700676    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.700820 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.700801    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.702266 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.702242    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.702598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.702579    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.730067 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.730032    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"]
Nov 03 17:26:43.763084 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.763018    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.763214 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.763148    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z6jm\" (UniqueName: \"kubernetes.io/projected/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-kube-api-access-9z6jm\") pod \"maas-default-gateway-openshift-default-58c6585447-wgvjn\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:43.827662 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:43.827624    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:44.250879 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:26:44.250850    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42e8a1e_2f87_4d9d_bbc4_c76d84d84dbb.slice/crio-0e77b9788d9234a9449759f34092ea749845b5741a752805e652071017dbaf8d WatchSource:0}: Error finding container 0e77b9788d9234a9449759f34092ea749845b5741a752805e652071017dbaf8d: Status 404 returned error can't find the container with id 0e77b9788d9234a9449759f34092ea749845b5741a752805e652071017dbaf8d
Nov 03 17:26:44.252768 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:44.252745    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"]
Nov 03 17:26:44.253325 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:44.253285    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:26:44.253549 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:44.253522    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:26:44.253628 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:44.253573    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:26:45.164244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:45.164206    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn" event={"ID":"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb","Type":"ContainerStarted","Data":"5e1cf1c5fe254aedbb4f43169db791e3199427bc008159b5858a57683616f0b1"}
Nov 03 17:26:45.164648 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:45.164267    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn" event={"ID":"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb","Type":"ContainerStarted","Data":"0e77b9788d9234a9449759f34092ea749845b5741a752805e652071017dbaf8d"}
Nov 03 17:26:45.369389 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:45.364180    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn" podStartSLOduration=2.364152624 podStartE2EDuration="2.364152624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:26:45.361150827 +0000 UTC m=+546.158863641" watchObservedRunningTime="2026-11-03 17:26:45.364152624 +0000 UTC m=+546.161865406"
Nov 03 17:26:45.828287 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:45.828248    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:45.854551 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:45.854513    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wgvjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 17:26:45.854735 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:45.854571    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn" podUID="e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 17:26:46.828075 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:46.828036    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wgvjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 17:26:46.828481 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:46.828094    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn" podUID="e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 17:26:47.829049 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:47.828968    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wgvjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 17:26:47.829049 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:47.829027    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn" podUID="e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 17:26:48.798370 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:48.798321    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:26:48.828170 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:48.828147    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wgvjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 17:26:48.828349 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:48.828311    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn" podUID="e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 17:26:48.997689 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:48.997658    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:26:49.828287 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:49.828253    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wgvjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 17:26:49.828474 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:49.828309    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn" podUID="e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 17:26:50.502894 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:50.502853    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:26:50.800701 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:50.800612    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:26:50.829062 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:50.829029    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wgvjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 17:26:50.829209 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:50.829090    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn" podUID="e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 17:26:51.828259 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:51.828225    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wgvjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 17:26:51.828670 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:51.828281    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn" podUID="e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 17:26:53.455904 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:53.455870    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:53.456408 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:53.456317    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:26:53.457043 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:26:53.457025    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:27:06.861445 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:27:06.861409    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:27:14.617029 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:14.616995    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:20.305248 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:20.305214    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:20.500664 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:20.500632    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:22.609840 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:22.609784    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:22.786454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:22.786419    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:23.204277 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.204242    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"]
Nov 03 17:27:23.207992 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.207971    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.273959 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.273927    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2355f4-openshift-default-dockercfg-mdxdx\""
Nov 03 17:27:23.325492 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.325452    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:23.327743 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.327718    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"]
Nov 03 17:27:23.337831 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.337792    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-podinfo\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.338122 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.338094    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-token\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.338316 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.338295    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-workload-certs\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.338443 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.338334    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-workload-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.338443 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.338406    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-envoy\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.338443 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.338430    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c795d8-dfff-43d8-88a2-de65cc01a51f-istiod-ca-cert\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.338585 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.338459    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-credential-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.338585 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.338508    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btkrm\" (UniqueName: \"kubernetes.io/projected/63c795d8-dfff-43d8-88a2-de65cc01a51f-kube-api-access-btkrm\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.338585 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.338537    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-data\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.439532 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.439491    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-token\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.439742 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.439554    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-workload-certs\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.439742 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.439575    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-workload-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.439742 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.439597    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-envoy\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.439742 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.439612    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c795d8-dfff-43d8-88a2-de65cc01a51f-istiod-ca-cert\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.439742 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.439631    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-credential-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.439742 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.439659    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btkrm\" (UniqueName: \"kubernetes.io/projected/63c795d8-dfff-43d8-88a2-de65cc01a51f-kube-api-access-btkrm\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.440079 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.439814    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-data\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.440079 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.439946    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-podinfo\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.440079 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.440000    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-credential-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.440079 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.440066    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-workload-certs\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.440243 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.440143    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-workload-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.440280 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.440266    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-data\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.440420 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.440399    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c795d8-dfff-43d8-88a2-de65cc01a51f-istiod-ca-cert\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.442139 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.442121    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-envoy\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.442469 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.442453    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-podinfo\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.507612 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.507583    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btkrm\" (UniqueName: \"kubernetes.io/projected/63c795d8-dfff-43d8-88a2-de65cc01a51f-kube-api-access-btkrm\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.507864 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.507840    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-token\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.521758 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.521731    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:23.528855 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.528806    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:23.535849 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.535827    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"]
Nov 03 17:27:23.539909 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.539891    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.606086 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.606047    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"]
Nov 03 17:27:23.642007 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.641977    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hktx\" (UniqueName: \"kubernetes.io/projected/84690202-b89e-4135-aebb-aa53c0e18463-kube-api-access-4hktx\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.642381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.642031    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-workload-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.642381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.642057    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-istio-envoy\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.642381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.642176    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-workload-certs\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.642381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.642239    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-istio-data\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.642554 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.642455    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-credential-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.642554 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.642489    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84690202-b89e-4135-aebb-aa53c0e18463-istio-token\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.642554 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.642518    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84690202-b89e-4135-aebb-aa53c0e18463-istiod-ca-cert\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.642664 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.642578    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84690202-b89e-4135-aebb-aa53c0e18463-istio-podinfo\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.643179 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.643156    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-687fb3-openshift-default-dockercfg-r5dqj\""
Nov 03 17:27:23.721305 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.721268    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:23.743615 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.743585    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-workload-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.743760 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.743625    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-istio-envoy\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.743760 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.743653    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-workload-certs\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.743760 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.743697    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-istio-data\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.743912 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.743771    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-credential-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.743912 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.743803    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84690202-b89e-4135-aebb-aa53c0e18463-istio-token\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.744012 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.743925    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84690202-b89e-4135-aebb-aa53c0e18463-istiod-ca-cert\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.744012 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.743980    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84690202-b89e-4135-aebb-aa53c0e18463-istio-podinfo\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.744116 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.744029    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hktx\" (UniqueName: \"kubernetes.io/projected/84690202-b89e-4135-aebb-aa53c0e18463-kube-api-access-4hktx\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.744116 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.744030    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-workload-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.744116 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.744107    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-workload-certs\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.744285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.744176    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-istio-data\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.744285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.744249    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-credential-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.744683 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.744661    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84690202-b89e-4135-aebb-aa53c0e18463-istiod-ca-cert\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.745988 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.745970    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-istio-envoy\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.746346 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.746327    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84690202-b89e-4135-aebb-aa53c0e18463-istio-podinfo\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.811013 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.810944    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hktx\" (UniqueName: \"kubernetes.io/projected/84690202-b89e-4135-aebb-aa53c0e18463-kube-api-access-4hktx\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.813975 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.813953    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84690202-b89e-4135-aebb-aa53c0e18463-istio-token\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.894585 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.894281    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:23.912566 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.912537    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:23.980096 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.980064    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"]
Nov 03 17:27:23.981655 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:27:23.981624    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c795d8_dfff_43d8_88a2_de65cc01a51f.slice/crio-8ee8bb931e915fda871b21389bca5ae6dd22f41adb0bc406bf6ed734335532a9 WatchSource:0}: Error finding container 8ee8bb931e915fda871b21389bca5ae6dd22f41adb0bc406bf6ed734335532a9: Status 404 returned error can't find the container with id 8ee8bb931e915fda871b21389bca5ae6dd22f41adb0bc406bf6ed734335532a9
Nov 03 17:27:23.984193 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.984159    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:27:23.984283 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.984227    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:27:23.984283 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:23.984274    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:27:24.279592 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:27:24.278814    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84690202_b89e_4135_aebb_aa53c0e18463.slice/crio-6f5419454a6e1264fbce682faf7ff010696c1a998a5c6444eda01bbaa51fc712 WatchSource:0}: Error finding container 6f5419454a6e1264fbce682faf7ff010696c1a998a5c6444eda01bbaa51fc712: Status 404 returned error can't find the container with id 6f5419454a6e1264fbce682faf7ff010696c1a998a5c6444eda01bbaa51fc712
Nov 03 17:27:24.281083 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.281058    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"]
Nov 03 17:27:24.282214 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.282178    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:27:24.282317 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.282265    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:27:24.282317 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.282307    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:27:24.388132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.387802    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5" event={"ID":"63c795d8-dfff-43d8-88a2-de65cc01a51f","Type":"ContainerStarted","Data":"30e23af5381fd9a80066543d1207c917691f7f5d1d864d76eabe6a0fc5d737db"}
Nov 03 17:27:24.388132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.387889    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5" event={"ID":"63c795d8-dfff-43d8-88a2-de65cc01a51f","Type":"ContainerStarted","Data":"8ee8bb931e915fda871b21389bca5ae6dd22f41adb0bc406bf6ed734335532a9"}
Nov 03 17:27:24.389984 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.389955    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c" event={"ID":"84690202-b89e-4135-aebb-aa53c0e18463","Type":"ContainerStarted","Data":"0582a647eadd190de49d8ba2d75e981eb1985247445fab56d13a22aa2c1975c9"}
Nov 03 17:27:24.390105 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.390002    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c" event={"ID":"84690202-b89e-4135-aebb-aa53c0e18463","Type":"ContainerStarted","Data":"6f5419454a6e1264fbce682faf7ff010696c1a998a5c6444eda01bbaa51fc712"}
Nov 03 17:27:24.499143 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.499112    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"]
Nov 03 17:27:24.502944 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.502924    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.515319 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.515276    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5" podStartSLOduration=1.515264013 podStartE2EDuration="1.515264013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:27:24.513289497 +0000 UTC m=+585.311002290" watchObservedRunningTime="2026-11-03 17:27:24.515264013 +0000 UTC m=+585.312976794"
Nov 03 17:27:24.522383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.522334    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:24.557515 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.557439    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-8fc5ec-openshift-default-dockercfg-ws74v\""
Nov 03 17:27:24.569490 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.569456    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"]
Nov 03 17:27:24.631978 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.631940    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:24.639820 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.639769    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c" podStartSLOduration=1.639755233 podStartE2EDuration="1.639755233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:27:24.638176958 +0000 UTC m=+585.435889751" watchObservedRunningTime="2026-11-03 17:27:24.639755233 +0000 UTC m=+585.437468014"
Nov 03 17:27:24.658015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.655965    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f87nf\" (UniqueName: \"kubernetes.io/projected/9a6b46be-ed11-4708-b1a9-4500870e4d2e-kube-api-access-f87nf\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.658015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.656186    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-workload-socket\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.658015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.656435    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-credential-socket\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.658015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.656562    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-data\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.658015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.656621    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istiod-ca-cert\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.658015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.656658    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-podinfo\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.658015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.656691    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-workload-certs\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.658015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.656710    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-envoy\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.658015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.656727    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-token\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.761146 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.758109    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-data\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.761146 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.758626    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-data\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.761146 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.758912    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istiod-ca-cert\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.768547 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.768446    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-podinfo\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.768547 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.768529    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-workload-certs\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.768783 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.768556    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-envoy\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.768783 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.768577    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-token\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.768783 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.768621    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87nf\" (UniqueName: \"kubernetes.io/projected/9a6b46be-ed11-4708-b1a9-4500870e4d2e-kube-api-access-f87nf\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.768783 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.768677    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-workload-socket\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.768783 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.768710    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-credential-socket\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.769199 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.769085    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-credential-socket\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.769651 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.769343    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-workload-socket\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.769651 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.769602    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-workload-certs\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.771784 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.771729    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istiod-ca-cert\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.771996 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.771953    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-envoy\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.775163 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.775140    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-podinfo\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.832623 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.832538    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-token\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.832754 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.832623    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f87nf\" (UniqueName: \"kubernetes.io/projected/9a6b46be-ed11-4708-b1a9-4500870e4d2e-kube-api-access-f87nf\") pod \"e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") " pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:24.895231 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.895180    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:24.953779 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:24.953745    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:25.116896 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:25.116820    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:25.528201 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:25.528165    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:25.543451 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:27:25.543420    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6b46be_ed11_4708_b1a9_4500870e4d2e.slice/crio-b9e526777553de18d2b083b25f94a2e6f2c6b4f8a7c8b03cf67bb23ff056b027 WatchSource:0}: Error finding container b9e526777553de18d2b083b25f94a2e6f2c6b4f8a7c8b03cf67bb23ff056b027: Status 404 returned error can't find the container with id b9e526777553de18d2b083b25f94a2e6f2c6b4f8a7c8b03cf67bb23ff056b027
Nov 03 17:27:25.546099 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:25.546064    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:27:25.546209 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:25.546146    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:27:25.546209 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:25.546190    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:27:25.546727 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:25.546606    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"]
Nov 03 17:27:25.902065 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:25.901960    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:26.403182 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:26.403145    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs" event={"ID":"9a6b46be-ed11-4708-b1a9-4500870e4d2e","Type":"ContainerStarted","Data":"b1652b314b78cb0371f38ad4ca032887e99d03ec747732d6435a8959ae300773"}
Nov 03 17:27:26.403376 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:26.403204    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs" event={"ID":"9a6b46be-ed11-4708-b1a9-4500870e4d2e","Type":"ContainerStarted","Data":"b9e526777553de18d2b083b25f94a2e6f2c6b4f8a7c8b03cf67bb23ff056b027"}
Nov 03 17:27:26.403558 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:26.403539    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:26.403863 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:26.403840    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:26.404567 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:26.404547    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:26.404728 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:26.404715    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:27:26.528871 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:26.528818    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs" podStartSLOduration=2.528803182 podStartE2EDuration="2.528803182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:27:26.527800674 +0000 UTC m=+587.325513455" watchObservedRunningTime="2026-11-03 17:27:26.528803182 +0000 UTC m=+587.326515962"
Nov 03 17:27:27.117278 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:27.117243    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:27.122436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:27.122408    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:27.409283 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:27.409146    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:27.410058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:27.410040    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:27:27.575089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:27.575053    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:27.806597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:27.806561    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:28.169610 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:28.169520    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:28.366277 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:28.366242    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:30.668268 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:30.668237    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:30.843966 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:30.843930    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:31.869430 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:31.869382    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:32.082965 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:32.082934    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:32.269869 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:32.269838    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:32.621248 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:32.621165    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:33.412636 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:33.412603    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:33.684641 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:33.684548    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:33.874168 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:33.874133    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:35.194169 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:35.194131    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:35.393837 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:35.393796    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:36.861704 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:27:36.861670    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:27:37.526155 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:37.526102    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:37.719267 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:37.719233    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:38.418336 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:38.418297    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:38.605711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:38.605678    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:38.809901 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:38.809865    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:39.394295 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:39.394268    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 17:27:39.398844 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:39.398820    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 17:27:39.412329 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:39.412310    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:27:39.416897 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:39.416877    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:27:43.065733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:43.065688    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:43.269340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:43.269302    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:43.463115 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:43.463083    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:45.754993 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:45.754935    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:47.762583 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:47.762552    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:51.584608 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:51.584569    6059 generic.go:356] "Generic (PLEG): container finished" podID="84690202-b89e-4135-aebb-aa53c0e18463" containerID="0582a647eadd190de49d8ba2d75e981eb1985247445fab56d13a22aa2c1975c9" exitCode=0
Nov 03 17:27:51.585308 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:51.585194    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c" event={"ID":"84690202-b89e-4135-aebb-aa53c0e18463","Type":"ContainerDied","Data":"0582a647eadd190de49d8ba2d75e981eb1985247445fab56d13a22aa2c1975c9"}
Nov 03 17:27:51.585863 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:51.585844    6059 scope.go:122] "RemoveContainer" containerID="0582a647eadd190de49d8ba2d75e981eb1985247445fab56d13a22aa2c1975c9"
Nov 03 17:27:51.587490 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:51.587455    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:27:51.587597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:51.587553    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:27:51.587666 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:51.587609    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:27:52.357515 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:52.357475    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:52.605999 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:52.604737    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c" event={"ID":"84690202-b89e-4135-aebb-aa53c0e18463","Type":"ContainerStarted","Data":"2ae27f4cfca9606459d75e02e45f3162805a5e58f43d5339e5dd6c65ffd91371"}
Nov 03 17:27:52.896015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:52.895979    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:52.896435 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:52.896412    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 17:27:52.896627 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:52.896604    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c" podUID="84690202-b89e-4135-aebb-aa53c0e18463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 17:27:53.379381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:53.379009    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v"]
Nov 03 17:27:53.401704 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:53.401667    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v"
Nov 03 17:27:53.417891 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:53.417862    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcef9d95-37e7-4d72-bd24-562f646c2ec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99aad8a2-z2t6v\" (UID: \"dcef9d95-37e7-4d72-bd24-562f646c2ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v"
Nov 03 17:27:53.418031 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:53.417894    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzwzc\" (UniqueName: \"kubernetes.io/projected/dcef9d95-37e7-4d72-bd24-562f646c2ec9-kube-api-access-hzwzc\") pod \"maas-api-revoke-sub-99aad8a2-z2t6v\" (UID: \"dcef9d95-37e7-4d72-bd24-562f646c2ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v"
Nov 03 17:27:53.418031 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:53.417917    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcef9d95-37e7-4d72-bd24-562f646c2ec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99aad8a2-z2t6v\" (UID: \"dcef9d95-37e7-4d72-bd24-562f646c2ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v"
Nov 03 17:27:53.450517 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:53.444999    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v"]
Nov 03 17:27:53.510445 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:53.505544    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:27:53.510445 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:53.505647    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:27:53.510445 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:53.505832    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:27:53.510445 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:53.505903    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:27:53.519265 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:53.519223    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcef9d95-37e7-4d72-bd24-562f646c2ec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99aad8a2-z2t6v\" (UID: \"dcef9d95-37e7-4d72-bd24-562f646c2ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v"
Nov 03 17:27:53.519553 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:53.519524    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwzc\" (UniqueName: \"kubernetes.io/projected/dcef9d95-37e7-4d72-bd24-562f646c2ec9-kube-api-access-hzwzc\") pod \"maas-api-revoke-sub-99aad8a2-z2t6v\" (UID: \"dcef9d95-37e7-4d72-bd24-562f646c2ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v"
Nov 03 17:27:53.519719 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:53.519694    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcef9d95-37e7-4d72-bd24-562f646c2ec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99aad8a2-z2t6v\" (UID: \"dcef9d95-37e7-4d72-bd24-562f646c2ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v"
Nov 03 17:27:53.520235 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:53.520209    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcef9d95-37e7-4d72-bd24-562f646c2ec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99aad8a2-z2t6v\" (UID: \"dcef9d95-37e7-4d72-bd24-562f646c2ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v"
Nov 03 17:27:53.521975 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:53.521956    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcef9d95-37e7-4d72-bd24-562f646c2ec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99aad8a2-z2t6v\" (UID: \"dcef9d95-37e7-4d72-bd24-562f646c2ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v"
Nov 03 17:27:53.609732 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:53.609647    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzwzc\" (UniqueName: \"kubernetes.io/projected/dcef9d95-37e7-4d72-bd24-562f646c2ec9-kube-api-access-hzwzc\") pod \"maas-api-revoke-sub-99aad8a2-z2t6v\" (UID: \"dcef9d95-37e7-4d72-bd24-562f646c2ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v"
Nov 03 17:27:53.721338 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:53.721305    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v"
Nov 03 17:27:53.904384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:53.902773    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 17:27:53.904384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:53.902839    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c" podUID="84690202-b89e-4135-aebb-aa53c0e18463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 17:27:54.252381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:54.249984    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v"]
Nov 03 17:27:54.310461 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:27:54.310088    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcef9d95_37e7_4d72_bd24_562f646c2ec9.slice/crio-f6501231ab6a54103d9e4622740be9184930d321e666ca0a46bc0b453f4183e3 WatchSource:0}: Error finding container f6501231ab6a54103d9e4622740be9184930d321e666ca0a46bc0b453f4183e3: Status 404 returned error can't find the container with id f6501231ab6a54103d9e4622740be9184930d321e666ca0a46bc0b453f4183e3
Nov 03 17:27:54.620349 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:54.620265    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v" event={"ID":"dcef9d95-37e7-4d72-bd24-562f646c2ec9","Type":"ContainerStarted","Data":"f6501231ab6a54103d9e4622740be9184930d321e666ca0a46bc0b453f4183e3"}
Nov 03 17:27:54.898440 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:54.897528    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 17:27:54.898440 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:54.897591    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c" podUID="84690202-b89e-4135-aebb-aa53c0e18463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 17:27:55.894953 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:55.894908    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 17:27:55.895469 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:55.894982    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c" podUID="84690202-b89e-4135-aebb-aa53c0e18463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 17:27:56.408409 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:56.404519    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:56.894882 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:56.894837    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 17:27:56.895094 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:56.894915    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c" podUID="84690202-b89e-4135-aebb-aa53c0e18463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 17:27:57.156015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:57.155906    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:57.384641 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:57.384582    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:57.455436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:57.453460    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:27:57.455436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:57.453524    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5" podUID="63c795d8-dfff-43d8-88a2-de65cc01a51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:27:57.867163 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:57.867072    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:57.896058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:57.895586    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 17:27:57.896058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:57.895650    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c" podUID="84690202-b89e-4135-aebb-aa53c0e18463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 17:27:58.464140 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:58.464094    6059 patch_prober.go:29] interesting pod/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:27:58.464423 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:58.464392    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs" podUID="9a6b46be-ed11-4708-b1a9-4500870e4d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:27:58.551751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:58.551715    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:58.753015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:58.752905    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:27:58.903133 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:58.903097    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:27:58.906052 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:27:58.906023    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:28:00.722716 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:00.722684    6059 generic.go:356] "Generic (PLEG): container finished" podID="9a6b46be-ed11-4708-b1a9-4500870e4d2e" containerID="b1652b314b78cb0371f38ad4ca032887e99d03ec747732d6435a8959ae300773" exitCode=0
Nov 03 17:28:00.723221 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:00.722738    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs" event={"ID":"9a6b46be-ed11-4708-b1a9-4500870e4d2e","Type":"ContainerDied","Data":"b1652b314b78cb0371f38ad4ca032887e99d03ec747732d6435a8959ae300773"}
Nov 03 17:28:00.723221 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:00.723098    6059 scope.go:122] "RemoveContainer" containerID="b1652b314b78cb0371f38ad4ca032887e99d03ec747732d6435a8959ae300773"
Nov 03 17:28:00.725226 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:00.725192    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:28:00.725313 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:00.725293    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:28:00.725400 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:00.725375    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:28:01.767226 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:01.766142    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs" event={"ID":"9a6b46be-ed11-4708-b1a9-4500870e4d2e","Type":"ContainerStarted","Data":"58ef2ec45f12d423530a12f40851cdff784f4dbeeadac2c5e3a75c65013b064d"}
Nov 03 17:28:02.117621 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:02.117141    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:28:02.117621 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:02.117275    6059 patch_prober.go:29] interesting pod/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 17:28:02.117621 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:02.117333    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs" podUID="9a6b46be-ed11-4708-b1a9-4500870e4d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 17:28:02.125676 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:28:02.125620    6059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c795d8_dfff_43d8_88a2_de65cc01a51f.slice/crio-30e23af5381fd9a80066543d1207c917691f7f5d1d864d76eabe6a0fc5d737db.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 17:28:02.472708 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:02.472672    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:02.782538 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:02.782465    6059 generic.go:356] "Generic (PLEG): container finished" podID="63c795d8-dfff-43d8-88a2-de65cc01a51f" containerID="30e23af5381fd9a80066543d1207c917691f7f5d1d864d76eabe6a0fc5d737db" exitCode=0
Nov 03 17:28:02.782960 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:02.782547    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5" event={"ID":"63c795d8-dfff-43d8-88a2-de65cc01a51f","Type":"ContainerDied","Data":"30e23af5381fd9a80066543d1207c917691f7f5d1d864d76eabe6a0fc5d737db"}
Nov 03 17:28:02.783071 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:02.783052    6059 scope.go:122] "RemoveContainer" containerID="30e23af5381fd9a80066543d1207c917691f7f5d1d864d76eabe6a0fc5d737db"
Nov 03 17:28:02.784549 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:02.784519    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:28:02.784633 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:02.784600    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:28:02.784691 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:02.784650    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:28:02.897236 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:02.897202    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"]
Nov 03 17:28:02.901236 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:02.901212    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:02.956387 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:02.955952    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"]
Nov 03 17:28:02.962052 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:02.961904    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-687fb3\"/\"openshift-service-ca.crt\""
Nov 03 17:28:02.962247 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:02.962079    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-687fb3\"/\"kube-root-ca.crt\""
Nov 03 17:28:03.004685 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.004609    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-687fb3\"/\"simulator-w0-687fb3-kserve-self-signed-certs\""
Nov 03 17:28:03.004685 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.004616    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-687fb3\"/\"default-dockercfg-l8q7m\""
Nov 03 17:28:03.017878 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.017846    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:03.027394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.027244    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-dshm\") pod \"simulator-w0-687fb3-kserve-5999dd6c95-ftv7v\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") " pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:03.027394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.027316    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twbx8\" (UniqueName: \"kubernetes.io/projected/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-kube-api-access-twbx8\") pod \"simulator-w0-687fb3-kserve-5999dd6c95-ftv7v\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") " pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:03.027572 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.027427    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-tmp-dir\") pod \"simulator-w0-687fb3-kserve-5999dd6c95-ftv7v\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") " pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:03.027572 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.027477    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-model-cache\") pod \"simulator-w0-687fb3-kserve-5999dd6c95-ftv7v\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") " pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:03.027572 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.027538    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-tls-certs\") pod \"simulator-w0-687fb3-kserve-5999dd6c95-ftv7v\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") " pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:03.027663 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.027577    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-home\") pod \"simulator-w0-687fb3-kserve-5999dd6c95-ftv7v\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") " pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:03.118086 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.118008    6059 patch_prober.go:29] interesting pod/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 17:28:03.118086 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.118071    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs" podUID="9a6b46be-ed11-4708-b1a9-4500870e4d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 17:28:03.128447 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.128424    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-tmp-dir\") pod \"simulator-w0-687fb3-kserve-5999dd6c95-ftv7v\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") " pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:03.128556 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.128460    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-model-cache\") pod \"simulator-w0-687fb3-kserve-5999dd6c95-ftv7v\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") " pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:03.128556 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.128492    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-tls-certs\") pod \"simulator-w0-687fb3-kserve-5999dd6c95-ftv7v\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") " pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:03.128556 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.128514    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-home\") pod \"simulator-w0-687fb3-kserve-5999dd6c95-ftv7v\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") " pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:03.128718 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.128579    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-dshm\") pod \"simulator-w0-687fb3-kserve-5999dd6c95-ftv7v\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") " pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:03.128718 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.128608    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twbx8\" (UniqueName: \"kubernetes.io/projected/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-kube-api-access-twbx8\") pod \"simulator-w0-687fb3-kserve-5999dd6c95-ftv7v\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") " pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:03.128826 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.128797    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-tmp-dir\") pod \"simulator-w0-687fb3-kserve-5999dd6c95-ftv7v\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") " pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:03.128931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.128910    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-model-cache\") pod \"simulator-w0-687fb3-kserve-5999dd6c95-ftv7v\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") " pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:03.129049 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.129025    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-home\") pod \"simulator-w0-687fb3-kserve-5999dd6c95-ftv7v\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") " pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:03.157380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.156785    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-tls-certs\") pod \"simulator-w0-687fb3-kserve-5999dd6c95-ftv7v\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") " pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:03.159436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.158417    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-dshm\") pod \"simulator-w0-687fb3-kserve-5999dd6c95-ftv7v\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") " pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:03.193140 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.193113    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twbx8\" (UniqueName: \"kubernetes.io/projected/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-kube-api-access-twbx8\") pod \"simulator-w0-687fb3-kserve-5999dd6c95-ftv7v\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") " pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:03.215405 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.215374    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:03.662438 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.643008    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"]
Nov 03 17:28:03.791065 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.791028    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v" event={"ID":"2bbc9082-b2da-4198-8a8a-b6ab26b5011e","Type":"ContainerStarted","Data":"691fb0173b09fcdf872c09c92d421e120068248534d74fda34fac94dba387815"}
Nov 03 17:28:03.792944 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.792910    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5" event={"ID":"63c795d8-dfff-43d8-88a2-de65cc01a51f","Type":"ContainerStarted","Data":"15526d5aeba76b625d24ca1f4d29cdc47da676cf93b15405e21565e458e6e775"}
Nov 03 17:28:03.794432 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:03.794402    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v" event={"ID":"dcef9d95-37e7-4d72-bd24-562f646c2ec9","Type":"ContainerStarted","Data":"6c5c764eb473fbfa14d778d3a88446ee8cde0dc03a6b239680826399c41d9ce2"}
Nov 03 17:28:04.057384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:04.056762    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v" podStartSLOduration=2.247526785 podStartE2EDuration="11.056728479s" totalImagesPullingTime="8.809201694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:28:04.054888904 +0000 UTC m=+624.852601687" watchObservedRunningTime="2026-11-03 17:28:04.056728479 +0000 UTC m=+624.854441260"
Nov 03 17:28:04.118204 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:04.118163    6059 patch_prober.go:29] interesting pod/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 17:28:04.118393 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:04.118226    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs" podUID="9a6b46be-ed11-4708-b1a9-4500870e4d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 17:28:04.522287 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:04.522255    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:28:04.522584 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:04.522561    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 17:28:04.522642 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:04.522623    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5" podUID="63c795d8-dfff-43d8-88a2-de65cc01a51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 17:28:04.801054 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:04.801015    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v" event={"ID":"2bbc9082-b2da-4198-8a8a-b6ab26b5011e","Type":"ContainerStarted","Data":"865b337eb340e3f06fc26671774e0097dd03638c98f2892b7b2d6fc6d36543b5"}
Nov 03 17:28:04.802798 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:04.802773    6059 generic.go:356] "Generic (PLEG): container finished" podID="dcef9d95-37e7-4d72-bd24-562f646c2ec9" containerID="6c5c764eb473fbfa14d778d3a88446ee8cde0dc03a6b239680826399c41d9ce2" exitCode=0
Nov 03 17:28:04.803699 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:04.803671    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v" event={"ID":"dcef9d95-37e7-4d72-bd24-562f646c2ec9","Type":"ContainerDied","Data":"6c5c764eb473fbfa14d778d3a88446ee8cde0dc03a6b239680826399c41d9ce2"}
Nov 03 17:28:04.899097 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:04.899066    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:05.094924 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:05.090883    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v" podStartSLOduration=3.090858718 podStartE2EDuration="3.090858718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:28:04.963623326 +0000 UTC m=+625.761336132" watchObservedRunningTime="2026-11-03 17:28:05.090858718 +0000 UTC m=+625.888571500"
Nov 03 17:28:05.117839 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:05.117809    6059 patch_prober.go:29] interesting pod/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 17:28:05.118020 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:05.117861    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs" podUID="9a6b46be-ed11-4708-b1a9-4500870e4d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 17:28:05.215757 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:05.215721    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:05.522325 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:05.522290    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 17:28:05.522514 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:05.522381    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5" podUID="63c795d8-dfff-43d8-88a2-de65cc01a51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 17:28:06.093534 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.093460    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v"
Nov 03 17:28:06.117883 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.117852    6059 patch_prober.go:29] interesting pod/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 17:28:06.118014 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.117902    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs" podUID="9a6b46be-ed11-4708-b1a9-4500870e4d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 17:28:06.169439 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.169249    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:06.207972 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.207934    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcef9d95-37e7-4d72-bd24-562f646c2ec9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcef9d95-37e7-4d72-bd24-562f646c2ec9-maas-api-openshift-ca\") pod \"dcef9d95-37e7-4d72-bd24-562f646c2ec9\" (UID: \"dcef9d95-37e7-4d72-bd24-562f646c2ec9\") "
Nov 03 17:28:06.208148 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.208041    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcef9d95-37e7-4d72-bd24-562f646c2ec9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcef9d95-37e7-4d72-bd24-562f646c2ec9-maas-api-xks-ca\") pod \"dcef9d95-37e7-4d72-bd24-562f646c2ec9\" (UID: \"dcef9d95-37e7-4d72-bd24-562f646c2ec9\") "
Nov 03 17:28:06.208148 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.208114    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcef9d95-37e7-4d72-bd24-562f646c2ec9-kube-api-access-hzwzc\" (UniqueName: \"kubernetes.io/projected/dcef9d95-37e7-4d72-bd24-562f646c2ec9-kube-api-access-hzwzc\") pod \"dcef9d95-37e7-4d72-bd24-562f646c2ec9\" (UID: \"dcef9d95-37e7-4d72-bd24-562f646c2ec9\") "
Nov 03 17:28:06.208766 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.208738    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcef9d95-37e7-4d72-bd24-562f646c2ec9-maas-api-openshift-ca" pod "dcef9d95-37e7-4d72-bd24-562f646c2ec9" (UID: "dcef9d95-37e7-4d72-bd24-562f646c2ec9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:28:06.210638 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.210610    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcef9d95-37e7-4d72-bd24-562f646c2ec9-maas-api-xks-ca" pod "dcef9d95-37e7-4d72-bd24-562f646c2ec9" (UID: "dcef9d95-37e7-4d72-bd24-562f646c2ec9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:28:06.210733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.210692    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcef9d95-37e7-4d72-bd24-562f646c2ec9-kube-api-access-hzwzc" pod "dcef9d95-37e7-4d72-bd24-562f646c2ec9" (UID: "dcef9d95-37e7-4d72-bd24-562f646c2ec9"). InnerVolumeSpecName "kube-api-access-hzwzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:28:06.309185 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.309146    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcef9d95-37e7-4d72-bd24-562f646c2ec9-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:28:06.309185 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.309180    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzwzc\" (UniqueName: \"kubernetes.io/projected/dcef9d95-37e7-4d72-bd24-562f646c2ec9-kube-api-access-hzwzc\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:28:06.309185 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.309193    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcef9d95-37e7-4d72-bd24-562f646c2ec9-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:28:06.522731 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.522689    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 17:28:06.522908 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.522744    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5" podUID="63c795d8-dfff-43d8-88a2-de65cc01a51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 17:28:06.719437 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.719403    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"]
Nov 03 17:28:06.719948 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.719929    6059 cpu_manager.go:414] "containerMap: removing container" podUID="dcef9d95-37e7-4d72-bd24-562f646c2ec9" containerName="revoke-keys"
Nov 03 17:28:06.720038 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.719950    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcef9d95-37e7-4d72-bd24-562f646c2ec9" containerName="revoke-keys"
Nov 03 17:28:06.720121 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.720107    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcef9d95-37e7-4d72-bd24-562f646c2ec9" containerName="revoke-keys"
Nov 03 17:28:06.723961 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.723941    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:06.794427 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.793209    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-3e5c4d-openshift-default-dockercfg-q4s2f\""
Nov 03 17:28:06.819170 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.819124    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v"
Nov 03 17:28:06.819589 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.819560    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v" event={"ID":"dcef9d95-37e7-4d72-bd24-562f646c2ec9","Type":"ContainerDied","Data":"f6501231ab6a54103d9e4622740be9184930d321e666ca0a46bc0b453f4183e3"}
Nov 03 17:28:06.819772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.819756    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6501231ab6a54103d9e4622740be9184930d321e666ca0a46bc0b453f4183e3"
Nov 03 17:28:06.858420 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.850605    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:06.917377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.916394    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"]
Nov 03 17:28:06.917377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.917095    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzs5h\" (UniqueName: \"kubernetes.io/projected/20fc6389-2903-456b-89f3-f2dcda8db98a-kube-api-access-wzs5h\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:06.917377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.917156    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-workload-socket\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:06.917377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.917213    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-data\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:06.917377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.917257    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20fc6389-2903-456b-89f3-f2dcda8db98a-istiod-ca-cert\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:06.917377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.917293    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-envoy\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:06.917377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.917386    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-credential-socket\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:06.917857 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.917456    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-token\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:06.917857 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.917527    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-podinfo\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:06.917857 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:06.917588    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-workload-certs\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:06.985623 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:28:06.985584    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:28:07.018769 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.018730    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-podinfo\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:07.018931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.018788    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-workload-certs\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:07.018931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.018828    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzs5h\" (UniqueName: \"kubernetes.io/projected/20fc6389-2903-456b-89f3-f2dcda8db98a-kube-api-access-wzs5h\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:07.018931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.018854    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-workload-socket\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:07.018931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.018904    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-data\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:07.018931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.018930    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20fc6389-2903-456b-89f3-f2dcda8db98a-istiod-ca-cert\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:07.019094 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.018952    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-envoy\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:07.019094 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.019018    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-credential-socket\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:07.019094 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.019070    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-token\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:07.019854 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.019833    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-data\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:07.020146 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.020122    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-workload-socket\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:07.020445 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.020424    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-workload-certs\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:07.020582 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.020558    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-credential-socket\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:07.021191 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.021170    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20fc6389-2903-456b-89f3-f2dcda8db98a-istiod-ca-cert\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:07.022242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.022208    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-envoy\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:07.023875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.023855    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-podinfo\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:07.093632 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.093545    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-token\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:07.093974 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.093956    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzs5h\" (UniqueName: \"kubernetes.io/projected/20fc6389-2903-456b-89f3-f2dcda8db98a-kube-api-access-wzs5h\") pod \"e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") " pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:07.118067 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.118034    6059 patch_prober.go:29] interesting pod/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 17:28:07.118231 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.118092    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs" podUID="9a6b46be-ed11-4708-b1a9-4500870e4d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 17:28:07.121110 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.121087    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:07.376881 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.376780    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:07.522844 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.522804    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 17:28:07.523042 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.522866    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5" podUID="63c795d8-dfff-43d8-88a2-de65cc01a51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 17:28:07.575122 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.574954    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:07.821975 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:28:07.821945    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20fc6389_2903_456b_89f3_f2dcda8db98a.slice/crio-c2005e722da505ca89da4fde2fa0a17e81a71efb5a85c391ae44a5ef6ed10fdc WatchSource:0}: Error finding container c2005e722da505ca89da4fde2fa0a17e81a71efb5a85c391ae44a5ef6ed10fdc: Status 404 returned error can't find the container with id c2005e722da505ca89da4fde2fa0a17e81a71efb5a85c391ae44a5ef6ed10fdc
Nov 03 17:28:07.824530 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.824503    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"]
Nov 03 17:28:07.824825 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.824795    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:28:07.824878 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.824858    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:28:07.824913 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:07.824886    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:28:08.127670 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:08.126717    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:28:08.127670 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:08.127442    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:28:08.128416 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:08.128396    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:28:08.144018 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:08.143922    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:08.523182 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:08.523147    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 17:28:08.523409 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:08.523212    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5" podUID="63c795d8-dfff-43d8-88a2-de65cc01a51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 17:28:08.763545 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:08.763479    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:08.857499 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:08.857416    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2" event={"ID":"20fc6389-2903-456b-89f3-f2dcda8db98a","Type":"ContainerStarted","Data":"a4a3b867238c8ade5daed58061b7b4b9681aef9ad064f4a2ffd73d925efd6b24"}
Nov 03 17:28:08.857499 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:08.857478    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2" event={"ID":"20fc6389-2903-456b-89f3-f2dcda8db98a","Type":"ContainerStarted","Data":"c2005e722da505ca89da4fde2fa0a17e81a71efb5a85c391ae44a5ef6ed10fdc"}
Nov 03 17:28:08.972761 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:08.972725    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:08.998584 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:08.998522    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2" podStartSLOduration=2.9985030310000003 podStartE2EDuration="2.998503031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:28:08.997141213 +0000 UTC m=+629.794854015" watchObservedRunningTime="2026-11-03 17:28:08.998503031 +0000 UTC m=+629.796215813"
Nov 03 17:28:09.377321 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:09.377278    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:09.382679 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:09.382653    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:09.526423 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:09.526387    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:28:09.526800 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:09.526778    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:28:09.527876 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:09.527854    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:28:09.862753 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:09.862718    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:09.863689 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:09.863668    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:11.930468 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:11.930427    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:12.152091 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:12.152057    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:12.654429 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:12.654343    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:13.215526 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:13.215491    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:13.228137 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:13.228113    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:13.914433 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:13.914397    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:28:14.354645 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.354601    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"]
Nov 03 17:28:14.361285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.361261    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.423868 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.423837    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.424047 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.423883    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.424047 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.423909    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.424126 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.424040    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flsls\" (UniqueName: \"kubernetes.io/projected/04181a54-5cd2-4dee-b102-5955e3e9b5a3-kube-api-access-flsls\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.424126 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.424081    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.424225 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.424203    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.424283 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.424265    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.424381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.424369    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.424430 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.424395    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.469809 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.469767    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"]
Nov 03 17:28:14.525036 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.524998    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.525237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.525047    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.525237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.525071    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.525237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.525096    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flsls\" (UniqueName: \"kubernetes.io/projected/04181a54-5cd2-4dee-b102-5955e3e9b5a3-kube-api-access-flsls\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.525237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.525123    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.525237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.525174    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.525237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.525217    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.525688 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.525295    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.525688 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.525330    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.525688 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.525588    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.525898 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.525870    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.525965 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.525913    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.526158 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.526137    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.526158 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.526147    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.527527 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.527495    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.527737 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.527721    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.590080 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.590042    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.590953 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.590928    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flsls\" (UniqueName: \"kubernetes.io/projected/04181a54-5cd2-4dee-b102-5955e3e9b5a3-kube-api-access-flsls\") pod \"maas-default-gateway-openshift-default-58c6585447-s48fw\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:14.675915 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:14.675825    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:15.115464 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:28:15.115434    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04181a54_5cd2_4dee_b102_5955e3e9b5a3.slice/crio-343b2a071abb9bc4fb53c60c3d683865a2ed3ff77ef3c9727f3f78d0ff177dfc WatchSource:0}: Error finding container 343b2a071abb9bc4fb53c60c3d683865a2ed3ff77ef3c9727f3f78d0ff177dfc: Status 404 returned error can't find the container with id 343b2a071abb9bc4fb53c60c3d683865a2ed3ff77ef3c9727f3f78d0ff177dfc
Nov 03 17:28:15.117844 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.117824    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"]
Nov 03 17:28:15.118114 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.118086    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:28:15.118164 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.118151    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:28:15.118224 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.118211    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:28:15.151787 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.151759    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:15.366583 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.366463    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:15.569171 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.569138    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr"]
Nov 03 17:28:15.573241 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.573222    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr"
Nov 03 17:28:15.637520 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.637434    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-3e5c4d\""
Nov 03 17:28:15.639580 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.639551    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-3e5c4d-dockercfg-srzwt\""
Nov 03 17:28:15.641949 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.641925    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr"]
Nov 03 17:28:15.650380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.650090    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a517f58a-ccdb-4629-bcd4-5aa9f7980cf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr\" (UID: \"a517f58a-ccdb-4629-bcd4-5aa9f7980cf3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr"
Nov 03 17:28:15.650380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.650157    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsvg2\" (UniqueName: \"kubernetes.io/projected/a517f58a-ccdb-4629-bcd4-5aa9f7980cf3-kube-api-access-wsvg2\") pod \"payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr\" (UID: \"a517f58a-ccdb-4629-bcd4-5aa9f7980cf3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr"
Nov 03 17:28:15.726569 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.726514    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6"]
Nov 03 17:28:15.737381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.734760    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6"
Nov 03 17:28:15.759611 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.751964    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a517f58a-ccdb-4629-bcd4-5aa9f7980cf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr\" (UID: \"a517f58a-ccdb-4629-bcd4-5aa9f7980cf3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr"
Nov 03 17:28:15.759611 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.752035    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsvg2\" (UniqueName: \"kubernetes.io/projected/a517f58a-ccdb-4629-bcd4-5aa9f7980cf3-kube-api-access-wsvg2\") pod \"payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr\" (UID: \"a517f58a-ccdb-4629-bcd4-5aa9f7980cf3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr"
Nov 03 17:28:15.773099 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.773054    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a517f58a-ccdb-4629-bcd4-5aa9f7980cf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr\" (UID: \"a517f58a-ccdb-4629-bcd4-5aa9f7980cf3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr"
Nov 03 17:28:15.788557 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.788518    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6"]
Nov 03 17:28:15.824785 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.824758    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsvg2\" (UniqueName: \"kubernetes.io/projected/a517f58a-ccdb-4629-bcd4-5aa9f7980cf3-kube-api-access-wsvg2\") pod \"payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr\" (UID: \"a517f58a-ccdb-4629-bcd4-5aa9f7980cf3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr"
Nov 03 17:28:15.855544 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.855248    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b201da2-6bd6-46c9-917b-03ab1da73865-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6\" (UID: \"0b201da2-6bd6-46c9-917b-03ab1da73865\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6"
Nov 03 17:28:15.855544 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.855389    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xffr2\" (UniqueName: \"kubernetes.io/projected/0b201da2-6bd6-46c9-917b-03ab1da73865-kube-api-access-xffr2\") pod \"payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6\" (UID: \"0b201da2-6bd6-46c9-917b-03ab1da73865\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6"
Nov 03 17:28:15.887811 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.887726    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr"
Nov 03 17:28:15.917262 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.917226    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw" event={"ID":"04181a54-5cd2-4dee-b102-5955e3e9b5a3","Type":"ContainerStarted","Data":"2a26b9b5de878006e2c7dbcec5a94d728792d35258240e7a9d3ff4c5c076287d"}
Nov 03 17:28:15.917404 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.917323    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw" event={"ID":"04181a54-5cd2-4dee-b102-5955e3e9b5a3","Type":"ContainerStarted","Data":"343b2a071abb9bc4fb53c60c3d683865a2ed3ff77ef3c9727f3f78d0ff177dfc"}
Nov 03 17:28:15.956098 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.956047    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b201da2-6bd6-46c9-917b-03ab1da73865-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6\" (UID: \"0b201da2-6bd6-46c9-917b-03ab1da73865\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6"
Nov 03 17:28:15.956303 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.956125    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xffr2\" (UniqueName: \"kubernetes.io/projected/0b201da2-6bd6-46c9-917b-03ab1da73865-kube-api-access-xffr2\") pod \"payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6\" (UID: \"0b201da2-6bd6-46c9-917b-03ab1da73865\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6"
Nov 03 17:28:15.957872 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:15.957836    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b201da2-6bd6-46c9-917b-03ab1da73865-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6\" (UID: \"0b201da2-6bd6-46c9-917b-03ab1da73865\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6"
Nov 03 17:28:16.020450 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:16.020414    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xffr2\" (UniqueName: \"kubernetes.io/projected/0b201da2-6bd6-46c9-917b-03ab1da73865-kube-api-access-xffr2\") pod \"payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6\" (UID: \"0b201da2-6bd6-46c9-917b-03ab1da73865\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6"
Nov 03 17:28:16.069303 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:16.069252    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw" podStartSLOduration=2.069237336 podStartE2EDuration="2.069237336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:28:16.06857328 +0000 UTC m=+636.866286057" watchObservedRunningTime="2026-11-03 17:28:16.069237336 +0000 UTC m=+636.866950117"
Nov 03 17:28:16.075959 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:16.075929    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6"
Nov 03 17:28:16.287557 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:28:16.287522    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda517f58a_ccdb_4629_bcd4_5aa9f7980cf3.slice/crio-03811d4c9bb654595a61607d8650fd77ad2b335f440fcb53338f8264bfd90bd3 WatchSource:0}: Error finding container 03811d4c9bb654595a61607d8650fd77ad2b335f440fcb53338f8264bfd90bd3: Status 404 returned error can't find the container with id 03811d4c9bb654595a61607d8650fd77ad2b335f440fcb53338f8264bfd90bd3
Nov 03 17:28:16.289782 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:16.289756    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr"]
Nov 03 17:28:16.469848 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:28:16.469811    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b201da2_6bd6_46c9_917b_03ab1da73865.slice/crio-b5e58a531cf65855d572d0578b99be3529460e42b278c306d9feccd6d59782ab WatchSource:0}: Error finding container b5e58a531cf65855d572d0578b99be3529460e42b278c306d9feccd6d59782ab: Status 404 returned error can't find the container with id b5e58a531cf65855d572d0578b99be3529460e42b278c306d9feccd6d59782ab
Nov 03 17:28:16.471640 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:16.471612    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6"]
Nov 03 17:28:16.676140 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:16.676104    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:16.677765 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:16.677739    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-s48fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 17:28:16.677870 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:16.677786    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw" podUID="04181a54-5cd2-4dee-b102-5955e3e9b5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 17:28:16.924934 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:16.924829    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6" event={"ID":"0b201da2-6bd6-46c9-917b-03ab1da73865","Type":"ContainerStarted","Data":"e05e24106f7d56d9bc2c219354cea208f2c5119d09fecf04197d8cacc1db17b6"}
Nov 03 17:28:16.924934 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:16.924902    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6" event={"ID":"0b201da2-6bd6-46c9-917b-03ab1da73865","Type":"ContainerStarted","Data":"b5e58a531cf65855d572d0578b99be3529460e42b278c306d9feccd6d59782ab"}
Nov 03 17:28:16.924934 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:16.924931    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6"
Nov 03 17:28:16.926640 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:16.926605    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr" event={"ID":"a517f58a-ccdb-4629-bcd4-5aa9f7980cf3","Type":"ContainerStarted","Data":"daf57a41024abbb62a680985d5f05c419bfac1477e147e7cf7631e7f86c9987a"}
Nov 03 17:28:16.926768 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:16.926659    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr" event={"ID":"a517f58a-ccdb-4629-bcd4-5aa9f7980cf3","Type":"ContainerStarted","Data":"03811d4c9bb654595a61607d8650fd77ad2b335f440fcb53338f8264bfd90bd3"}
Nov 03 17:28:17.048067 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:17.047999    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6" podStartSLOduration=2.047979499 podStartE2EDuration="2.047979499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:28:17.045229718 +0000 UTC m=+637.842942511" watchObservedRunningTime="2026-11-03 17:28:17.047979499 +0000 UTC m=+637.845692281"
Nov 03 17:28:17.168249 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:17.168184    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr" podStartSLOduration=2.168165631 podStartE2EDuration="2.168165631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:28:17.166755331 +0000 UTC m=+637.964468116" watchObservedRunningTime="2026-11-03 17:28:17.168165631 +0000 UTC m=+637.965878412"
Nov 03 17:28:17.676489 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:17.676407    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-s48fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 17:28:17.676489 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:17.676467    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw" podUID="04181a54-5cd2-4dee-b102-5955e3e9b5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 17:28:17.771651 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:17.771619    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:17.955515 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:17.955413    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr"
Nov 03 17:28:18.676875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:18.676837    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-s48fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 17:28:18.677280 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:18.676898    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw" podUID="04181a54-5cd2-4dee-b102-5955e3e9b5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 17:28:19.676272 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:19.676208    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-s48fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 17:28:19.676516 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:19.676273    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw" podUID="04181a54-5cd2-4dee-b102-5955e3e9b5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 17:28:20.676467 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:20.676429    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-s48fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 17:28:20.676857 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:20.676486    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw" podUID="04181a54-5cd2-4dee-b102-5955e3e9b5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 17:28:21.279742 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:21.279704    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:21.676460 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:21.676352    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-s48fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 17:28:21.676460 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:21.676432    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw" podUID="04181a54-5cd2-4dee-b102-5955e3e9b5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 17:28:22.330701 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:22.330667    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:22.519903 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:22.519865    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:22.676649 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:22.676561    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-s48fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 17:28:22.677001 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:22.676641    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw" podUID="04181a54-5cd2-4dee-b102-5955e3e9b5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 17:28:23.091329 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:23.091286    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:23.296053 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:23.296015    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:24.574558 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:24.574519    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:24.575392 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:24.575368    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:24.576779 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:24.576751    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:28:26.672985 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:26.672953    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:27.025706 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:27.025675    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:27.401060 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:27.400980    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:27.614085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:27.614052    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:27.958233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:27.958204    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6"
Nov 03 17:28:28.963966 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:28.963938    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr"
Nov 03 17:28:30.559246 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:30.559215    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:30.745713 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:30.745681    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:31.119693 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:31.119660    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg"]
Nov 03 17:28:31.124193 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:31.124169    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg"
Nov 03 17:28:31.186689 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:31.186652    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-af9e3c\""
Nov 03 17:28:31.238466 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:31.238428    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-af9e3c-dockercfg-nhn8k\""
Nov 03 17:28:31.242243 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:31.242216    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg"]
Nov 03 17:28:31.253984 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:31.253951    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26fpv\" (UniqueName: \"kubernetes.io/projected/6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e-kube-api-access-26fpv\") pod \"payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg\" (UID: \"6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg"
Nov 03 17:28:31.254113 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:31.254016    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg\" (UID: \"6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg"
Nov 03 17:28:31.355831 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:31.355791    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26fpv\" (UniqueName: \"kubernetes.io/projected/6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e-kube-api-access-26fpv\") pod \"payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg\" (UID: \"6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg"
Nov 03 17:28:31.356006 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:31.355858    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg\" (UID: \"6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg"
Nov 03 17:28:31.356651 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:31.356623    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg\" (UID: \"6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg"
Nov 03 17:28:31.420903 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:31.420824    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26fpv\" (UniqueName: \"kubernetes.io/projected/6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e-kube-api-access-26fpv\") pod \"payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg\" (UID: \"6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg"
Nov 03 17:28:31.444568 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:31.444535    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg"
Nov 03 17:28:31.823472 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:31.823410    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg"]
Nov 03 17:28:31.985048 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:31.985012    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:32.057193 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:32.057156    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg" event={"ID":"6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e","Type":"ContainerStarted","Data":"ffe0b5f125351f2af54e830a750aa4486b02f13049b132dfbb9db513b0eeb8b2"}
Nov 03 17:28:32.057345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:32.057215    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg" event={"ID":"6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e","Type":"ContainerStarted","Data":"c6e7de8b8000e53d632f9319e297256cdcd48b9b4e039e68f72a4e338ae9bced"}
Nov 03 17:28:32.057419 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:32.057388    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg"
Nov 03 17:28:32.177452 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:32.177280    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg" podStartSLOduration=1.177265031 podStartE2EDuration="1.177265031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:28:32.176927526 +0000 UTC m=+652.974640308" watchObservedRunningTime="2026-11-03 17:28:32.177265031 +0000 UTC m=+652.974977811"
Nov 03 17:28:32.184553 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:32.183677    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:32.377686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:32.377651    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:33.786025 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:33.785995    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:36.863744 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:28:36.863704    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:28:36.896370 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:36.896321    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:41.572379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:41.556754    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:43.065079 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:43.065051    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg"
Nov 03 17:28:43.892823 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:43.892787    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:46.044071 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:46.044032    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:50.189716 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:50.189680    6059 generic.go:356] "Generic (PLEG): container finished" podID="20fc6389-2903-456b-89f3-f2dcda8db98a" containerID="a4a3b867238c8ade5daed58061b7b4b9681aef9ad064f4a2ffd73d925efd6b24" exitCode=0
Nov 03 17:28:50.190262 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:50.189749    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2" event={"ID":"20fc6389-2903-456b-89f3-f2dcda8db98a","Type":"ContainerDied","Data":"a4a3b867238c8ade5daed58061b7b4b9681aef9ad064f4a2ffd73d925efd6b24"}
Nov 03 17:28:50.190262 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:50.190158    6059 scope.go:122] "RemoveContainer" containerID="a4a3b867238c8ade5daed58061b7b4b9681aef9ad064f4a2ffd73d925efd6b24"
Nov 03 17:28:50.191859 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:50.191823    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:28:50.191988 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:50.191920    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:28:50.191988 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:50.191974    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:28:50.481072 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:50.481031    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:50.673911 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:50.673882    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:50.855974 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:50.855887    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:51.197529 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:51.197445    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2" event={"ID":"20fc6389-2903-456b-89f3-f2dcda8db98a","Type":"ContainerStarted","Data":"165e105b5e8720fbf4d53e9aedd7708c731c59f53b6ae412b8e649dcce92eae0"}
Nov 03 17:28:51.377837 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:51.377801    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:51.378964 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:51.378937    6059 patch_prober.go:29] interesting pod/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 17:28:51.379079 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:51.378991    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2" podUID="20fc6389-2903-456b-89f3-f2dcda8db98a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 17:28:52.377868 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:52.377828    6059 patch_prober.go:29] interesting pod/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 17:28:52.378393 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:52.377887    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2" podUID="20fc6389-2903-456b-89f3-f2dcda8db98a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 17:28:53.377451 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:53.377412    6059 patch_prober.go:29] interesting pod/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 17:28:53.377634 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:53.377470    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2" podUID="20fc6389-2903-456b-89f3-f2dcda8db98a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 17:28:54.377739 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:54.377708    6059 patch_prober.go:29] interesting pod/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 17:28:54.378121 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:54.377766    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2" podUID="20fc6389-2903-456b-89f3-f2dcda8db98a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 17:28:54.863347 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:54.863295    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:55.188039 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:55.187961    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:55.378022 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:55.377990    6059 patch_prober.go:29] interesting pod/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 17:28:55.378422 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:55.378046    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2" podUID="20fc6389-2903-456b-89f3-f2dcda8db98a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 17:28:55.381192 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:55.381161    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:28:56.386772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:56.386746    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:56.388629 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:56.388609    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:28:59.957233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:28:59.957199    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:04.520465 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:04.520426    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:06.859759 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:29:06.859724    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:29:07.922345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:07.922299    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:08.608114 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:08.608078    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:09.571060 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:09.571016    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:14.357557 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:14.357520    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:19.126123 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:19.126052    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:22.821142 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:22.821098    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"]
Nov 03 17:29:22.824756 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:22.824736    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:22.884176 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:22.884136    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-8fc5ec\"/\"openshift-service-ca.crt\""
Nov 03 17:29:22.884458 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:22.884430    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-8fc5ec\"/\"kube-root-ca.crt\""
Nov 03 17:29:22.918054 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:22.918020    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vppst\" (UniqueName: \"kubernetes.io/projected/38247383-766c-4605-a65b-90a7c5763344-kube-api-access-vppst\") pod \"praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") " pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:22.918236 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:22.918069    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38247383-766c-4605-a65b-90a7c5763344-tls-certs\") pod \"praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") " pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:22.918236 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:22.918098    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-dshm\") pod \"praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") " pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:22.918382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:22.918259    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-model-cache\") pod \"praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") " pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:22.918428 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:22.918410    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-tmp-dir\") pod \"praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") " pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:22.918481 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:22.918464    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-home\") pod \"praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") " pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:22.936024 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:22.935995    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-8fc5ec\"/\"default-dockercfg-v8nxq\""
Nov 03 17:29:22.936156 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:22.936103    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-8fc5ec\"/\"praxis-route-8fc5ec-kserve-self-signed-certs\""
Nov 03 17:29:22.938707 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:22.938681    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"]
Nov 03 17:29:23.011337 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:23.011299    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:23.030039 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:23.025011    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vppst\" (UniqueName: \"kubernetes.io/projected/38247383-766c-4605-a65b-90a7c5763344-kube-api-access-vppst\") pod \"praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") " pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:23.030881 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:23.030372    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38247383-766c-4605-a65b-90a7c5763344-tls-certs\") pod \"praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") " pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:23.031118 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:23.031099    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-dshm\") pod \"praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") " pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:23.031235 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:23.031222    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-model-cache\") pod \"praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") " pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:23.031564 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:23.031541    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-tmp-dir\") pod \"praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") " pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:23.031803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:23.031779    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-home\") pod \"praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") " pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:23.032538 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:23.032518    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-home\") pod \"praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") " pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:23.034866 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:23.034837    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-model-cache\") pod \"praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") " pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:23.035178 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:23.035142    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-tmp-dir\") pod \"praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") " pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:23.036745 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:23.036710    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38247383-766c-4605-a65b-90a7c5763344-tls-certs\") pod \"praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") " pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:23.037763 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:23.037743    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-dshm\") pod \"praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") " pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:23.094344 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:23.094255    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vppst\" (UniqueName: \"kubernetes.io/projected/38247383-766c-4605-a65b-90a7c5763344-kube-api-access-vppst\") pod \"praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") " pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:23.136928 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:23.136891    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:23.513592 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:29:23.513558    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38247383_766c_4605_a65b_90a7c5763344.slice/crio-0dc708a2b3d1052ae96b06b8bf9dc870b70560c6a2df2f804d3f3c0c139ca906 WatchSource:0}: Error finding container 0dc708a2b3d1052ae96b06b8bf9dc870b70560c6a2df2f804d3f3c0c139ca906: Status 404 returned error can't find the container with id 0dc708a2b3d1052ae96b06b8bf9dc870b70560c6a2df2f804d3f3c0c139ca906
Nov 03 17:29:23.516640 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:23.516615    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"]
Nov 03 17:29:23.784342 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:23.784233    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:24.395137 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:24.395101    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w" event={"ID":"38247383-766c-4605-a65b-90a7c5763344","Type":"ContainerStarted","Data":"bc8fb24883787356609ed26f480f5fe54402d96d4cee839f7b5603aaa7caa6f9"}
Nov 03 17:29:24.395564 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:24.395148    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w" event={"ID":"38247383-766c-4605-a65b-90a7c5763344","Type":"ContainerStarted","Data":"0dc708a2b3d1052ae96b06b8bf9dc870b70560c6a2df2f804d3f3c0c139ca906"}
Nov 03 17:29:24.520224 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:24.520172    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w" podStartSLOduration=2.520158238 podStartE2EDuration="2.520158238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:29:24.519441572 +0000 UTC m=+705.317154354" watchObservedRunningTime="2026-11-03 17:29:24.520158238 +0000 UTC m=+705.317871019"
Nov 03 17:29:25.137295 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:25.137254    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:28.560049 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:28.560007    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:28.777793 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:28.777761    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:29.121181 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.121144    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:29.761141 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.761105    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"]
Nov 03 17:29:29.766163 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.766135    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.814039 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.814008    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-podinfo\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.814215 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.814054    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-token\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.814215 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.814096    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l87ll\" (UniqueName: \"kubernetes.io/projected/012064a7-fd12-465b-bdc8-ef6d4ca289b8-kube-api-access-l87ll\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.814294 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.814223    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-credential-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.814294 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.814266    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-data\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.814384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.814327    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-envoy\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.814384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.814349    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-workload-certs\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.814479 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.814393    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-workload-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.814479 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.814440    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istiod-ca-cert\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.862428 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.862389    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"]
Nov 03 17:29:29.893032 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.892999    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:29.914513 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.914480    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-147834-openshift-default-dockercfg-2pfbf\""
Nov 03 17:29:29.916044 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.916011    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-podinfo\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.916184 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.916078    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-token\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.916184 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.916172    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l87ll\" (UniqueName: \"kubernetes.io/projected/012064a7-fd12-465b-bdc8-ef6d4ca289b8-kube-api-access-l87ll\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.916268 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.916240    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-credential-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.916304 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.916285    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-data\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.916337 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.916322    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-envoy\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.916411 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.916349    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-workload-certs\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.916463 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.916418    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-workload-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.916463 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.916444    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istiod-ca-cert\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.918652 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.918270    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-workload-certs\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.918652 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.918538    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-data\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.918652 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.918601    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-workload-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.927693 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.921003    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-credential-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.927693 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.921646    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istiod-ca-cert\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.927693 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.924257    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-envoy\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.935452 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.935418    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-podinfo\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.991702 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.991669    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l87ll\" (UniqueName: \"kubernetes.io/projected/012064a7-fd12-465b-bdc8-ef6d4ca289b8-kube-api-access-l87ll\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:29.992195 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:29.992168    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-token\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:30.081852 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:30.081769    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:30.167904 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:30.167861    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:30.520481 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:30.520440    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"]
Nov 03 17:29:30.548697 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:29:30.548659    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012064a7_fd12_465b_bdc8_ef6d4ca289b8.slice/crio-d42499864b4c7fab994e37be32f8fc61f3e83ffe6fe6c7930c29943f273c3fe7 WatchSource:0}: Error finding container d42499864b4c7fab994e37be32f8fc61f3e83ffe6fe6c7930c29943f273c3fe7: Status 404 returned error can't find the container with id d42499864b4c7fab994e37be32f8fc61f3e83ffe6fe6c7930c29943f273c3fe7
Nov 03 17:29:30.551045 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:30.551011    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:29:30.551127 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:30.551082    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:29:30.551127 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:30.551111    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:29:31.439625 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:31.439584    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx" event={"ID":"012064a7-fd12-465b-bdc8-ef6d4ca289b8","Type":"ContainerStarted","Data":"4273dbde4f713e9e9355e86370d12cd49133e36861a93b6300401162988e55ce"}
Nov 03 17:29:31.440090 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:31.439659    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx" event={"ID":"012064a7-fd12-465b-bdc8-ef6d4ca289b8","Type":"ContainerStarted","Data":"d42499864b4c7fab994e37be32f8fc61f3e83ffe6fe6c7930c29943f273c3fe7"}
Nov 03 17:29:31.669483 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:31.669426    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx" podStartSLOduration=2.669412658 podStartE2EDuration="2.669412658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:29:31.617110989 +0000 UTC m=+712.414823770" watchObservedRunningTime="2026-11-03 17:29:31.669412658 +0000 UTC m=+712.467125437"
Nov 03 17:29:32.082969 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:32.082931    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:32.088791 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:32.088761    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:32.445479 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:32.445394    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:32.446478 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:32.446457    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:29:33.137625 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:33.137586    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:33.149982 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:33.149952    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:33.395026 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:33.394936    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:33.462029 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:33.461995    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:29:34.118532 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:34.118502    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:34.501934 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:34.501214    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:34.759733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:34.759640    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:36.862145 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:29:36.862106    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:29:38.522689 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:38.522649    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:41.027741 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:41.027708    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:41.216755 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:41.216716    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:41.408834 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:41.408751    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq"]
Nov 03 17:29:41.412916 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:41.412891    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq"
Nov 03 17:29:41.497435 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:41.494093    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-147834-dockercfg-6967d\""
Nov 03 17:29:41.534681 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:41.534646    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-147834\""
Nov 03 17:29:41.539795 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:41.539754    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq"]
Nov 03 17:29:41.564339 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:41.564290    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7268\" (UniqueName: \"kubernetes.io/projected/d2e5b0a4-8ae5-41d3-97b4-a13804480f77-kube-api-access-s7268\") pod \"payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq\" (UID: \"d2e5b0a4-8ae5-41d3-97b4-a13804480f77\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq"
Nov 03 17:29:41.564571 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:41.564502    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e5b0a4-8ae5-41d3-97b4-a13804480f77-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq\" (UID: \"d2e5b0a4-8ae5-41d3-97b4-a13804480f77\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq"
Nov 03 17:29:41.665776 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:41.665671    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e5b0a4-8ae5-41d3-97b4-a13804480f77-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq\" (UID: \"d2e5b0a4-8ae5-41d3-97b4-a13804480f77\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq"
Nov 03 17:29:41.665957 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:41.665777    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7268\" (UniqueName: \"kubernetes.io/projected/d2e5b0a4-8ae5-41d3-97b4-a13804480f77-kube-api-access-s7268\") pod \"payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq\" (UID: \"d2e5b0a4-8ae5-41d3-97b4-a13804480f77\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq"
Nov 03 17:29:41.666506 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:41.666478    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e5b0a4-8ae5-41d3-97b4-a13804480f77-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq\" (UID: \"d2e5b0a4-8ae5-41d3-97b4-a13804480f77\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq"
Nov 03 17:29:41.728966 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:41.728930    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7268\" (UniqueName: \"kubernetes.io/projected/d2e5b0a4-8ae5-41d3-97b4-a13804480f77-kube-api-access-s7268\") pod \"payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq\" (UID: \"d2e5b0a4-8ae5-41d3-97b4-a13804480f77\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq"
Nov 03 17:29:42.026845 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:42.026802    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq"
Nov 03 17:29:42.405382 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:29:42.405332    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e5b0a4_8ae5_41d3_97b4_a13804480f77.slice/crio-0b08d12e0bf042e14a9d61e93c45f82027d55965c9fb7669d885b2285d95b1c8 WatchSource:0}: Error finding container 0b08d12e0bf042e14a9d61e93c45f82027d55965c9fb7669d885b2285d95b1c8: Status 404 returned error can't find the container with id 0b08d12e0bf042e14a9d61e93c45f82027d55965c9fb7669d885b2285d95b1c8
Nov 03 17:29:42.406986 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:42.406959    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq"]
Nov 03 17:29:42.558725 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:42.558685    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq" event={"ID":"d2e5b0a4-8ae5-41d3-97b4-a13804480f77","Type":"ContainerStarted","Data":"f64ff090587f2e41e5e8b6bd96f46e91dd123487e982b28d20caba6de29ec0d3"}
Nov 03 17:29:42.558874 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:42.558744    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq" event={"ID":"d2e5b0a4-8ae5-41d3-97b4-a13804480f77","Type":"ContainerStarted","Data":"0b08d12e0bf042e14a9d61e93c45f82027d55965c9fb7669d885b2285d95b1c8"}
Nov 03 17:29:42.558874 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:42.558818    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq"
Nov 03 17:29:42.683427 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:42.683298    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq" podStartSLOduration=1.683284729 podStartE2EDuration="1.683284729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:29:42.681857424 +0000 UTC m=+723.479570206" watchObservedRunningTime="2026-11-03 17:29:42.683284729 +0000 UTC m=+723.480997511"
Nov 03 17:29:43.041067 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:43.041033    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:47.693000 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:47.692908    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:49.980455 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:49.980415    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e38015c-hk58z"]
Nov 03 17:29:49.984727 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:49.984703    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e38015c-hk58z"
Nov 03 17:29:50.039458 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:50.039424    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:29:50.067131 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:50.067095    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/320ccb69-f96d-47eb-81b0-e69a5758af8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e38015c-hk58z\" (UID: \"320ccb69-f96d-47eb-81b0-e69a5758af8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e38015c-hk58z"
Nov 03 17:29:50.067323 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:50.067140    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-445ff\" (UniqueName: \"kubernetes.io/projected/320ccb69-f96d-47eb-81b0-e69a5758af8d-kube-api-access-445ff\") pod \"maas-api-revoke-sub-7e38015c-hk58z\" (UID: \"320ccb69-f96d-47eb-81b0-e69a5758af8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e38015c-hk58z"
Nov 03 17:29:50.067402 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:50.067329    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/320ccb69-f96d-47eb-81b0-e69a5758af8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e38015c-hk58z\" (UID: \"320ccb69-f96d-47eb-81b0-e69a5758af8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e38015c-hk58z"
Nov 03 17:29:50.091203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:50.091165    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:29:50.091203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:50.091168    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:29:50.091203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:50.091179    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:29:50.093796 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:50.093767    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e38015c-hk58z"]
Nov 03 17:29:50.168794 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:50.168753    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/320ccb69-f96d-47eb-81b0-e69a5758af8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e38015c-hk58z\" (UID: \"320ccb69-f96d-47eb-81b0-e69a5758af8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e38015c-hk58z"
Nov 03 17:29:50.169012 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:50.168846    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/320ccb69-f96d-47eb-81b0-e69a5758af8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e38015c-hk58z\" (UID: \"320ccb69-f96d-47eb-81b0-e69a5758af8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e38015c-hk58z"
Nov 03 17:29:50.169012 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:50.168875    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-445ff\" (UniqueName: \"kubernetes.io/projected/320ccb69-f96d-47eb-81b0-e69a5758af8d-kube-api-access-445ff\") pod \"maas-api-revoke-sub-7e38015c-hk58z\" (UID: \"320ccb69-f96d-47eb-81b0-e69a5758af8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e38015c-hk58z"
Nov 03 17:29:50.169704 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:50.169681    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/320ccb69-f96d-47eb-81b0-e69a5758af8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e38015c-hk58z\" (UID: \"320ccb69-f96d-47eb-81b0-e69a5758af8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e38015c-hk58z"
Nov 03 17:29:50.171178 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:50.171156    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/320ccb69-f96d-47eb-81b0-e69a5758af8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e38015c-hk58z\" (UID: \"320ccb69-f96d-47eb-81b0-e69a5758af8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e38015c-hk58z"
Nov 03 17:29:50.232125 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:50.232046    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-445ff\" (UniqueName: \"kubernetes.io/projected/320ccb69-f96d-47eb-81b0-e69a5758af8d-kube-api-access-445ff\") pod \"maas-api-revoke-sub-7e38015c-hk58z\" (UID: \"320ccb69-f96d-47eb-81b0-e69a5758af8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e38015c-hk58z"
Nov 03 17:29:50.298674 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:50.298624    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e38015c-hk58z"
Nov 03 17:29:50.681383 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:29:50.681319    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320ccb69_f96d_47eb_81b0_e69a5758af8d.slice/crio-890efc578471fd094d035bfbfc492f2d218ad7b0a3eb97993e17d34168fc23fc WatchSource:0}: Error finding container 890efc578471fd094d035bfbfc492f2d218ad7b0a3eb97993e17d34168fc23fc: Status 404 returned error can't find the container with id 890efc578471fd094d035bfbfc492f2d218ad7b0a3eb97993e17d34168fc23fc
Nov 03 17:29:50.683856 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:50.683826    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e38015c-hk58z"]
Nov 03 17:29:51.141045 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.140996    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:51.613884 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.613846    6059 generic.go:356] "Generic (PLEG): container finished" podID="320ccb69-f96d-47eb-81b0-e69a5758af8d" containerID="8070cc0a1429b5db0bf100c44c450119493315f67e2d853e7f89a2b8c6bb129a" exitCode=0
Nov 03 17:29:51.614058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.613922    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e38015c-hk58z" event={"ID":"320ccb69-f96d-47eb-81b0-e69a5758af8d","Type":"ContainerDied","Data":"8070cc0a1429b5db0bf100c44c450119493315f67e2d853e7f89a2b8c6bb129a"}
Nov 03 17:29:51.614058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.613970    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e38015c-hk58z" event={"ID":"320ccb69-f96d-47eb-81b0-e69a5758af8d","Type":"ContainerStarted","Data":"890efc578471fd094d035bfbfc492f2d218ad7b0a3eb97993e17d34168fc23fc"}
Nov 03 17:29:51.723639 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.723590    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"]
Nov 03 17:29:51.789007 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.767692    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.819695 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.819664    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"]
Nov 03 17:29:51.822675 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.822649    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3f1cf1-openshift-default-dockercfg-zr9s2\""
Nov 03 17:29:51.869687 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.869643    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:51.890020 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.889984    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istiod-ca-cert\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.890198 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.890029    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8zx2\" (UniqueName: \"kubernetes.io/projected/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-kube-api-access-v8zx2\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.890198 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.890059    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-token\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.890311 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.890260    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-podinfo\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.890373 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.890314    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-workload-certs\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.890424 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.890394    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-envoy\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.890491 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.890427    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-credential-socket\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.890491 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.890453    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-workload-socket\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.890597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.890565    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-data\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.991745 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.991706    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-workload-certs\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.991953 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.991770    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-envoy\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.991953 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.991822    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-credential-socket\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.991953 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.991859    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-workload-socket\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.991953 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.991893    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-data\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.992173 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.992016    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istiod-ca-cert\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.992173 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.992057    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8zx2\" (UniqueName: \"kubernetes.io/projected/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-kube-api-access-v8zx2\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.992173 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.992104    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-token\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.992335 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.992312    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-workload-certs\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.992471 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.992441    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-data\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.992660 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.992624    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-credential-socket\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.992861 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.992716    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-workload-socket\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.993061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.993037    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istiod-ca-cert\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.993505 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.993477    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-podinfo\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.994890 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.994864    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-envoy\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:51.996302 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:51.996275    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-podinfo\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:52.057139 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:52.057062    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-token\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:52.057139 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:52.057121    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8zx2\" (UniqueName: \"kubernetes.io/projected/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-kube-api-access-v8zx2\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:52.088118 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:52.088085    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:52.108088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:52.108046    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:52.312427 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:52.312243    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:52.589807 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:29:52.589774    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb7314fe_6d72_4e51_8cdc_acfcce172a1c.slice/crio-7e0d12c6717ebf6d1c7cfa33ac45e02f971ec2bcdb35ade4913457d65eef9554 WatchSource:0}: Error finding container 7e0d12c6717ebf6d1c7cfa33ac45e02f971ec2bcdb35ade4913457d65eef9554: Status 404 returned error can't find the container with id 7e0d12c6717ebf6d1c7cfa33ac45e02f971ec2bcdb35ade4913457d65eef9554
Nov 03 17:29:52.592527 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:52.592492    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:29:52.592631 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:52.592565    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:29:52.592631 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:52.592598    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:29:52.593662 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:52.593638    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"]
Nov 03 17:29:52.621612 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:52.621578    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg" event={"ID":"eb7314fe-6d72-4e51-8cdc-acfcce172a1c","Type":"ContainerStarted","Data":"7e0d12c6717ebf6d1c7cfa33ac45e02f971ec2bcdb35ade4913457d65eef9554"}
Nov 03 17:29:52.779743 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:52.779709    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e38015c-hk58z"
Nov 03 17:29:52.912523 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:52.912446    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/320ccb69-f96d-47eb-81b0-e69a5758af8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/320ccb69-f96d-47eb-81b0-e69a5758af8d-maas-api-xks-ca\") pod \"320ccb69-f96d-47eb-81b0-e69a5758af8d\" (UID: \"320ccb69-f96d-47eb-81b0-e69a5758af8d\") "
Nov 03 17:29:52.912523 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:52.912512    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320ccb69-f96d-47eb-81b0-e69a5758af8d-kube-api-access-445ff\" (UniqueName: \"kubernetes.io/projected/320ccb69-f96d-47eb-81b0-e69a5758af8d-kube-api-access-445ff\") pod \"320ccb69-f96d-47eb-81b0-e69a5758af8d\" (UID: \"320ccb69-f96d-47eb-81b0-e69a5758af8d\") "
Nov 03 17:29:52.912679 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:52.912557    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/320ccb69-f96d-47eb-81b0-e69a5758af8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/320ccb69-f96d-47eb-81b0-e69a5758af8d-maas-api-openshift-ca\") pod \"320ccb69-f96d-47eb-81b0-e69a5758af8d\" (UID: \"320ccb69-f96d-47eb-81b0-e69a5758af8d\") "
Nov 03 17:29:52.913673 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:52.913641    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/320ccb69-f96d-47eb-81b0-e69a5758af8d-maas-api-openshift-ca" pod "320ccb69-f96d-47eb-81b0-e69a5758af8d" (UID: "320ccb69-f96d-47eb-81b0-e69a5758af8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:29:52.914890 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:52.914864    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/320ccb69-f96d-47eb-81b0-e69a5758af8d-maas-api-xks-ca" pod "320ccb69-f96d-47eb-81b0-e69a5758af8d" (UID: "320ccb69-f96d-47eb-81b0-e69a5758af8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:29:52.915186 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:52.915164    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320ccb69-f96d-47eb-81b0-e69a5758af8d-kube-api-access-445ff" pod "320ccb69-f96d-47eb-81b0-e69a5758af8d" (UID: "320ccb69-f96d-47eb-81b0-e69a5758af8d"). InnerVolumeSpecName "kube-api-access-445ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:29:53.013946 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:53.013912    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/320ccb69-f96d-47eb-81b0-e69a5758af8d-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:29:53.013946 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:53.013943    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-445ff\" (UniqueName: \"kubernetes.io/projected/320ccb69-f96d-47eb-81b0-e69a5758af8d-kube-api-access-445ff\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:29:53.013946 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:53.013953    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/320ccb69-f96d-47eb-81b0-e69a5758af8d-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:29:53.568894 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:53.568859    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq"
Nov 03 17:29:53.629274 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:53.629246    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e38015c-hk58z"
Nov 03 17:29:53.629274 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:53.629258    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e38015c-hk58z" event={"ID":"320ccb69-f96d-47eb-81b0-e69a5758af8d","Type":"ContainerDied","Data":"890efc578471fd094d035bfbfc492f2d218ad7b0a3eb97993e17d34168fc23fc"}
Nov 03 17:29:53.629518 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:53.629317    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="890efc578471fd094d035bfbfc492f2d218ad7b0a3eb97993e17d34168fc23fc"
Nov 03 17:29:53.631227 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:53.631200    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg" event={"ID":"eb7314fe-6d72-4e51-8cdc-acfcce172a1c","Type":"ContainerStarted","Data":"6c6d1fd85a52eddd9360384fa3b3428da88fdf0b81e3d9ba4f4e47d8b5e80bfd"}
Nov 03 17:29:53.817959 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:53.817906    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg" podStartSLOduration=2.8178894850000002 podStartE2EDuration="2.817889485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:29:53.816664728 +0000 UTC m=+734.614377509" watchObservedRunningTime="2026-11-03 17:29:53.817889485 +0000 UTC m=+734.615602265"
Nov 03 17:29:54.109100 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:54.109065    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:54.114505 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:54.114481    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:54.637266 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:54.637232    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:54.638307 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:54.638286    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:29:56.736575 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:56.736541    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:56.930045 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:56.930012    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:57.134934 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:57.134853    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:29:59.920324 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:29:59.920275    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:00.153426 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.153267    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:00.185779 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.185680    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66"]
Nov 03 17:30:00.186451 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.186429    6059 cpu_manager.go:414] "containerMap: removing container" podUID="320ccb69-f96d-47eb-81b0-e69a5758af8d" containerName="revoke-keys"
Nov 03 17:30:00.186636 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.186451    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="320ccb69-f96d-47eb-81b0-e69a5758af8d" containerName="revoke-keys"
Nov 03 17:30:00.186636 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.186625    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="320ccb69-f96d-47eb-81b0-e69a5758af8d" containerName="revoke-keys"
Nov 03 17:30:00.190792 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.190768    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66"
Nov 03 17:30:00.197728 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.197693    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg"]
Nov 03 17:30:00.199483 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.199456    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9c420a-3483-4dbc-8244-1b324c0fd94f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895450-4zb66\" (UID: \"2d9c420a-3483-4dbc-8244-1b324c0fd94f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66"
Nov 03 17:30:00.200149 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.200124    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9lps\" (UniqueName: \"kubernetes.io/projected/2d9c420a-3483-4dbc-8244-1b324c0fd94f-kube-api-access-g9lps\") pod \"maas-api-key-cleanup-29895450-4zb66\" (UID: \"2d9c420a-3483-4dbc-8244-1b324c0fd94f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66"
Nov 03 17:30:00.202574 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.202551    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg"
Nov 03 17:30:00.216185 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.216156    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg"]
Nov 03 17:30:00.220006 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.219980    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg"
Nov 03 17:30:00.245721 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.245675    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x"]
Nov 03 17:30:00.250428 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.250400    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:30:00.250874 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.250852    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x"
Nov 03 17:30:00.262302 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.262275    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:30:00.262446 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.262282    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:30:00.298346 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.298309    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66"]
Nov 03 17:30:00.301349 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.301313    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-475nt\" (UniqueName: \"kubernetes.io/projected/07083bd4-854a-4967-b7a7-a04c488adf34-kube-api-access-475nt\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg\" (UID: \"07083bd4-854a-4967-b7a7-a04c488adf34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg"
Nov 03 17:30:00.301574 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.301549    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07083bd4-854a-4967-b7a7-a04c488adf34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg\" (UID: \"07083bd4-854a-4967-b7a7-a04c488adf34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg"
Nov 03 17:30:00.301650 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.301614    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf39df4-fab9-4076-89ee-042fdc5476dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x\" (UID: \"5cf39df4-fab9-4076-89ee-042fdc5476dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x"
Nov 03 17:30:00.301710 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.301664    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9c420a-3483-4dbc-8244-1b324c0fd94f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895450-4zb66\" (UID: \"2d9c420a-3483-4dbc-8244-1b324c0fd94f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66"
Nov 03 17:30:00.301772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.301730    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brtcf\" (UniqueName: \"kubernetes.io/projected/212d9a56-7dfc-434e-b52b-914bb17f0ee9-kube-api-access-brtcf\") pod \"maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg\" (UID: \"212d9a56-7dfc-434e-b52b-914bb17f0ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg"
Nov 03 17:30:00.301849 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.301831    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wspt4\" (UniqueName: \"kubernetes.io/projected/5cf39df4-fab9-4076-89ee-042fdc5476dc-kube-api-access-wspt4\") pod \"maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x\" (UID: \"5cf39df4-fab9-4076-89ee-042fdc5476dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x"
Nov 03 17:30:00.301908 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.301888    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9lps\" (UniqueName: \"kubernetes.io/projected/2d9c420a-3483-4dbc-8244-1b324c0fd94f-kube-api-access-g9lps\") pod \"maas-api-key-cleanup-29895450-4zb66\" (UID: \"2d9c420a-3483-4dbc-8244-1b324c0fd94f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66"
Nov 03 17:30:00.301965 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.301929    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212d9a56-7dfc-434e-b52b-914bb17f0ee9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg\" (UID: \"212d9a56-7dfc-434e-b52b-914bb17f0ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg"
Nov 03 17:30:00.302767 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.302743    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9c420a-3483-4dbc-8244-1b324c0fd94f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895450-4zb66\" (UID: \"2d9c420a-3483-4dbc-8244-1b324c0fd94f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66"
Nov 03 17:30:00.312006 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.311977    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg"]
Nov 03 17:30:00.322458 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.322421    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg"]
Nov 03 17:30:00.346742 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.346697    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x"]
Nov 03 17:30:00.388808 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.388777    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9lps\" (UniqueName: \"kubernetes.io/projected/2d9c420a-3483-4dbc-8244-1b324c0fd94f-kube-api-access-g9lps\") pod \"maas-api-key-cleanup-29895450-4zb66\" (UID: \"2d9c420a-3483-4dbc-8244-1b324c0fd94f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66"
Nov 03 17:30:00.415092 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.411825    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brtcf\" (UniqueName: \"kubernetes.io/projected/212d9a56-7dfc-434e-b52b-914bb17f0ee9-kube-api-access-brtcf\") pod \"maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg\" (UID: \"212d9a56-7dfc-434e-b52b-914bb17f0ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg"
Nov 03 17:30:00.415092 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.411977    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wspt4\" (UniqueName: \"kubernetes.io/projected/5cf39df4-fab9-4076-89ee-042fdc5476dc-kube-api-access-wspt4\") pod \"maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x\" (UID: \"5cf39df4-fab9-4076-89ee-042fdc5476dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x"
Nov 03 17:30:00.415092 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.412054    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212d9a56-7dfc-434e-b52b-914bb17f0ee9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg\" (UID: \"212d9a56-7dfc-434e-b52b-914bb17f0ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg"
Nov 03 17:30:00.415092 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.412104    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-475nt\" (UniqueName: \"kubernetes.io/projected/07083bd4-854a-4967-b7a7-a04c488adf34-kube-api-access-475nt\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg\" (UID: \"07083bd4-854a-4967-b7a7-a04c488adf34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg"
Nov 03 17:30:00.415092 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.412202    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07083bd4-854a-4967-b7a7-a04c488adf34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg\" (UID: \"07083bd4-854a-4967-b7a7-a04c488adf34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg"
Nov 03 17:30:00.415092 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.412239    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf39df4-fab9-4076-89ee-042fdc5476dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x\" (UID: \"5cf39df4-fab9-4076-89ee-042fdc5476dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x"
Nov 03 17:30:00.415092 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.413195    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf39df4-fab9-4076-89ee-042fdc5476dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x\" (UID: \"5cf39df4-fab9-4076-89ee-042fdc5476dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x"
Nov 03 17:30:00.449450 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.429979    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07083bd4-854a-4967-b7a7-a04c488adf34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg\" (UID: \"07083bd4-854a-4967-b7a7-a04c488adf34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg"
Nov 03 17:30:00.449450 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.442211    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212d9a56-7dfc-434e-b52b-914bb17f0ee9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg\" (UID: \"212d9a56-7dfc-434e-b52b-914bb17f0ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg"
Nov 03 17:30:00.512570 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.500882    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brtcf\" (UniqueName: \"kubernetes.io/projected/212d9a56-7dfc-434e-b52b-914bb17f0ee9-kube-api-access-brtcf\") pod \"maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg\" (UID: \"212d9a56-7dfc-434e-b52b-914bb17f0ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg"
Nov 03 17:30:00.512570 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.502564    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wspt4\" (UniqueName: \"kubernetes.io/projected/5cf39df4-fab9-4076-89ee-042fdc5476dc-kube-api-access-wspt4\") pod \"maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x\" (UID: \"5cf39df4-fab9-4076-89ee-042fdc5476dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x"
Nov 03 17:30:00.512570 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.503584    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-475nt\" (UniqueName: \"kubernetes.io/projected/07083bd4-854a-4967-b7a7-a04c488adf34-kube-api-access-475nt\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg\" (UID: \"07083bd4-854a-4967-b7a7-a04c488adf34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg"
Nov 03 17:30:00.520773 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.520524    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg"
Nov 03 17:30:00.521030 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.521011    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66"
Nov 03 17:30:00.533612 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.533401    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg"
Nov 03 17:30:00.570403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.570344    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x"
Nov 03 17:30:00.946608 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:30:00.946578    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07083bd4_854a_4967_b7a7_a04c488adf34.slice/crio-21a3b360aacb1c62c3845570b678669e64dc96e5c24edf0bfbdf0a3d54748961 WatchSource:0}: Error finding container 21a3b360aacb1c62c3845570b678669e64dc96e5c24edf0bfbdf0a3d54748961: Status 404 returned error can't find the container with id 21a3b360aacb1c62c3845570b678669e64dc96e5c24edf0bfbdf0a3d54748961
Nov 03 17:30:00.949178 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:00.949152    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg"]
Nov 03 17:30:01.010037 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:30:01.010002    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf39df4_fab9_4076_89ee_042fdc5476dc.slice/crio-9dc79ab11c1162847e1bc110225b4812b19079bcbe85f0605b66c47a7ddec0c3 WatchSource:0}: Error finding container 9dc79ab11c1162847e1bc110225b4812b19079bcbe85f0605b66c47a7ddec0c3: Status 404 returned error can't find the container with id 9dc79ab11c1162847e1bc110225b4812b19079bcbe85f0605b66c47a7ddec0c3
Nov 03 17:30:01.012404 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:01.012315    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x"]
Nov 03 17:30:01.014978 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:01.014958    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg"]
Nov 03 17:30:01.017477 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:01.017448    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66"]
Nov 03 17:30:01.021010 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:30:01.020985    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212d9a56_7dfc_434e_b52b_914bb17f0ee9.slice/crio-9f282835aa9f42ee7bf42813788e66350a84e7e0753f6620ded7b2206585352e WatchSource:0}: Error finding container 9f282835aa9f42ee7bf42813788e66350a84e7e0753f6620ded7b2206585352e: Status 404 returned error can't find the container with id 9f282835aa9f42ee7bf42813788e66350a84e7e0753f6620ded7b2206585352e
Nov 03 17:30:01.023342 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:30:01.023313    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9c420a_3483_4dbc_8244_1b324c0fd94f.slice/crio-e752828351c6bc06aa0eb20c6e1cf4a7d357da92a0e403abccb1cd5c7aa1d042 WatchSource:0}: Error finding container e752828351c6bc06aa0eb20c6e1cf4a7d357da92a0e403abccb1cd5c7aa1d042: Status 404 returned error can't find the container with id e752828351c6bc06aa0eb20c6e1cf4a7d357da92a0e403abccb1cd5c7aa1d042
Nov 03 17:30:01.694479 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:01.694437    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66" event={"ID":"2d9c420a-3483-4dbc-8244-1b324c0fd94f","Type":"ContainerStarted","Data":"93fe37c8e2a9a0c387ba4788b327e5c9e0dbfe80183ee265f30f162bcbd0f614"}
Nov 03 17:30:01.694670 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:01.694530    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66" event={"ID":"2d9c420a-3483-4dbc-8244-1b324c0fd94f","Type":"ContainerStarted","Data":"e752828351c6bc06aa0eb20c6e1cf4a7d357da92a0e403abccb1cd5c7aa1d042"}
Nov 03 17:30:01.698237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:01.696234    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x" event={"ID":"5cf39df4-fab9-4076-89ee-042fdc5476dc","Type":"ContainerStarted","Data":"46949f5ffe52ff6a58d277c84dacfee88f9962fbe59abcae78038f16ceacef8f"}
Nov 03 17:30:01.698237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:01.696291    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x" event={"ID":"5cf39df4-fab9-4076-89ee-042fdc5476dc","Type":"ContainerStarted","Data":"9dc79ab11c1162847e1bc110225b4812b19079bcbe85f0605b66c47a7ddec0c3"}
Nov 03 17:30:01.698472 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:01.698339    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg" event={"ID":"07083bd4-854a-4967-b7a7-a04c488adf34","Type":"ContainerStarted","Data":"4b2a39a9fbd5c363f58c0714253d5c34c95ec4cd82e0fd5ffa54ec465d625a4e"}
Nov 03 17:30:01.698472 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:01.698410    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg" event={"ID":"07083bd4-854a-4967-b7a7-a04c488adf34","Type":"ContainerStarted","Data":"21a3b360aacb1c62c3845570b678669e64dc96e5c24edf0bfbdf0a3d54748961"}
Nov 03 17:30:01.699831 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:01.699808    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg" event={"ID":"212d9a56-7dfc-434e-b52b-914bb17f0ee9","Type":"ContainerStarted","Data":"fee74757162fb5a21b71e9ef26d544364688501eb825dc9f878516efab5a2864"}
Nov 03 17:30:01.699946 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:01.699852    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg" event={"ID":"212d9a56-7dfc-434e-b52b-914bb17f0ee9","Type":"ContainerStarted","Data":"9f282835aa9f42ee7bf42813788e66350a84e7e0753f6620ded7b2206585352e"}
Nov 03 17:30:01.817530 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:01.817414    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66" podStartSLOduration=1.817388754 podStartE2EDuration="1.817388754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:01.815861684 +0000 UTC m=+742.613574476" watchObservedRunningTime="2026-11-03 17:30:01.817388754 +0000 UTC m=+742.615101532"
Nov 03 17:30:01.944474 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:01.944426    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x" podStartSLOduration=1.944408284 podStartE2EDuration="1.944408284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:01.941110489 +0000 UTC m=+742.738823271" watchObservedRunningTime="2026-11-03 17:30:01.944408284 +0000 UTC m=+742.742121056"
Nov 03 17:30:02.060087 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:02.060037    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg" podStartSLOduration=2.060022461 podStartE2EDuration="2.060022461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:02.05941724 +0000 UTC m=+742.857130021" watchObservedRunningTime="2026-11-03 17:30:02.060022461 +0000 UTC m=+742.857735288"
Nov 03 17:30:02.129922 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:02.129819    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:02.181462 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:02.181406    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg" podStartSLOduration=2.181390337 podStartE2EDuration="2.181390337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:02.17891437 +0000 UTC m=+742.976627152" watchObservedRunningTime="2026-11-03 17:30:02.181390337 +0000 UTC m=+742.979103112"
Nov 03 17:30:02.303686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:02.303652    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:02.491212 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:02.491177    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:02.707374 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:02.707332    6059 generic.go:356] "Generic (PLEG): container finished" podID="5cf39df4-fab9-4076-89ee-042fdc5476dc" containerID="46949f5ffe52ff6a58d277c84dacfee88f9962fbe59abcae78038f16ceacef8f" exitCode=60
Nov 03 17:30:02.707573 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:02.707422    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x" event={"ID":"5cf39df4-fab9-4076-89ee-042fdc5476dc","Type":"ContainerDied","Data":"46949f5ffe52ff6a58d277c84dacfee88f9962fbe59abcae78038f16ceacef8f"}
Nov 03 17:30:02.707734 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:02.707713    6059 scope.go:122] "RemoveContainer" containerID="46949f5ffe52ff6a58d277c84dacfee88f9962fbe59abcae78038f16ceacef8f"
Nov 03 17:30:02.709078 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:02.709056    6059 generic.go:356] "Generic (PLEG): container finished" podID="07083bd4-854a-4967-b7a7-a04c488adf34" containerID="4b2a39a9fbd5c363f58c0714253d5c34c95ec4cd82e0fd5ffa54ec465d625a4e" exitCode=60
Nov 03 17:30:02.709189 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:02.709134    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg" event={"ID":"07083bd4-854a-4967-b7a7-a04c488adf34","Type":"ContainerDied","Data":"4b2a39a9fbd5c363f58c0714253d5c34c95ec4cd82e0fd5ffa54ec465d625a4e"}
Nov 03 17:30:02.709402 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:02.709388    6059 scope.go:122] "RemoveContainer" containerID="4b2a39a9fbd5c363f58c0714253d5c34c95ec4cd82e0fd5ffa54ec465d625a4e"
Nov 03 17:30:02.714735 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:02.711878    6059 generic.go:356] "Generic (PLEG): container finished" podID="212d9a56-7dfc-434e-b52b-914bb17f0ee9" containerID="fee74757162fb5a21b71e9ef26d544364688501eb825dc9f878516efab5a2864" exitCode=60
Nov 03 17:30:02.714735 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:02.711961    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg" event={"ID":"212d9a56-7dfc-434e-b52b-914bb17f0ee9","Type":"ContainerDied","Data":"fee74757162fb5a21b71e9ef26d544364688501eb825dc9f878516efab5a2864"}
Nov 03 17:30:02.714735 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:02.712294    6059 scope.go:122] "RemoveContainer" containerID="fee74757162fb5a21b71e9ef26d544364688501eb825dc9f878516efab5a2864"
Nov 03 17:30:02.715633 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:02.715601    6059 generic.go:356] "Generic (PLEG): container finished" podID="2d9c420a-3483-4dbc-8244-1b324c0fd94f" containerID="93fe37c8e2a9a0c387ba4788b327e5c9e0dbfe80183ee265f30f162bcbd0f614" exitCode=60
Nov 03 17:30:02.715735 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:02.715659    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66" event={"ID":"2d9c420a-3483-4dbc-8244-1b324c0fd94f","Type":"ContainerDied","Data":"93fe37c8e2a9a0c387ba4788b327e5c9e0dbfe80183ee265f30f162bcbd0f614"}
Nov 03 17:30:02.715924 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:02.715908    6059 scope.go:122] "RemoveContainer" containerID="93fe37c8e2a9a0c387ba4788b327e5c9e0dbfe80183ee265f30f162bcbd0f614"
Nov 03 17:30:03.724151 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:03.724115    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg" event={"ID":"212d9a56-7dfc-434e-b52b-914bb17f0ee9","Type":"ContainerStarted","Data":"b5be54b1b59cd9e7ca549bcce42b7156236d61ede1895ef557eeea63fa0a1c74"}
Nov 03 17:30:03.726458 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:03.726428    6059 generic.go:356] "Generic (PLEG): container finished" podID="2d9c420a-3483-4dbc-8244-1b324c0fd94f" containerID="3f0963bf4028738b3ec10f43cbd773a2480b29178006c6f2a4179771410499a9" exitCode=60
Nov 03 17:30:03.726608 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:03.726506    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66" event={"ID":"2d9c420a-3483-4dbc-8244-1b324c0fd94f","Type":"ContainerDied","Data":"3f0963bf4028738b3ec10f43cbd773a2480b29178006c6f2a4179771410499a9"}
Nov 03 17:30:03.726608 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:03.726569    6059 scope.go:122] "RemoveContainer" containerID="93fe37c8e2a9a0c387ba4788b327e5c9e0dbfe80183ee265f30f162bcbd0f614"
Nov 03 17:30:03.726802 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:03.726777    6059 scope.go:122] "RemoveContainer" containerID="3f0963bf4028738b3ec10f43cbd773a2480b29178006c6f2a4179771410499a9"
Nov 03 17:30:03.727085 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:30:03.727055    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895450-4zb66_odh-ai-gateway-infra(2d9c420a-3483-4dbc-8244-1b324c0fd94f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66" podUID="2d9c420a-3483-4dbc-8244-1b324c0fd94f"
Nov 03 17:30:03.728492 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:03.728468    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x" event={"ID":"5cf39df4-fab9-4076-89ee-042fdc5476dc","Type":"ContainerStarted","Data":"23a81d3d9a771291a8a02a78157737c3d2dbd54dfdca83e2da806e0afb490206"}
Nov 03 17:30:03.730645 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:03.730623    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg" event={"ID":"07083bd4-854a-4967-b7a7-a04c488adf34","Type":"ContainerStarted","Data":"2e5c0bb6077856daa4b541c573c15c3537489155580db42987b0ea9b3edb4e1c"}
Nov 03 17:30:04.739183 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:04.739144    6059 generic.go:356] "Generic (PLEG): container finished" podID="212d9a56-7dfc-434e-b52b-914bb17f0ee9" containerID="b5be54b1b59cd9e7ca549bcce42b7156236d61ede1895ef557eeea63fa0a1c74" exitCode=60
Nov 03 17:30:04.740476 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:04.740075    6059 scope.go:122] "RemoveContainer" containerID="b5be54b1b59cd9e7ca549bcce42b7156236d61ede1895ef557eeea63fa0a1c74"
Nov 03 17:30:04.740632 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:04.740334    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg" event={"ID":"212d9a56-7dfc-434e-b52b-914bb17f0ee9","Type":"ContainerDied","Data":"b5be54b1b59cd9e7ca549bcce42b7156236d61ede1895ef557eeea63fa0a1c74"}
Nov 03 17:30:04.740632 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:04.740610    6059 scope.go:122] "RemoveContainer" containerID="fee74757162fb5a21b71e9ef26d544364688501eb825dc9f878516efab5a2864"
Nov 03 17:30:04.740790 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:30:04.740774    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg_odh-ai-gateway-infra(212d9a56-7dfc-434e-b52b-914bb17f0ee9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg" podUID="212d9a56-7dfc-434e-b52b-914bb17f0ee9"
Nov 03 17:30:04.742933 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:04.742914    6059 scope.go:122] "RemoveContainer" containerID="3f0963bf4028738b3ec10f43cbd773a2480b29178006c6f2a4179771410499a9"
Nov 03 17:30:04.743245 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:30:04.743219    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895450-4zb66_odh-ai-gateway-infra(2d9c420a-3483-4dbc-8244-1b324c0fd94f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66" podUID="2d9c420a-3483-4dbc-8244-1b324c0fd94f"
Nov 03 17:30:04.744938 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:04.744917    6059 generic.go:356] "Generic (PLEG): container finished" podID="5cf39df4-fab9-4076-89ee-042fdc5476dc" containerID="23a81d3d9a771291a8a02a78157737c3d2dbd54dfdca83e2da806e0afb490206" exitCode=60
Nov 03 17:30:04.745033 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:04.744996    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x" event={"ID":"5cf39df4-fab9-4076-89ee-042fdc5476dc","Type":"ContainerDied","Data":"23a81d3d9a771291a8a02a78157737c3d2dbd54dfdca83e2da806e0afb490206"}
Nov 03 17:30:04.745239 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:04.745223    6059 scope.go:122] "RemoveContainer" containerID="23a81d3d9a771291a8a02a78157737c3d2dbd54dfdca83e2da806e0afb490206"
Nov 03 17:30:04.745534 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:30:04.745517    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x_odh-ai-gateway-infra(5cf39df4-fab9-4076-89ee-042fdc5476dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x" podUID="5cf39df4-fab9-4076-89ee-042fdc5476dc"
Nov 03 17:30:04.746946 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:04.746925    6059 generic.go:356] "Generic (PLEG): container finished" podID="07083bd4-854a-4967-b7a7-a04c488adf34" containerID="2e5c0bb6077856daa4b541c573c15c3537489155580db42987b0ea9b3edb4e1c" exitCode=60
Nov 03 17:30:04.747078 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:04.747059    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg" event={"ID":"07083bd4-854a-4967-b7a7-a04c488adf34","Type":"ContainerDied","Data":"2e5c0bb6077856daa4b541c573c15c3537489155580db42987b0ea9b3edb4e1c"}
Nov 03 17:30:04.747374 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:04.747344    6059 scope.go:122] "RemoveContainer" containerID="2e5c0bb6077856daa4b541c573c15c3537489155580db42987b0ea9b3edb4e1c"
Nov 03 17:30:04.747589 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:30:04.747571    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg_odh-ai-gateway-infra(07083bd4-854a-4967-b7a7-a04c488adf34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg" podUID="07083bd4-854a-4967-b7a7-a04c488adf34"
Nov 03 17:30:04.755164 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:04.755139    6059 scope.go:122] "RemoveContainer" containerID="46949f5ffe52ff6a58d277c84dacfee88f9962fbe59abcae78038f16ceacef8f"
Nov 03 17:30:04.768280 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:04.768253    6059 scope.go:122] "RemoveContainer" containerID="4b2a39a9fbd5c363f58c0714253d5c34c95ec4cd82e0fd5ffa54ec465d625a4e"
Nov 03 17:30:05.755480 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:05.755446    6059 scope.go:122] "RemoveContainer" containerID="23a81d3d9a771291a8a02a78157737c3d2dbd54dfdca83e2da806e0afb490206"
Nov 03 17:30:05.755988 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:30:05.755766    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x_odh-ai-gateway-infra(5cf39df4-fab9-4076-89ee-042fdc5476dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x" podUID="5cf39df4-fab9-4076-89ee-042fdc5476dc"
Nov 03 17:30:05.757115 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:05.757094    6059 scope.go:122] "RemoveContainer" containerID="2e5c0bb6077856daa4b541c573c15c3537489155580db42987b0ea9b3edb4e1c"
Nov 03 17:30:05.757405 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:30:05.757375    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg_odh-ai-gateway-infra(07083bd4-854a-4967-b7a7-a04c488adf34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg" podUID="07083bd4-854a-4967-b7a7-a04c488adf34"
Nov 03 17:30:05.758803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:05.758786    6059 scope.go:122] "RemoveContainer" containerID="b5be54b1b59cd9e7ca549bcce42b7156236d61ede1895ef557eeea63fa0a1c74"
Nov 03 17:30:05.759026 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:30:05.759007    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg_odh-ai-gateway-infra(212d9a56-7dfc-434e-b52b-914bb17f0ee9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg" podUID="212d9a56-7dfc-434e-b52b-914bb17f0ee9"
Nov 03 17:30:06.862743 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:30:06.862698    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:30:06.932918 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:06.932888    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:11.504996 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:11.504948    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:15.329641 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:15.329607    6059 scope.go:122] "RemoveContainer" containerID="23a81d3d9a771291a8a02a78157737c3d2dbd54dfdca83e2da806e0afb490206"
Nov 03 17:30:15.340385 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:15.330645    6059 scope.go:122] "RemoveContainer" containerID="2e5c0bb6077856daa4b541c573c15c3537489155580db42987b0ea9b3edb4e1c"
Nov 03 17:30:15.822283 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:15.822244    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x" event={"ID":"5cf39df4-fab9-4076-89ee-042fdc5476dc","Type":"ContainerStarted","Data":"83026f2eef7e4b2688ebe499a1aeb77a3a036528b3e9ac5561f43b5f76e4c313"}
Nov 03 17:30:15.824222 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:15.824189    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg" event={"ID":"07083bd4-854a-4967-b7a7-a04c488adf34","Type":"ContainerStarted","Data":"3e7b55f0500ebe5eedef2ce7cae7654223ea68340b1318b79b919f37cbb4045a"}
Nov 03 17:30:16.245537 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:16.245502    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:16.852348 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:16.852252    6059 generic.go:356] "Generic (PLEG): container finished" podID="5cf39df4-fab9-4076-89ee-042fdc5476dc" containerID="83026f2eef7e4b2688ebe499a1aeb77a3a036528b3e9ac5561f43b5f76e4c313" exitCode=60
Nov 03 17:30:16.854492 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:16.852408    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x" event={"ID":"5cf39df4-fab9-4076-89ee-042fdc5476dc","Type":"ContainerDied","Data":"83026f2eef7e4b2688ebe499a1aeb77a3a036528b3e9ac5561f43b5f76e4c313"}
Nov 03 17:30:16.854671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:16.854646    6059 scope.go:122] "RemoveContainer" containerID="23a81d3d9a771291a8a02a78157737c3d2dbd54dfdca83e2da806e0afb490206"
Nov 03 17:30:16.855117 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:16.855092    6059 scope.go:122] "RemoveContainer" containerID="83026f2eef7e4b2688ebe499a1aeb77a3a036528b3e9ac5561f43b5f76e4c313"
Nov 03 17:30:16.855697 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:30:16.855663    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x_odh-ai-gateway-infra(5cf39df4-fab9-4076-89ee-042fdc5476dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x" podUID="5cf39df4-fab9-4076-89ee-042fdc5476dc"
Nov 03 17:30:16.857007 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:16.856982    6059 generic.go:356] "Generic (PLEG): container finished" podID="07083bd4-854a-4967-b7a7-a04c488adf34" containerID="3e7b55f0500ebe5eedef2ce7cae7654223ea68340b1318b79b919f37cbb4045a" exitCode=60
Nov 03 17:30:16.857110 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:16.857026    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg" event={"ID":"07083bd4-854a-4967-b7a7-a04c488adf34","Type":"ContainerDied","Data":"3e7b55f0500ebe5eedef2ce7cae7654223ea68340b1318b79b919f37cbb4045a"}
Nov 03 17:30:16.857336 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:16.857321    6059 scope.go:122] "RemoveContainer" containerID="3e7b55f0500ebe5eedef2ce7cae7654223ea68340b1318b79b919f37cbb4045a"
Nov 03 17:30:16.857578 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:30:16.857556    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg_odh-ai-gateway-infra(07083bd4-854a-4967-b7a7-a04c488adf34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg" podUID="07083bd4-854a-4967-b7a7-a04c488adf34"
Nov 03 17:30:16.867198 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:16.867173    6059 scope.go:122] "RemoveContainer" containerID="2e5c0bb6077856daa4b541c573c15c3537489155580db42987b0ea9b3edb4e1c"
Nov 03 17:30:16.964061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:16.964028    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x"]
Nov 03 17:30:17.117915 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:17.117825    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg"]
Nov 03 17:30:17.328986 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:17.328958    6059 scope.go:122] "RemoveContainer" containerID="b5be54b1b59cd9e7ca549bcce42b7156236d61ede1895ef557eeea63fa0a1c74"
Nov 03 17:30:17.329133 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:17.329085    6059 scope.go:122] "RemoveContainer" containerID="3f0963bf4028738b3ec10f43cbd773a2480b29178006c6f2a4179771410499a9"
Nov 03 17:30:17.872582 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:17.872549    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66" event={"ID":"2d9c420a-3483-4dbc-8244-1b324c0fd94f","Type":"ContainerStarted","Data":"e52c4d2096fee3879e04ac40c2035e3de207db1037d496181493bcf8689c39d5"}
Nov 03 17:30:18.075454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.075417    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg"
Nov 03 17:30:18.103657 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.103621    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x"
Nov 03 17:30:18.192736 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.192641    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07083bd4-854a-4967-b7a7-a04c488adf34-kube-api-access-475nt\" (UniqueName: \"kubernetes.io/projected/07083bd4-854a-4967-b7a7-a04c488adf34-kube-api-access-475nt\") pod \"07083bd4-854a-4967-b7a7-a04c488adf34\" (UID: \"07083bd4-854a-4967-b7a7-a04c488adf34\") "
Nov 03 17:30:18.192736 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.192702    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cf39df4-fab9-4076-89ee-042fdc5476dc-kube-api-access-wspt4\" (UniqueName: \"kubernetes.io/projected/5cf39df4-fab9-4076-89ee-042fdc5476dc-kube-api-access-wspt4\") pod \"5cf39df4-fab9-4076-89ee-042fdc5476dc\" (UID: \"5cf39df4-fab9-4076-89ee-042fdc5476dc\") "
Nov 03 17:30:18.192984 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.192757    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07083bd4-854a-4967-b7a7-a04c488adf34-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07083bd4-854a-4967-b7a7-a04c488adf34-maas-api-service-ca\") pod \"07083bd4-854a-4967-b7a7-a04c488adf34\" (UID: \"07083bd4-854a-4967-b7a7-a04c488adf34\") "
Nov 03 17:30:18.193066 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.192964    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cf39df4-fab9-4076-89ee-042fdc5476dc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf39df4-fab9-4076-89ee-042fdc5476dc-maas-api-service-ca\") pod \"5cf39df4-fab9-4076-89ee-042fdc5476dc\" (UID: \"5cf39df4-fab9-4076-89ee-042fdc5476dc\") "
Nov 03 17:30:18.193124 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.193095    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07083bd4-854a-4967-b7a7-a04c488adf34-maas-api-service-ca" pod "07083bd4-854a-4967-b7a7-a04c488adf34" (UID: "07083bd4-854a-4967-b7a7-a04c488adf34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:30:18.193273 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.193250    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf39df4-fab9-4076-89ee-042fdc5476dc-maas-api-service-ca" pod "5cf39df4-fab9-4076-89ee-042fdc5476dc" (UID: "5cf39df4-fab9-4076-89ee-042fdc5476dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:30:18.193543 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.193529    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07083bd4-854a-4967-b7a7-a04c488adf34-maas-api-service-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:30:18.193586 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.193547    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf39df4-fab9-4076-89ee-042fdc5476dc-maas-api-service-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:30:18.194798 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.194775    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf39df4-fab9-4076-89ee-042fdc5476dc-kube-api-access-wspt4" pod "5cf39df4-fab9-4076-89ee-042fdc5476dc" (UID: "5cf39df4-fab9-4076-89ee-042fdc5476dc"). InnerVolumeSpecName "kube-api-access-wspt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:30:18.194867 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.194836    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07083bd4-854a-4967-b7a7-a04c488adf34-kube-api-access-475nt" pod "07083bd4-854a-4967-b7a7-a04c488adf34" (UID: "07083bd4-854a-4967-b7a7-a04c488adf34"). InnerVolumeSpecName "kube-api-access-475nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:30:18.294527 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.294492    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-475nt\" (UniqueName: \"kubernetes.io/projected/07083bd4-854a-4967-b7a7-a04c488adf34-kube-api-access-475nt\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:30:18.294527 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.294519    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wspt4\" (UniqueName: \"kubernetes.io/projected/5cf39df4-fab9-4076-89ee-042fdc5476dc-kube-api-access-wspt4\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:30:18.880785 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.880750    6059 generic.go:356] "Generic (PLEG): container finished" podID="212d9a56-7dfc-434e-b52b-914bb17f0ee9" containerID="044b2b116150dbe26fea6f43db033692087d46035e031009cc9d61459cda8441" exitCode=60
Nov 03 17:30:18.881196 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.880826    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg" event={"ID":"212d9a56-7dfc-434e-b52b-914bb17f0ee9","Type":"ContainerDied","Data":"044b2b116150dbe26fea6f43db033692087d46035e031009cc9d61459cda8441"}
Nov 03 17:30:18.881196 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.880886    6059 scope.go:122] "RemoveContainer" containerID="b5be54b1b59cd9e7ca549bcce42b7156236d61ede1895ef557eeea63fa0a1c74"
Nov 03 17:30:18.881324 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.881252    6059 scope.go:122] "RemoveContainer" containerID="044b2b116150dbe26fea6f43db033692087d46035e031009cc9d61459cda8441"
Nov 03 17:30:18.881616 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:30:18.881587    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg_odh-ai-gateway-infra(212d9a56-7dfc-434e-b52b-914bb17f0ee9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg" podUID="212d9a56-7dfc-434e-b52b-914bb17f0ee9"
Nov 03 17:30:18.883611 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.883592    6059 generic.go:356] "Generic (PLEG): container finished" podID="2d9c420a-3483-4dbc-8244-1b324c0fd94f" containerID="e52c4d2096fee3879e04ac40c2035e3de207db1037d496181493bcf8689c39d5" exitCode=60
Nov 03 17:30:18.883708 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.883660    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66" event={"ID":"2d9c420a-3483-4dbc-8244-1b324c0fd94f","Type":"ContainerDied","Data":"e52c4d2096fee3879e04ac40c2035e3de207db1037d496181493bcf8689c39d5"}
Nov 03 17:30:18.884016 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.883990    6059 scope.go:122] "RemoveContainer" containerID="e52c4d2096fee3879e04ac40c2035e3de207db1037d496181493bcf8689c39d5"
Nov 03 17:30:18.884269 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:30:18.884246    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895450-4zb66_odh-ai-gateway-infra(2d9c420a-3483-4dbc-8244-1b324c0fd94f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66" podUID="2d9c420a-3483-4dbc-8244-1b324c0fd94f"
Nov 03 17:30:18.885064 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.885047    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x"
Nov 03 17:30:18.885064 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.885059    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x" event={"ID":"5cf39df4-fab9-4076-89ee-042fdc5476dc","Type":"ContainerDied","Data":"9dc79ab11c1162847e1bc110225b4812b19079bcbe85f0605b66c47a7ddec0c3"}
Nov 03 17:30:18.886525 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.886503    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg"
Nov 03 17:30:18.886597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.886542    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg" event={"ID":"07083bd4-854a-4967-b7a7-a04c488adf34","Type":"ContainerDied","Data":"21a3b360aacb1c62c3845570b678669e64dc96e5c24edf0bfbdf0a3d54748961"}
Nov 03 17:30:18.894552 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.894532    6059 scope.go:122] "RemoveContainer" containerID="3f0963bf4028738b3ec10f43cbd773a2480b29178006c6f2a4179771410499a9"
Nov 03 17:30:18.907728 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.907709    6059 scope.go:122] "RemoveContainer" containerID="83026f2eef7e4b2688ebe499a1aeb77a3a036528b3e9ac5561f43b5f76e4c313"
Nov 03 17:30:18.916924 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:18.916903    6059 scope.go:122] "RemoveContainer" containerID="3e7b55f0500ebe5eedef2ce7cae7654223ea68340b1318b79b919f37cbb4045a"
Nov 03 17:30:19.122814 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:19.122781    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66"]
Nov 03 17:30:19.129299 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:19.129263    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg"]
Nov 03 17:30:19.135187 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:19.135114    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895450-hnjjg"]
Nov 03 17:30:19.336999 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:19.336968    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07083bd4-854a-4967-b7a7-a04c488adf34" path="/var/lib/kubelet/pods/07083bd4-854a-4967-b7a7-a04c488adf34/volumes"
Nov 03 17:30:19.373220 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:19.373178    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x"]
Nov 03 17:30:19.379315 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:19.379277    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3e5c4d-29895450-8sq9x"]
Nov 03 17:30:20.030011 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.029971    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg"]
Nov 03 17:30:20.057854 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.057821    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66"
Nov 03 17:30:20.118220 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.118185    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d9c420a-3483-4dbc-8244-1b324c0fd94f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9c420a-3483-4dbc-8244-1b324c0fd94f-maas-api-service-ca\") pod \"2d9c420a-3483-4dbc-8244-1b324c0fd94f\" (UID: \"2d9c420a-3483-4dbc-8244-1b324c0fd94f\") "
Nov 03 17:30:20.118388 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.118242    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d9c420a-3483-4dbc-8244-1b324c0fd94f-kube-api-access-g9lps\" (UniqueName: \"kubernetes.io/projected/2d9c420a-3483-4dbc-8244-1b324c0fd94f-kube-api-access-g9lps\") pod \"2d9c420a-3483-4dbc-8244-1b324c0fd94f\" (UID: \"2d9c420a-3483-4dbc-8244-1b324c0fd94f\") "
Nov 03 17:30:20.118751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.118723    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9c420a-3483-4dbc-8244-1b324c0fd94f-maas-api-service-ca" pod "2d9c420a-3483-4dbc-8244-1b324c0fd94f" (UID: "2d9c420a-3483-4dbc-8244-1b324c0fd94f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:30:20.120552 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.120526    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9c420a-3483-4dbc-8244-1b324c0fd94f-kube-api-access-g9lps" pod "2d9c420a-3483-4dbc-8244-1b324c0fd94f" (UID: "2d9c420a-3483-4dbc-8244-1b324c0fd94f"). InnerVolumeSpecName "kube-api-access-g9lps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:30:20.210819 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.210791    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg"
Nov 03 17:30:20.219886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.219849    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9c420a-3483-4dbc-8244-1b324c0fd94f-maas-api-service-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:30:20.220041 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.219888    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9lps\" (UniqueName: \"kubernetes.io/projected/2d9c420a-3483-4dbc-8244-1b324c0fd94f-kube-api-access-g9lps\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:30:20.323845 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.323797    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/212d9a56-7dfc-434e-b52b-914bb17f0ee9-kube-api-access-brtcf\" (UniqueName: \"kubernetes.io/projected/212d9a56-7dfc-434e-b52b-914bb17f0ee9-kube-api-access-brtcf\") pod \"212d9a56-7dfc-434e-b52b-914bb17f0ee9\" (UID: \"212d9a56-7dfc-434e-b52b-914bb17f0ee9\") "
Nov 03 17:30:20.324006 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.323964    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/212d9a56-7dfc-434e-b52b-914bb17f0ee9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212d9a56-7dfc-434e-b52b-914bb17f0ee9-maas-api-service-ca\") pod \"212d9a56-7dfc-434e-b52b-914bb17f0ee9\" (UID: \"212d9a56-7dfc-434e-b52b-914bb17f0ee9\") "
Nov 03 17:30:20.325186 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.325154    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/212d9a56-7dfc-434e-b52b-914bb17f0ee9-maas-api-service-ca" pod "212d9a56-7dfc-434e-b52b-914bb17f0ee9" (UID: "212d9a56-7dfc-434e-b52b-914bb17f0ee9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:30:20.326858 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.326831    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212d9a56-7dfc-434e-b52b-914bb17f0ee9-kube-api-access-brtcf" pod "212d9a56-7dfc-434e-b52b-914bb17f0ee9" (UID: "212d9a56-7dfc-434e-b52b-914bb17f0ee9"). InnerVolumeSpecName "kube-api-access-brtcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:30:20.426305 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.426214    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brtcf\" (UniqueName: \"kubernetes.io/projected/212d9a56-7dfc-434e-b52b-914bb17f0ee9-kube-api-access-brtcf\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:30:20.426305 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.426253    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212d9a56-7dfc-434e-b52b-914bb17f0ee9-maas-api-service-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:30:20.548593 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.548554    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:20.807978 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.807643    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:20.907891 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.907847    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg" event={"ID":"212d9a56-7dfc-434e-b52b-914bb17f0ee9","Type":"ContainerDied","Data":"9f282835aa9f42ee7bf42813788e66350a84e7e0753f6620ded7b2206585352e"}
Nov 03 17:30:20.908088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.907890    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg"
Nov 03 17:30:20.908088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.907923    6059 scope.go:122] "RemoveContainer" containerID="044b2b116150dbe26fea6f43db033692087d46035e031009cc9d61459cda8441"
Nov 03 17:30:20.909292 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.909266    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66" event={"ID":"2d9c420a-3483-4dbc-8244-1b324c0fd94f","Type":"ContainerDied","Data":"e752828351c6bc06aa0eb20c6e1cf4a7d357da92a0e403abccb1cd5c7aa1d042"}
Nov 03 17:30:20.909406 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.909323    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66"
Nov 03 17:30:20.925152 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:20.925124    6059 scope.go:122] "RemoveContainer" containerID="e52c4d2096fee3879e04ac40c2035e3de207db1037d496181493bcf8689c39d5"
Nov 03 17:30:21.081925 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:21.081851    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66"]
Nov 03 17:30:21.084121 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:21.084098    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895450-4zb66"]
Nov 03 17:30:21.198240 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:21.198203    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg"]
Nov 03 17:30:21.204443 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:21.204405    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895450-rt9mg"]
Nov 03 17:30:21.335626 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:21.335551    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="212d9a56-7dfc-434e-b52b-914bb17f0ee9" path="/var/lib/kubelet/pods/212d9a56-7dfc-434e-b52b-914bb17f0ee9/volumes"
Nov 03 17:30:21.335980 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:21.335966    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d9c420a-3483-4dbc-8244-1b324c0fd94f" path="/var/lib/kubelet/pods/2d9c420a-3483-4dbc-8244-1b324c0fd94f/volumes"
Nov 03 17:30:21.336321 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:21.336310    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cf39df4-fab9-4076-89ee-042fdc5476dc" path="/var/lib/kubelet/pods/5cf39df4-fab9-4076-89ee-042fdc5476dc/volumes"
Nov 03 17:30:22.926748 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:22.926646    6059 generic.go:356] "Generic (PLEG): container finished" podID="012064a7-fd12-465b-bdc8-ef6d4ca289b8" containerID="4273dbde4f713e9e9355e86370d12cd49133e36861a93b6300401162988e55ce" exitCode=0
Nov 03 17:30:22.926748 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:22.926712    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx" event={"ID":"012064a7-fd12-465b-bdc8-ef6d4ca289b8","Type":"ContainerDied","Data":"4273dbde4f713e9e9355e86370d12cd49133e36861a93b6300401162988e55ce"}
Nov 03 17:30:22.927200 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:22.927125    6059 scope.go:122] "RemoveContainer" containerID="4273dbde4f713e9e9355e86370d12cd49133e36861a93b6300401162988e55ce"
Nov 03 17:30:22.928597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:22.928573    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:30:22.928654 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:22.928638    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:30:22.928690 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:22.928668    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:30:23.959651 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:23.959616    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx" event={"ID":"012064a7-fd12-465b-bdc8-ef6d4ca289b8","Type":"ContainerStarted","Data":"ba3eb4fb0314502f6b0917100436f52d8e9d7065eec92be5a27ec5006eb62106"}
Nov 03 17:30:24.082512 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:24.082478    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:30:24.082730 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:24.082702    6059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 17:30:24.082816 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:24.082762    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx" podUID="012064a7-fd12-465b-bdc8-ef6d4ca289b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 17:30:25.082168 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.082134    6059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 17:30:25.082589 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.082202    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx" podUID="012064a7-fd12-465b-bdc8-ef6d4ca289b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 17:30:25.115588 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.115551    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"]
Nov 03 17:30:25.116050 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116037    6059 cpu_manager.go:414] "containerMap: removing container" podUID="07083bd4-854a-4967-b7a7-a04c488adf34" containerName="cleanup"
Nov 03 17:30:25.116101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116051    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07083bd4-854a-4967-b7a7-a04c488adf34" containerName="cleanup"
Nov 03 17:30:25.116101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116064    6059 cpu_manager.go:414] "containerMap: removing container" podUID="2d9c420a-3483-4dbc-8244-1b324c0fd94f" containerName="cleanup"
Nov 03 17:30:25.116101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116069    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d9c420a-3483-4dbc-8244-1b324c0fd94f" containerName="cleanup"
Nov 03 17:30:25.116101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116081    6059 cpu_manager.go:414] "containerMap: removing container" podUID="5cf39df4-fab9-4076-89ee-042fdc5476dc" containerName="cleanup"
Nov 03 17:30:25.116101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116088    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf39df4-fab9-4076-89ee-042fdc5476dc" containerName="cleanup"
Nov 03 17:30:25.116101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116102    6059 cpu_manager.go:414] "containerMap: removing container" podUID="212d9a56-7dfc-434e-b52b-914bb17f0ee9" containerName="cleanup"
Nov 03 17:30:25.116297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116107    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212d9a56-7dfc-434e-b52b-914bb17f0ee9" containerName="cleanup"
Nov 03 17:30:25.116297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116118    6059 cpu_manager.go:414] "containerMap: removing container" podUID="5cf39df4-fab9-4076-89ee-042fdc5476dc" containerName="cleanup"
Nov 03 17:30:25.116297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116122    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf39df4-fab9-4076-89ee-042fdc5476dc" containerName="cleanup"
Nov 03 17:30:25.116297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116132    6059 cpu_manager.go:414] "containerMap: removing container" podUID="07083bd4-854a-4967-b7a7-a04c488adf34" containerName="cleanup"
Nov 03 17:30:25.116297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116136    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07083bd4-854a-4967-b7a7-a04c488adf34" containerName="cleanup"
Nov 03 17:30:25.116297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116142    6059 cpu_manager.go:414] "containerMap: removing container" podUID="212d9a56-7dfc-434e-b52b-914bb17f0ee9" containerName="cleanup"
Nov 03 17:30:25.116297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116147    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212d9a56-7dfc-434e-b52b-914bb17f0ee9" containerName="cleanup"
Nov 03 17:30:25.116297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116154    6059 cpu_manager.go:414] "containerMap: removing container" podUID="212d9a56-7dfc-434e-b52b-914bb17f0ee9" containerName="cleanup"
Nov 03 17:30:25.116297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116158    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212d9a56-7dfc-434e-b52b-914bb17f0ee9" containerName="cleanup"
Nov 03 17:30:25.116297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116178    6059 cpu_manager.go:414] "containerMap: removing container" podUID="2d9c420a-3483-4dbc-8244-1b324c0fd94f" containerName="cleanup"
Nov 03 17:30:25.116297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116183    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d9c420a-3483-4dbc-8244-1b324c0fd94f" containerName="cleanup"
Nov 03 17:30:25.116297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116257    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="07083bd4-854a-4967-b7a7-a04c488adf34" containerName="cleanup"
Nov 03 17:30:25.116297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116263    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="212d9a56-7dfc-434e-b52b-914bb17f0ee9" containerName="cleanup"
Nov 03 17:30:25.116297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116270    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="212d9a56-7dfc-434e-b52b-914bb17f0ee9" containerName="cleanup"
Nov 03 17:30:25.116297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116280    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d9c420a-3483-4dbc-8244-1b324c0fd94f" containerName="cleanup"
Nov 03 17:30:25.116297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116286    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cf39df4-fab9-4076-89ee-042fdc5476dc" containerName="cleanup"
Nov 03 17:30:25.116297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116290    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="07083bd4-854a-4967-b7a7-a04c488adf34" containerName="cleanup"
Nov 03 17:30:25.116297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116295    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d9c420a-3483-4dbc-8244-1b324c0fd94f" containerName="cleanup"
Nov 03 17:30:25.116297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116303    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d9c420a-3483-4dbc-8244-1b324c0fd94f" containerName="cleanup"
Nov 03 17:30:25.116970 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116308    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cf39df4-fab9-4076-89ee-042fdc5476dc" containerName="cleanup"
Nov 03 17:30:25.116970 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.116316    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cf39df4-fab9-4076-89ee-042fdc5476dc" containerName="cleanup"
Nov 03 17:30:25.119926 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.119905    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.182171 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.182140    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-data\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.182171 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.182177    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-podinfo\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.182417 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.182204    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-token\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.182417 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.182314    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-credential-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.182484 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.182438    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istiod-ca-cert\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.182484 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.182477    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-envoy\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.182549 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.182511    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-workload-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.182583 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.182551    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-workload-certs\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.182681 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.182663    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwktw\" (UniqueName: \"kubernetes.io/projected/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-kube-api-access-xwktw\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.220774 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.220733    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"]
Nov 03 17:30:25.283499 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.283463    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-envoy\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.283708 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.283539    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-workload-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.283708 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.283565    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-workload-certs\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.283708 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.283681    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwktw\" (UniqueName: \"kubernetes.io/projected/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-kube-api-access-xwktw\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.283879 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.283706    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-data\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.283879 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.283740    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-podinfo\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.283879 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.283776    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-token\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.283879 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.283819    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-credential-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.284071 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.283903    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istiod-ca-cert\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.284651 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.284624    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-workload-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.284816 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.284791    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istiod-ca-cert\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.285145 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.285119    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-data\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.285233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.285138    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-credential-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.285233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.285183    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-workload-certs\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.287133 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.287110    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-podinfo\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.287263 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.287247    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-envoy\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.358228 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.358144    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-token\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.358768 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.358747    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwktw\" (UniqueName: \"kubernetes.io/projected/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-kube-api-access-xwktw\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.453377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.450912    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:25.458719 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.458419    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:25.905994 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:30:25.905962    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6c7b1f_0d7d_447f_9d3a_5626b83c4b73.slice/crio-2574dc16995247032318478d0109046ba28458ec892f6df8bbb1e52c6192cc56 WatchSource:0}: Error finding container 2574dc16995247032318478d0109046ba28458ec892f6df8bbb1e52c6192cc56: Status 404 returned error can't find the container with id 2574dc16995247032318478d0109046ba28458ec892f6df8bbb1e52c6192cc56
Nov 03 17:30:25.908828 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.908805    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"]
Nov 03 17:30:25.909066 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.909039    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:30:25.909109 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.909097    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:30:25.909146 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.909131    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:30:25.973733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:25.973697    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj" event={"ID":"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73","Type":"ContainerStarted","Data":"2574dc16995247032318478d0109046ba28458ec892f6df8bbb1e52c6192cc56"}
Nov 03 17:30:26.082289 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:26.082247    6059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 17:30:26.082801 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:26.082305    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx" podUID="012064a7-fd12-465b-bdc8-ef6d4ca289b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 17:30:26.988007 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:26.987961    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj" event={"ID":"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73","Type":"ContainerStarted","Data":"7348867872b9d5e2a56f7bbb4d97654e43557219048d510d805fdc191e1792ac"}
Nov 03 17:30:27.085525 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:27.085481    6059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 17:30:27.085995 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:27.085547    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx" podUID="012064a7-fd12-465b-bdc8-ef6d4ca289b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 17:30:27.211664 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:27.211607    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj" podStartSLOduration=2.211583625 podStartE2EDuration="2.211583625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:27.210411568 +0000 UTC m=+768.008124346" watchObservedRunningTime="2026-11-03 17:30:27.211583625 +0000 UTC m=+768.009296407"
Nov 03 17:30:27.463575 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:27.461733    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:27.475546 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:27.475498    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 17:30:27.475877 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:27.475835    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj" podUID="cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 17:30:28.084380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:28.082894    6059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 17:30:28.084380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:28.082960    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx" podUID="012064a7-fd12-465b-bdc8-ef6d4ca289b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 17:30:28.473388 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:28.465760    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 17:30:28.473388 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:28.465826    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj" podUID="cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 17:30:29.083797 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.083728    6059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 17:30:29.083797 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.083795    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx" podUID="012064a7-fd12-465b-bdc8-ef6d4ca289b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 17:30:29.221558 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.221495    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf"]
Nov 03 17:30:29.222609 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.222580    6059 cpu_manager.go:414] "containerMap: removing container" podUID="07083bd4-854a-4967-b7a7-a04c488adf34" containerName="cleanup"
Nov 03 17:30:29.222786 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.222771    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07083bd4-854a-4967-b7a7-a04c488adf34" containerName="cleanup"
Nov 03 17:30:29.222933 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.222919    6059 cpu_manager.go:414] "containerMap: removing container" podUID="2d9c420a-3483-4dbc-8244-1b324c0fd94f" containerName="cleanup"
Nov 03 17:30:29.223038 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.223026    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d9c420a-3483-4dbc-8244-1b324c0fd94f" containerName="cleanup"
Nov 03 17:30:29.223166 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.223153    6059 cpu_manager.go:414] "containerMap: removing container" podUID="5cf39df4-fab9-4076-89ee-042fdc5476dc" containerName="cleanup"
Nov 03 17:30:29.223242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.223231    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf39df4-fab9-4076-89ee-042fdc5476dc" containerName="cleanup"
Nov 03 17:30:29.223529 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.223503    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="07083bd4-854a-4967-b7a7-a04c488adf34" containerName="cleanup"
Nov 03 17:30:29.223647 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.223634    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="212d9a56-7dfc-434e-b52b-914bb17f0ee9" containerName="cleanup"
Nov 03 17:30:29.227850 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.227831    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf"
Nov 03 17:30:29.289321 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.289278    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf"]
Nov 03 17:30:29.345444 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.337689    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:30:29.370174 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.370125    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2733a212-3155-401f-b9d5-e3800a07bc79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7f136fb-sthsf\" (UID: \"2733a212-3155-401f-b9d5-e3800a07bc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf"
Nov 03 17:30:29.370580 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.370538    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2733a212-3155-401f-b9d5-e3800a07bc79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7f136fb-sthsf\" (UID: \"2733a212-3155-401f-b9d5-e3800a07bc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf"
Nov 03 17:30:29.389384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.374537    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ttrb\" (UniqueName: \"kubernetes.io/projected/2733a212-3155-401f-b9d5-e3800a07bc79-kube-api-access-9ttrb\") pod \"maas-api-revoke-sub-c7f136fb-sthsf\" (UID: \"2733a212-3155-401f-b9d5-e3800a07bc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf"
Nov 03 17:30:29.421640 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.395423    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:30:29.421640 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.395628    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:30:29.421640 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.395742    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:30:29.473487 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.473422    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 17:30:29.474128 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.474077    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj" podUID="cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 17:30:29.491696 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.491651    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ttrb\" (UniqueName: \"kubernetes.io/projected/2733a212-3155-401f-b9d5-e3800a07bc79-kube-api-access-9ttrb\") pod \"maas-api-revoke-sub-c7f136fb-sthsf\" (UID: \"2733a212-3155-401f-b9d5-e3800a07bc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf"
Nov 03 17:30:29.491901 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.491800    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2733a212-3155-401f-b9d5-e3800a07bc79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7f136fb-sthsf\" (UID: \"2733a212-3155-401f-b9d5-e3800a07bc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf"
Nov 03 17:30:29.491977 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.491907    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2733a212-3155-401f-b9d5-e3800a07bc79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7f136fb-sthsf\" (UID: \"2733a212-3155-401f-b9d5-e3800a07bc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf"
Nov 03 17:30:29.492862 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.492833    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2733a212-3155-401f-b9d5-e3800a07bc79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7f136fb-sthsf\" (UID: \"2733a212-3155-401f-b9d5-e3800a07bc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf"
Nov 03 17:30:29.497110 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.497076    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2733a212-3155-401f-b9d5-e3800a07bc79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7f136fb-sthsf\" (UID: \"2733a212-3155-401f-b9d5-e3800a07bc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf"
Nov 03 17:30:29.562924 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.562855    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ttrb\" (UniqueName: \"kubernetes.io/projected/2733a212-3155-401f-b9d5-e3800a07bc79-kube-api-access-9ttrb\") pod \"maas-api-revoke-sub-c7f136fb-sthsf\" (UID: \"2733a212-3155-401f-b9d5-e3800a07bc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf"
Nov 03 17:30:29.582760 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.582479    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf"
Nov 03 17:30:29.982774 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:29.982736    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:30.080649 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:30.080607    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf"]
Nov 03 17:30:30.084003 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:30:30.083973    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2733a212_3155_401f_b9d5_e3800a07bc79.slice/crio-4a6ef733ab5ccf1409aee6d39e9aebda9deb674e48290e8fb6a947651e16f936 WatchSource:0}: Error finding container 4a6ef733ab5ccf1409aee6d39e9aebda9deb674e48290e8fb6a947651e16f936: Status 404 returned error can't find the container with id 4a6ef733ab5ccf1409aee6d39e9aebda9deb674e48290e8fb6a947651e16f936
Nov 03 17:30:30.090830 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:30.089284    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:30:30.090830 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:30.089644    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:30:30.090830 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:30.090790    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:30:30.185982 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:30.185920    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:30.423797 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:30.423757    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:30.463673 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:30.462905    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 17:30:30.463673 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:30.462960    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj" podUID="cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 17:30:31.056403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:31.054677    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf" event={"ID":"2733a212-3155-401f-b9d5-e3800a07bc79","Type":"ContainerStarted","Data":"7b8102c03567df33f4dd331e98bc13be1e24ce5fd5b081f5617e68d600fbe342"}
Nov 03 17:30:31.056403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:31.054758    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf" event={"ID":"2733a212-3155-401f-b9d5-e3800a07bc79","Type":"ContainerStarted","Data":"4a6ef733ab5ccf1409aee6d39e9aebda9deb674e48290e8fb6a947651e16f936"}
Nov 03 17:30:31.175587 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:31.175526    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf" podStartSLOduration=2.17550239 podStartE2EDuration="2.17550239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:31.1736919 +0000 UTC m=+771.971404683" watchObservedRunningTime="2026-11-03 17:30:31.17550239 +0000 UTC m=+771.973215170"
Nov 03 17:30:31.459796 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:31.459768    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 17:30:31.459943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:31.459820    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj" podUID="cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 17:30:32.056781 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:32.056741    6059 generic.go:356] "Generic (PLEG): container finished" podID="2733a212-3155-401f-b9d5-e3800a07bc79" containerID="7b8102c03567df33f4dd331e98bc13be1e24ce5fd5b081f5617e68d600fbe342" exitCode=0
Nov 03 17:30:32.057232 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:32.056823    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf" event={"ID":"2733a212-3155-401f-b9d5-e3800a07bc79","Type":"ContainerDied","Data":"7b8102c03567df33f4dd331e98bc13be1e24ce5fd5b081f5617e68d600fbe342"}
Nov 03 17:30:32.459866 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:32.459830    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 17:30:32.460024 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:32.459892    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj" podUID="cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 17:30:33.231778 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:33.231749    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf"
Nov 03 17:30:33.280894 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:33.280856    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2733a212-3155-401f-b9d5-e3800a07bc79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2733a212-3155-401f-b9d5-e3800a07bc79-maas-api-openshift-ca\") pod \"2733a212-3155-401f-b9d5-e3800a07bc79\" (UID: \"2733a212-3155-401f-b9d5-e3800a07bc79\") "
Nov 03 17:30:33.281094 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:33.280919    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2733a212-3155-401f-b9d5-e3800a07bc79-kube-api-access-9ttrb\" (UniqueName: \"kubernetes.io/projected/2733a212-3155-401f-b9d5-e3800a07bc79-kube-api-access-9ttrb\") pod \"2733a212-3155-401f-b9d5-e3800a07bc79\" (UID: \"2733a212-3155-401f-b9d5-e3800a07bc79\") "
Nov 03 17:30:33.281094 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:33.280987    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2733a212-3155-401f-b9d5-e3800a07bc79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2733a212-3155-401f-b9d5-e3800a07bc79-maas-api-xks-ca\") pod \"2733a212-3155-401f-b9d5-e3800a07bc79\" (UID: \"2733a212-3155-401f-b9d5-e3800a07bc79\") "
Nov 03 17:30:33.281320 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:33.281295    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2733a212-3155-401f-b9d5-e3800a07bc79-maas-api-openshift-ca" pod "2733a212-3155-401f-b9d5-e3800a07bc79" (UID: "2733a212-3155-401f-b9d5-e3800a07bc79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:30:33.283139 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:33.283106    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2733a212-3155-401f-b9d5-e3800a07bc79-maas-api-xks-ca" pod "2733a212-3155-401f-b9d5-e3800a07bc79" (UID: "2733a212-3155-401f-b9d5-e3800a07bc79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:30:33.283236 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:33.283217    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2733a212-3155-401f-b9d5-e3800a07bc79-kube-api-access-9ttrb" pod "2733a212-3155-401f-b9d5-e3800a07bc79" (UID: "2733a212-3155-401f-b9d5-e3800a07bc79"). InnerVolumeSpecName "kube-api-access-9ttrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:30:33.382184 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:33.382143    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2733a212-3155-401f-b9d5-e3800a07bc79-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:30:33.382184 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:33.382183    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ttrb\" (UniqueName: \"kubernetes.io/projected/2733a212-3155-401f-b9d5-e3800a07bc79-kube-api-access-9ttrb\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:30:33.382413 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:33.382198    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2733a212-3155-401f-b9d5-e3800a07bc79-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:30:33.956950 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:33.956918    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:33.957252 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:33.957225    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:33.958389 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:33.958326    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:30:34.070989 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:34.070963    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf"
Nov 03 17:30:34.071173 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:34.070969    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf" event={"ID":"2733a212-3155-401f-b9d5-e3800a07bc79","Type":"ContainerDied","Data":"4a6ef733ab5ccf1409aee6d39e9aebda9deb674e48290e8fb6a947651e16f936"}
Nov 03 17:30:34.071173 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:34.071075    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a6ef733ab5ccf1409aee6d39e9aebda9deb674e48290e8fb6a947651e16f936"
Nov 03 17:30:34.463548 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:34.463508    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:34.727157 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:34.727074    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:36.862683 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:30:36.862641    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:30:39.247122 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:39.247081    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:43.989305 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:43.989261    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:46.008655 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:46.008620    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:48.545926 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:48.545891    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:48.827603 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:48.827493    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:53.440067 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:53.440029    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:53.545598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:53.545563    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"]
Nov 03 17:30:53.660774 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:53.660730    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:54.186230 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:54.186193    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:30:58.009410 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:30:58.009331    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:02.450017 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:02.449977    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a507cc28-l2n25"]
Nov 03 17:31:02.450550 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:02.450534    6059 cpu_manager.go:414] "containerMap: removing container" podUID="2733a212-3155-401f-b9d5-e3800a07bc79" containerName="revoke-keys"
Nov 03 17:31:02.450597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:02.450552    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2733a212-3155-401f-b9d5-e3800a07bc79" containerName="revoke-keys"
Nov 03 17:31:02.459725 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:02.451209    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2733a212-3155-401f-b9d5-e3800a07bc79" containerName="revoke-keys"
Nov 03 17:31:02.459725 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:02.457726    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a507cc28-l2n25"
Nov 03 17:31:02.515661 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:02.515627    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:31:02.515661 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:02.515638    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:31:02.522694 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:02.522640    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:02.548529 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:02.548491    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61a3bf97-aa63-49f9-b2dd-75414055f4ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a507cc28-l2n25\" (UID: \"61a3bf97-aa63-49f9-b2dd-75414055f4ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a507cc28-l2n25"
Nov 03 17:31:02.549101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:02.549067    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61a3bf97-aa63-49f9-b2dd-75414055f4ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a507cc28-l2n25\" (UID: \"61a3bf97-aa63-49f9-b2dd-75414055f4ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a507cc28-l2n25"
Nov 03 17:31:02.549266 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:02.549249    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z95h2\" (UniqueName: \"kubernetes.io/projected/61a3bf97-aa63-49f9-b2dd-75414055f4ab-kube-api-access-z95h2\") pod \"maas-api-revoke-sub-a507cc28-l2n25\" (UID: \"61a3bf97-aa63-49f9-b2dd-75414055f4ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a507cc28-l2n25"
Nov 03 17:31:02.565184 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:02.565144    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:31:02.565351 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:02.565158    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:31:02.569526 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:02.569496    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a507cc28-l2n25"]
Nov 03 17:31:02.651175 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:02.651118    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61a3bf97-aa63-49f9-b2dd-75414055f4ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a507cc28-l2n25\" (UID: \"61a3bf97-aa63-49f9-b2dd-75414055f4ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a507cc28-l2n25"
Nov 03 17:31:02.651392 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:02.651184    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z95h2\" (UniqueName: \"kubernetes.io/projected/61a3bf97-aa63-49f9-b2dd-75414055f4ab-kube-api-access-z95h2\") pod \"maas-api-revoke-sub-a507cc28-l2n25\" (UID: \"61a3bf97-aa63-49f9-b2dd-75414055f4ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a507cc28-l2n25"
Nov 03 17:31:02.651445 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:02.651403    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61a3bf97-aa63-49f9-b2dd-75414055f4ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a507cc28-l2n25\" (UID: \"61a3bf97-aa63-49f9-b2dd-75414055f4ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a507cc28-l2n25"
Nov 03 17:31:02.652048 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:02.652023    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61a3bf97-aa63-49f9-b2dd-75414055f4ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a507cc28-l2n25\" (UID: \"61a3bf97-aa63-49f9-b2dd-75414055f4ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a507cc28-l2n25"
Nov 03 17:31:02.653580 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:02.653555    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61a3bf97-aa63-49f9-b2dd-75414055f4ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a507cc28-l2n25\" (UID: \"61a3bf97-aa63-49f9-b2dd-75414055f4ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a507cc28-l2n25"
Nov 03 17:31:02.715446 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:02.715410    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z95h2\" (UniqueName: \"kubernetes.io/projected/61a3bf97-aa63-49f9-b2dd-75414055f4ab-kube-api-access-z95h2\") pod \"maas-api-revoke-sub-a507cc28-l2n25\" (UID: \"61a3bf97-aa63-49f9-b2dd-75414055f4ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a507cc28-l2n25"
Nov 03 17:31:02.773614 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:02.773581    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a507cc28-l2n25"
Nov 03 17:31:03.149021 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:31:03.148982    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a3bf97_aa63_49f9_b2dd_75414055f4ab.slice/crio-082831f7f99b44452dbd61ce6b119bea234898a1ce74967739b1644ec30a91f1 WatchSource:0}: Error finding container 082831f7f99b44452dbd61ce6b119bea234898a1ce74967739b1644ec30a91f1: Status 404 returned error can't find the container with id 082831f7f99b44452dbd61ce6b119bea234898a1ce74967739b1644ec30a91f1
Nov 03 17:31:03.151940 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:03.151916    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a507cc28-l2n25"]
Nov 03 17:31:03.263203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:03.263163    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a507cc28-l2n25" event={"ID":"61a3bf97-aa63-49f9-b2dd-75414055f4ab","Type":"ContainerStarted","Data":"082831f7f99b44452dbd61ce6b119bea234898a1ce74967739b1644ec30a91f1"}
Nov 03 17:31:04.271524 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:04.271487    6059 generic.go:356] "Generic (PLEG): container finished" podID="61a3bf97-aa63-49f9-b2dd-75414055f4ab" containerID="577f91781d042b6e6b83748ab5362abdbe79013a48fcb4c568ed24fe7256166a" exitCode=0
Nov 03 17:31:04.271951 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:04.271577    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a507cc28-l2n25" event={"ID":"61a3bf97-aa63-49f9-b2dd-75414055f4ab","Type":"ContainerDied","Data":"577f91781d042b6e6b83748ab5362abdbe79013a48fcb4c568ed24fe7256166a"}
Nov 03 17:31:05.434270 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:05.434246    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a507cc28-l2n25"
Nov 03 17:31:05.584111 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:05.584009    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a3bf97-aa63-49f9-b2dd-75414055f4ab-kube-api-access-z95h2\" (UniqueName: \"kubernetes.io/projected/61a3bf97-aa63-49f9-b2dd-75414055f4ab-kube-api-access-z95h2\") pod \"61a3bf97-aa63-49f9-b2dd-75414055f4ab\" (UID: \"61a3bf97-aa63-49f9-b2dd-75414055f4ab\") "
Nov 03 17:31:05.584111 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:05.584045    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61a3bf97-aa63-49f9-b2dd-75414055f4ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61a3bf97-aa63-49f9-b2dd-75414055f4ab-maas-api-openshift-ca\") pod \"61a3bf97-aa63-49f9-b2dd-75414055f4ab\" (UID: \"61a3bf97-aa63-49f9-b2dd-75414055f4ab\") "
Nov 03 17:31:05.584111 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:05.584088    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61a3bf97-aa63-49f9-b2dd-75414055f4ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61a3bf97-aa63-49f9-b2dd-75414055f4ab-maas-api-xks-ca\") pod \"61a3bf97-aa63-49f9-b2dd-75414055f4ab\" (UID: \"61a3bf97-aa63-49f9-b2dd-75414055f4ab\") "
Nov 03 17:31:05.584499 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:05.584474    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a3bf97-aa63-49f9-b2dd-75414055f4ab-maas-api-openshift-ca" pod "61a3bf97-aa63-49f9-b2dd-75414055f4ab" (UID: "61a3bf97-aa63-49f9-b2dd-75414055f4ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:31:05.584625 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:05.584609    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61a3bf97-aa63-49f9-b2dd-75414055f4ab-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:31:05.586211 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:05.586184    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61a3bf97-aa63-49f9-b2dd-75414055f4ab-maas-api-xks-ca" pod "61a3bf97-aa63-49f9-b2dd-75414055f4ab" (UID: "61a3bf97-aa63-49f9-b2dd-75414055f4ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:31:05.586641 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:05.586625    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a3bf97-aa63-49f9-b2dd-75414055f4ab-kube-api-access-z95h2" pod "61a3bf97-aa63-49f9-b2dd-75414055f4ab" (UID: "61a3bf97-aa63-49f9-b2dd-75414055f4ab"). InnerVolumeSpecName "kube-api-access-z95h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:31:05.686084 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:05.686032    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z95h2\" (UniqueName: \"kubernetes.io/projected/61a3bf97-aa63-49f9-b2dd-75414055f4ab-kube-api-access-z95h2\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:31:05.686084 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:05.686076    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61a3bf97-aa63-49f9-b2dd-75414055f4ab-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:31:06.291397 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:06.288589    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a507cc28-l2n25" event={"ID":"61a3bf97-aa63-49f9-b2dd-75414055f4ab","Type":"ContainerDied","Data":"082831f7f99b44452dbd61ce6b119bea234898a1ce74967739b1644ec30a91f1"}
Nov 03 17:31:06.291397 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:06.288614    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a507cc28-l2n25"
Nov 03 17:31:06.291397 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:06.288678    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="082831f7f99b44452dbd61ce6b119bea234898a1ce74967739b1644ec30a91f1"
Nov 03 17:31:06.880303 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:31:06.880220    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:31:07.151935 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:07.151833    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:08.566958 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:08.566916    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w" podUID="38247383-766c-4605-a65b-90a7c5763344" containerName="main" containerID="cri-o://bc8fb24883787356609ed26f480f5fe54402d96d4cee839f7b5603aaa7caa6f9" gracePeriod=45
Nov 03 17:31:08.835184 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:08.835122    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:31:08.928187 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:08.928149    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-home\" (UniqueName: \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-home\") pod \"38247383-766c-4605-a65b-90a7c5763344\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") "
Nov 03 17:31:08.928405 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:08.928246    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-tmp-dir\") pod \"38247383-766c-4605-a65b-90a7c5763344\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") "
Nov 03 17:31:08.928405 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:08.928280    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-model-cache\") pod \"38247383-766c-4605-a65b-90a7c5763344\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") "
Nov 03 17:31:08.928405 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:08.928348    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38247383-766c-4605-a65b-90a7c5763344-kube-api-access-vppst\" (UniqueName: \"kubernetes.io/projected/38247383-766c-4605-a65b-90a7c5763344-kube-api-access-vppst\") pod \"38247383-766c-4605-a65b-90a7c5763344\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") "
Nov 03 17:31:08.928571 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:08.928424    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-dshm\" (UniqueName: \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-dshm\") pod \"38247383-766c-4605-a65b-90a7c5763344\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") "
Nov 03 17:31:08.928571 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:08.928448    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-home" pod "38247383-766c-4605-a65b-90a7c5763344" (UID: "38247383-766c-4605-a65b-90a7c5763344"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:31:08.928571 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:08.928518    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38247383-766c-4605-a65b-90a7c5763344-tls-certs\" (UniqueName: \"kubernetes.io/secret/38247383-766c-4605-a65b-90a7c5763344-tls-certs\") pod \"38247383-766c-4605-a65b-90a7c5763344\" (UID: \"38247383-766c-4605-a65b-90a7c5763344\") "
Nov 03 17:31:08.928571 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:08.928543    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-tmp-dir" pod "38247383-766c-4605-a65b-90a7c5763344" (UID: "38247383-766c-4605-a65b-90a7c5763344"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:31:08.928759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:08.928619    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-model-cache" pod "38247383-766c-4605-a65b-90a7c5763344" (UID: "38247383-766c-4605-a65b-90a7c5763344"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:31:08.928956 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:08.928938    6059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-home\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:31:08.929019 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:08.928963    6059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-tmp-dir\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:31:08.929019 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:08.928977    6059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-model-cache\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:31:08.930736 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:08.930701    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38247383-766c-4605-a65b-90a7c5763344-kube-api-access-vppst" pod "38247383-766c-4605-a65b-90a7c5763344" (UID: "38247383-766c-4605-a65b-90a7c5763344"). InnerVolumeSpecName "kube-api-access-vppst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:31:08.930835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:08.930739    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-dshm" pod "38247383-766c-4605-a65b-90a7c5763344" (UID: "38247383-766c-4605-a65b-90a7c5763344"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:31:08.930835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:08.930797    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38247383-766c-4605-a65b-90a7c5763344-tls-certs" pod "38247383-766c-4605-a65b-90a7c5763344" (UID: "38247383-766c-4605-a65b-90a7c5763344"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:31:09.030584 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:09.030554    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vppst\" (UniqueName: \"kubernetes.io/projected/38247383-766c-4605-a65b-90a7c5763344-kube-api-access-vppst\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:31:09.030584 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:09.030581    6059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38247383-766c-4605-a65b-90a7c5763344-dshm\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:31:09.030584 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:09.030590    6059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38247383-766c-4605-a65b-90a7c5763344-tls-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:31:09.309681 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:09.309644    6059 generic.go:356] "Generic (PLEG): container finished" podID="38247383-766c-4605-a65b-90a7c5763344" containerID="bc8fb24883787356609ed26f480f5fe54402d96d4cee839f7b5603aaa7caa6f9" exitCode=0
Nov 03 17:31:09.309681 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:09.309685    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"
Nov 03 17:31:09.309929 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:09.309740    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w" event={"ID":"38247383-766c-4605-a65b-90a7c5763344","Type":"ContainerDied","Data":"bc8fb24883787356609ed26f480f5fe54402d96d4cee839f7b5603aaa7caa6f9"}
Nov 03 17:31:09.309929 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:09.309784    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w" event={"ID":"38247383-766c-4605-a65b-90a7c5763344","Type":"ContainerDied","Data":"0dc708a2b3d1052ae96b06b8bf9dc870b70560c6a2df2f804d3f3c0c139ca906"}
Nov 03 17:31:09.309929 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:09.309801    6059 scope.go:122] "RemoveContainer" containerID="bc8fb24883787356609ed26f480f5fe54402d96d4cee839f7b5603aaa7caa6f9"
Nov 03 17:31:09.320283 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:09.320262    6059 scope.go:122] "RemoveContainer" containerID="bc8fb24883787356609ed26f480f5fe54402d96d4cee839f7b5603aaa7caa6f9"
Nov 03 17:31:09.320609 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:31:09.320589    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc8fb24883787356609ed26f480f5fe54402d96d4cee839f7b5603aaa7caa6f9\": container with ID starting with bc8fb24883787356609ed26f480f5fe54402d96d4cee839f7b5603aaa7caa6f9 not found: ID does not exist" containerID="bc8fb24883787356609ed26f480f5fe54402d96d4cee839f7b5603aaa7caa6f9"
Nov 03 17:31:09.320695 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:09.320615    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc8fb24883787356609ed26f480f5fe54402d96d4cee839f7b5603aaa7caa6f9"} err="failed to get container status \"bc8fb24883787356609ed26f480f5fe54402d96d4cee839f7b5603aaa7caa6f9\": rpc error: code = NotFound desc = could not find container \"bc8fb24883787356609ed26f480f5fe54402d96d4cee839f7b5603aaa7caa6f9\": container with ID starting with bc8fb24883787356609ed26f480f5fe54402d96d4cee839f7b5603aaa7caa6f9 not found: ID does not exist"
Nov 03 17:31:09.481728 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:09.481690    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"]
Nov 03 17:31:09.484042 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:09.484011    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-8fc5ec/praxis-route-8fc5ec-kserve-64f58fb64f-5hh6w"]
Nov 03 17:31:09.735473 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:09.735434    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:09.987157 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:09.987070    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:11.360226 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:11.360189    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38247383-766c-4605-a65b-90a7c5763344" path="/var/lib/kubelet/pods/38247383-766c-4605-a65b-90a7c5763344/volumes"
Nov 03 17:31:11.825443 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:11.825408    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:13.257626 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:13.257574    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:13.693119 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:13.693008    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:14.268871 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.268823    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"]
Nov 03 17:31:14.269643 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.269620    6059 cpu_manager.go:414] "containerMap: removing container" podUID="61a3bf97-aa63-49f9-b2dd-75414055f4ab" containerName="revoke-keys"
Nov 03 17:31:14.269755 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.269644    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a3bf97-aa63-49f9-b2dd-75414055f4ab" containerName="revoke-keys"
Nov 03 17:31:14.269755 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.269662    6059 cpu_manager.go:414] "containerMap: removing container" podUID="38247383-766c-4605-a65b-90a7c5763344" containerName="main"
Nov 03 17:31:14.269755 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.269669    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38247383-766c-4605-a65b-90a7c5763344" containerName="main"
Nov 03 17:31:14.269917 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.269801    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a3bf97-aa63-49f9-b2dd-75414055f4ab" containerName="revoke-keys"
Nov 03 17:31:14.269917 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.269815    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="38247383-766c-4605-a65b-90a7c5763344" containerName="main"
Nov 03 17:31:14.274867 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.274846    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.308436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.308394    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltnxq\" (UniqueName: \"kubernetes.io/projected/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-kube-api-access-ltnxq\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.308584 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.308445    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-workload-socket\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.308584 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.308532    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-credential-socket\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.308584 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.308570    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-podinfo\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.308772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.308618    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-envoy\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.308772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.308692    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-data\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.308772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.308721    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-token\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.308912 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.308870    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istiod-ca-cert\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.308964 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.308926    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-workload-certs\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.345993 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.329596    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e34dd2-openshift-default-dockercfg-66s84\""
Nov 03 17:31:14.384602 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.384565    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"]
Nov 03 17:31:14.406765 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.406732    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:14.410738 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.410703    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istiod-ca-cert\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.410885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.410772    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-workload-certs\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.410885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.410850    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltnxq\" (UniqueName: \"kubernetes.io/projected/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-kube-api-access-ltnxq\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.411039 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.410883    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-workload-socket\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.411039 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.410921    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-credential-socket\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.411039 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.410946    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-podinfo\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.411039 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.410987    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-envoy\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.411263 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.411056    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-data\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.411263 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.411085    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-token\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.411607 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.411513    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istiod-ca-cert\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.411803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.411779    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-workload-certs\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.412069 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.412015    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-credential-socket\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.412431 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.412405    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-data\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.412556 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.412479    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-workload-socket\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.415020 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.414991    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-envoy\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.415601 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.415576    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-podinfo\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.515201 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.515135    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-token\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.515782 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.515756    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltnxq\" (UniqueName: \"kubernetes.io/projected/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-kube-api-access-ltnxq\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.612091 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.611995    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:14.711968 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:14.711935    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:15.069863 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:15.069822    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"]
Nov 03 17:31:15.100428 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:31:15.100398    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db8f565_8a57_45ff_b31c_aa5a4a7093fa.slice/crio-412a3b626564d7467d90816dd8a70a3f2562d08e0d634acce91a44de3734b690 WatchSource:0}: Error finding container 412a3b626564d7467d90816dd8a70a3f2562d08e0d634acce91a44de3734b690: Status 404 returned error can't find the container with id 412a3b626564d7467d90816dd8a70a3f2562d08e0d634acce91a44de3734b690
Nov 03 17:31:15.102833 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:15.102792    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:31:15.102947 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:15.102883    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:31:15.102947 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:15.102928    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:31:15.381562 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:15.381468    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns" event={"ID":"3db8f565-8a57-45ff-b31c-aa5a4a7093fa","Type":"ContainerStarted","Data":"d1f7556315653fbd1a28346e5983a15d8ec09118a3ee7cd9543a599f8e0d004f"}
Nov 03 17:31:15.381562 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:15.381524    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns" event={"ID":"3db8f565-8a57-45ff-b31c-aa5a4a7093fa","Type":"ContainerStarted","Data":"412a3b626564d7467d90816dd8a70a3f2562d08e0d634acce91a44de3734b690"}
Nov 03 17:31:15.520913 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:15.520851    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns" podStartSLOduration=1.520833567 podStartE2EDuration="1.520833567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:31:15.520090058 +0000 UTC m=+816.317802839" watchObservedRunningTime="2026-11-03 17:31:15.520833567 +0000 UTC m=+816.318546348"
Nov 03 17:31:15.613050 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:15.613007    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:16.618546 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:16.618518    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:16.668780 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:16.668708    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:17.395471 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:17.395393    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:17.396855 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:17.396828    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:31:18.074254 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:18.074217    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc"]
Nov 03 17:31:18.079414 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:18.079385    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc"
Nov 03 17:31:18.134347 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:18.134314    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:31:18.135087 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:18.134657    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:31:18.135281 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:18.135255    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:31:18.135444 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:18.135351    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:31:18.138986 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:18.138959    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc"]
Nov 03 17:31:18.182678 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:18.182639    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b27261-8577-4b18-af00-8aebc33b13a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e57e242-4xlgc\" (UID: \"96b27261-8577-4b18-af00-8aebc33b13a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc"
Nov 03 17:31:18.182678 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:18.182683    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b27261-8577-4b18-af00-8aebc33b13a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e57e242-4xlgc\" (UID: \"96b27261-8577-4b18-af00-8aebc33b13a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc"
Nov 03 17:31:18.182892 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:18.182707    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrd45\" (UniqueName: \"kubernetes.io/projected/96b27261-8577-4b18-af00-8aebc33b13a7-kube-api-access-vrd45\") pod \"maas-api-revoke-sub-3e57e242-4xlgc\" (UID: \"96b27261-8577-4b18-af00-8aebc33b13a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc"
Nov 03 17:31:18.283986 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:18.283945    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b27261-8577-4b18-af00-8aebc33b13a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e57e242-4xlgc\" (UID: \"96b27261-8577-4b18-af00-8aebc33b13a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc"
Nov 03 17:31:18.283986 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:18.283991    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrd45\" (UniqueName: \"kubernetes.io/projected/96b27261-8577-4b18-af00-8aebc33b13a7-kube-api-access-vrd45\") pod \"maas-api-revoke-sub-3e57e242-4xlgc\" (UID: \"96b27261-8577-4b18-af00-8aebc33b13a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc"
Nov 03 17:31:18.284203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:18.284164    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b27261-8577-4b18-af00-8aebc33b13a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e57e242-4xlgc\" (UID: \"96b27261-8577-4b18-af00-8aebc33b13a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc"
Nov 03 17:31:18.284842 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:18.284814    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b27261-8577-4b18-af00-8aebc33b13a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e57e242-4xlgc\" (UID: \"96b27261-8577-4b18-af00-8aebc33b13a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc"
Nov 03 17:31:18.286639 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:18.286602    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b27261-8577-4b18-af00-8aebc33b13a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e57e242-4xlgc\" (UID: \"96b27261-8577-4b18-af00-8aebc33b13a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc"
Nov 03 17:31:18.350319 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:18.350237    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrd45\" (UniqueName: \"kubernetes.io/projected/96b27261-8577-4b18-af00-8aebc33b13a7-kube-api-access-vrd45\") pod \"maas-api-revoke-sub-3e57e242-4xlgc\" (UID: \"96b27261-8577-4b18-af00-8aebc33b13a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc"
Nov 03 17:31:18.394418 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:18.394346    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc"
Nov 03 17:31:18.427322 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:18.427288    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:18.641249 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:18.641169    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:18.783986 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:31:18.783950    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b27261_8577_4b18_af00_8aebc33b13a7.slice/crio-ff1412286567b055cc5b510eeae82804d2d545db5f9bdad41db80ee8a7ab3968 WatchSource:0}: Error finding container ff1412286567b055cc5b510eeae82804d2d545db5f9bdad41db80ee8a7ab3968: Status 404 returned error can't find the container with id ff1412286567b055cc5b510eeae82804d2d545db5f9bdad41db80ee8a7ab3968
Nov 03 17:31:18.786874 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:18.786848    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc"]
Nov 03 17:31:19.411072 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:19.411027    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc" event={"ID":"96b27261-8577-4b18-af00-8aebc33b13a7","Type":"ContainerStarted","Data":"7ce580a5a776a21f2405c5875b3af60cf4d4711c82a7dea6fa7bd1be594a79f4"}
Nov 03 17:31:19.411513 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:19.411096    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc" event={"ID":"96b27261-8577-4b18-af00-8aebc33b13a7","Type":"ContainerStarted","Data":"ff1412286567b055cc5b510eeae82804d2d545db5f9bdad41db80ee8a7ab3968"}
Nov 03 17:31:19.547773 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:19.547711    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc" podStartSLOduration=1.54769443 podStartE2EDuration="1.54769443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:31:19.54521983 +0000 UTC m=+820.342932612" watchObservedRunningTime="2026-11-03 17:31:19.54769443 +0000 UTC m=+820.345407211"
Nov 03 17:31:20.419420 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:20.419314    6059 generic.go:356] "Generic (PLEG): container finished" podID="96b27261-8577-4b18-af00-8aebc33b13a7" containerID="7ce580a5a776a21f2405c5875b3af60cf4d4711c82a7dea6fa7bd1be594a79f4" exitCode=0
Nov 03 17:31:20.419420 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:20.419404    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc" event={"ID":"96b27261-8577-4b18-af00-8aebc33b13a7","Type":"ContainerDied","Data":"7ce580a5a776a21f2405c5875b3af60cf4d4711c82a7dea6fa7bd1be594a79f4"}
Nov 03 17:31:20.421181 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:20.421159    6059 generic.go:356] "Generic (PLEG): container finished" podID="eb7314fe-6d72-4e51-8cdc-acfcce172a1c" containerID="6c6d1fd85a52eddd9360384fa3b3428da88fdf0b81e3d9ba4f4e47d8b5e80bfd" exitCode=0
Nov 03 17:31:20.421294 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:20.421198    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg" event={"ID":"eb7314fe-6d72-4e51-8cdc-acfcce172a1c","Type":"ContainerDied","Data":"6c6d1fd85a52eddd9360384fa3b3428da88fdf0b81e3d9ba4f4e47d8b5e80bfd"}
Nov 03 17:31:20.421575 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:20.421561    6059 scope.go:122] "RemoveContainer" containerID="6c6d1fd85a52eddd9360384fa3b3428da88fdf0b81e3d9ba4f4e47d8b5e80bfd"
Nov 03 17:31:20.422972 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:20.422944    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:31:20.423060 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:20.423025    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:31:20.423133 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:20.423074    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:31:21.429274 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.429241    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg" event={"ID":"eb7314fe-6d72-4e51-8cdc-acfcce172a1c","Type":"ContainerStarted","Data":"308332826f9a21a6a96bae42e28098c17477478b49f07103a8829c9893a287db"}
Nov 03 17:31:21.461237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.461208    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:21.616971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.616944    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc"
Nov 03 17:31:21.655908 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.655867    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:21.730769 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.724704    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b27261-8577-4b18-af00-8aebc33b13a7-kube-api-access-vrd45\" (UniqueName: \"kubernetes.io/projected/96b27261-8577-4b18-af00-8aebc33b13a7-kube-api-access-vrd45\") pod \"96b27261-8577-4b18-af00-8aebc33b13a7\" (UID: \"96b27261-8577-4b18-af00-8aebc33b13a7\") "
Nov 03 17:31:21.730769 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.724887    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96b27261-8577-4b18-af00-8aebc33b13a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b27261-8577-4b18-af00-8aebc33b13a7-maas-api-openshift-ca\") pod \"96b27261-8577-4b18-af00-8aebc33b13a7\" (UID: \"96b27261-8577-4b18-af00-8aebc33b13a7\") "
Nov 03 17:31:21.730769 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.724979    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96b27261-8577-4b18-af00-8aebc33b13a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b27261-8577-4b18-af00-8aebc33b13a7-maas-api-xks-ca\") pod \"96b27261-8577-4b18-af00-8aebc33b13a7\" (UID: \"96b27261-8577-4b18-af00-8aebc33b13a7\") "
Nov 03 17:31:21.736241 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.734715    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b27261-8577-4b18-af00-8aebc33b13a7-maas-api-openshift-ca" pod "96b27261-8577-4b18-af00-8aebc33b13a7" (UID: "96b27261-8577-4b18-af00-8aebc33b13a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:31:21.759103 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.759046    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b27261-8577-4b18-af00-8aebc33b13a7-kube-api-access-vrd45" pod "96b27261-8577-4b18-af00-8aebc33b13a7" (UID: "96b27261-8577-4b18-af00-8aebc33b13a7"). InnerVolumeSpecName "kube-api-access-vrd45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:31:21.768698 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.768659    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96b27261-8577-4b18-af00-8aebc33b13a7-maas-api-xks-ca" pod "96b27261-8577-4b18-af00-8aebc33b13a7" (UID: "96b27261-8577-4b18-af00-8aebc33b13a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:31:21.881966 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.881847    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b27261-8577-4b18-af00-8aebc33b13a7-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:31:21.881966 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.881887    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrd45\" (UniqueName: \"kubernetes.io/projected/96b27261-8577-4b18-af00-8aebc33b13a7-kube-api-access-vrd45\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:31:21.881966 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.881902    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b27261-8577-4b18-af00-8aebc33b13a7-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:31:21.883650 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.883620    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x"]
Nov 03 17:31:21.884198 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.884179    6059 cpu_manager.go:414] "containerMap: removing container" podUID="96b27261-8577-4b18-af00-8aebc33b13a7" containerName="revoke-keys"
Nov 03 17:31:21.884287 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.884199    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b27261-8577-4b18-af00-8aebc33b13a7" containerName="revoke-keys"
Nov 03 17:31:21.884426 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.884411    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b27261-8577-4b18-af00-8aebc33b13a7" containerName="revoke-keys"
Nov 03 17:31:21.887778 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.887758    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x"
Nov 03 17:31:21.983438 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.982773    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e34dd2-dockercfg-2zd85\""
Nov 03 17:31:21.983438 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.982864    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e34dd2\""
Nov 03 17:31:21.990374 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.984101    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e65883-b429-48a8-85df-f5122c5aa67d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x\" (UID: \"15e65883-b429-48a8-85df-f5122c5aa67d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x"
Nov 03 17:31:21.990374 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.984139    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbl8k\" (UniqueName: \"kubernetes.io/projected/15e65883-b429-48a8-85df-f5122c5aa67d-kube-api-access-wbl8k\") pod \"payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x\" (UID: \"15e65883-b429-48a8-85df-f5122c5aa67d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x"
Nov 03 17:31:21.990374 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:21.985140    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x"]
Nov 03 17:31:22.094464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:22.092088    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e65883-b429-48a8-85df-f5122c5aa67d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x\" (UID: \"15e65883-b429-48a8-85df-f5122c5aa67d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x"
Nov 03 17:31:22.094464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:22.092142    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbl8k\" (UniqueName: \"kubernetes.io/projected/15e65883-b429-48a8-85df-f5122c5aa67d-kube-api-access-wbl8k\") pod \"payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x\" (UID: \"15e65883-b429-48a8-85df-f5122c5aa67d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x"
Nov 03 17:31:22.094464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:22.094398    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e65883-b429-48a8-85df-f5122c5aa67d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x\" (UID: \"15e65883-b429-48a8-85df-f5122c5aa67d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x"
Nov 03 17:31:22.095094 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:22.095063    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh"]
Nov 03 17:31:22.105751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:22.100626    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh"
Nov 03 17:31:22.118403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:22.114644    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:31:22.118403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:22.115789    6059 patch_prober.go:29] interesting pod/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 17:31:22.118403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:22.115887    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg" podUID="eb7314fe-6d72-4e51-8cdc-acfcce172a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 17:31:22.122810 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:22.122781    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh"]
Nov 03 17:31:22.176101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:22.175715    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbl8k\" (UniqueName: \"kubernetes.io/projected/15e65883-b429-48a8-85df-f5122c5aa67d-kube-api-access-wbl8k\") pod \"payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x\" (UID: \"15e65883-b429-48a8-85df-f5122c5aa67d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x"
Nov 03 17:31:22.266555 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:22.266308    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x"
Nov 03 17:31:22.293730 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:22.293696    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7d5da9-069f-4387-b93d-3f3687d7a43a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh\" (UID: \"9f7d5da9-069f-4387-b93d-3f3687d7a43a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh"
Nov 03 17:31:22.293893 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:22.293787    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv48g\" (UniqueName: \"kubernetes.io/projected/9f7d5da9-069f-4387-b93d-3f3687d7a43a-kube-api-access-jv48g\") pod \"payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh\" (UID: \"9f7d5da9-069f-4387-b93d-3f3687d7a43a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh"
Nov 03 17:31:22.398066 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:22.396240    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv48g\" (UniqueName: \"kubernetes.io/projected/9f7d5da9-069f-4387-b93d-3f3687d7a43a-kube-api-access-jv48g\") pod \"payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh\" (UID: \"9f7d5da9-069f-4387-b93d-3f3687d7a43a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh"
Nov 03 17:31:22.398066 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:22.396445    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7d5da9-069f-4387-b93d-3f3687d7a43a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh\" (UID: \"9f7d5da9-069f-4387-b93d-3f3687d7a43a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh"
Nov 03 17:31:22.398066 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:22.397403    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7d5da9-069f-4387-b93d-3f3687d7a43a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh\" (UID: \"9f7d5da9-069f-4387-b93d-3f3687d7a43a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh"
Nov 03 17:31:22.516241 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:22.516123    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv48g\" (UniqueName: \"kubernetes.io/projected/9f7d5da9-069f-4387-b93d-3f3687d7a43a-kube-api-access-jv48g\") pod \"payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh\" (UID: \"9f7d5da9-069f-4387-b93d-3f3687d7a43a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh"
Nov 03 17:31:22.528469 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:22.527809    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc"
Nov 03 17:31:22.528642 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:22.528552    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc" event={"ID":"96b27261-8577-4b18-af00-8aebc33b13a7","Type":"ContainerDied","Data":"ff1412286567b055cc5b510eeae82804d2d545db5f9bdad41db80ee8a7ab3968"}
Nov 03 17:31:22.528642 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:22.528603    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff1412286567b055cc5b510eeae82804d2d545db5f9bdad41db80ee8a7ab3968"
Nov 03 17:31:22.710153 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:22.705818    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x"]
Nov 03 17:31:22.737907 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:22.737597    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh"
Nov 03 17:31:23.116466 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:23.113524    6059 patch_prober.go:29] interesting pod/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 17:31:23.116466 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:23.113588    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg" podUID="eb7314fe-6d72-4e51-8cdc-acfcce172a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 17:31:23.315240 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:31:23.315203    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7d5da9_069f_4387_b93d_3f3687d7a43a.slice/crio-fdd7d6b481147b15deb541612e88dcbd6780171bec4b6ab85b0e87f032d5dd46 WatchSource:0}: Error finding container fdd7d6b481147b15deb541612e88dcbd6780171bec4b6ab85b0e87f032d5dd46: Status 404 returned error can't find the container with id fdd7d6b481147b15deb541612e88dcbd6780171bec4b6ab85b0e87f032d5dd46
Nov 03 17:31:23.318891 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:23.318864    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh"]
Nov 03 17:31:23.558376 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:23.558321    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh" event={"ID":"9f7d5da9-069f-4387-b93d-3f3687d7a43a","Type":"ContainerStarted","Data":"e4de9fb28f654396dfaf7a76570b3a18e59f4595c7010acf53168a477f2c78af"}
Nov 03 17:31:23.558830 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:23.558405    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh" event={"ID":"9f7d5da9-069f-4387-b93d-3f3687d7a43a","Type":"ContainerStarted","Data":"fdd7d6b481147b15deb541612e88dcbd6780171bec4b6ab85b0e87f032d5dd46"}
Nov 03 17:31:23.558830 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:23.558434    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh"
Nov 03 17:31:23.560136 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:23.560108    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x" event={"ID":"15e65883-b429-48a8-85df-f5122c5aa67d","Type":"ContainerStarted","Data":"9d31a7d0d7f3a2c9674accf9cb41a6b2f50ee0f810b96d58b4b8cf4425fa535e"}
Nov 03 17:31:23.560256 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:23.560152    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x" event={"ID":"15e65883-b429-48a8-85df-f5122c5aa67d","Type":"ContainerStarted","Data":"e24e894b6be883564b9cea62b94ef9a57dd94d2aa2b4fbace10532c716fb8319"}
Nov 03 17:31:23.560256 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:23.560181    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x"
Nov 03 17:31:23.683285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:23.683174    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh" podStartSLOduration=2.683159306 podStartE2EDuration="2.683159306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:31:23.682083535 +0000 UTC m=+824.479796316" watchObservedRunningTime="2026-11-03 17:31:23.683159306 +0000 UTC m=+824.480872106"
Nov 03 17:31:23.812789 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:23.812738    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x" podStartSLOduration=2.812720331 podStartE2EDuration="2.812720331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:31:23.807489516 +0000 UTC m=+824.605202299" watchObservedRunningTime="2026-11-03 17:31:23.812720331 +0000 UTC m=+824.610433145"
Nov 03 17:31:24.108616 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:24.108582    6059 patch_prober.go:29] interesting pod/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 17:31:24.108793 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:24.108637    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg" podUID="eb7314fe-6d72-4e51-8cdc-acfcce172a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 17:31:24.645395 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:24.644473    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:31:25.109510 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.109473    6059 patch_prober.go:29] interesting pod/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 17:31:25.109703 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.109530    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg" podUID="eb7314fe-6d72-4e51-8cdc-acfcce172a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 17:31:25.608537 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.608502    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"]
Nov 03 17:31:25.615490 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.615452    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.662601 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.662564    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"]
Nov 03 17:31:25.666627 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.666597    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:25.666784 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.666624    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-credential-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.666784 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.666654    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-istio-data\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.666784 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.666686    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c1509f-87be-4742-86ac-65c438f9523b-istio-token\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.666956 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.666799    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c1509f-87be-4742-86ac-65c438f9523b-istio-podinfo\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.666956 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.666858    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-workload-certs\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.666956 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.666922    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-istio-envoy\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.667079 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.666970    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c1509f-87be-4742-86ac-65c438f9523b-istiod-ca-cert\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.667079 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.667034    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-workload-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.667150 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.667079    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wbmt\" (UniqueName: \"kubernetes.io/projected/89c1509f-87be-4742-86ac-65c438f9523b-kube-api-access-8wbmt\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.755877 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.755842    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2355f4\"/\"kube-root-ca.crt\""
Nov 03 17:31:25.756123 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.756094    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2355f4\"/\"e2e-conflict-b-0ff8cfd1-kserve-self-signed-certs\""
Nov 03 17:31:25.756408 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.755960    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2355f4\"/\"openshift-service-ca.crt\""
Nov 03 17:31:25.756577 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.755862    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2355f4\"/\"default-dockercfg-r6smg\""
Nov 03 17:31:25.770575 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.770530    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"]
Nov 03 17:31:25.771303 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.771275    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c1509f-87be-4742-86ac-65c438f9523b-istio-podinfo\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.771461 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.771345    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-workload-certs\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.771461 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.771397    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-home\") pod \"e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:25.771461 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.771440    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-istio-envoy\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.771660 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.771481    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c1509f-87be-4742-86ac-65c438f9523b-istiod-ca-cert\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.771660 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.771518    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-workload-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.771660 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.771549    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wbmt\" (UniqueName: \"kubernetes.io/projected/89c1509f-87be-4742-86ac-65c438f9523b-kube-api-access-8wbmt\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.771660 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.771586    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-tmp-dir\") pod \"e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:25.771920 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.771750    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq9tg\" (UniqueName: \"kubernetes.io/projected/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-kube-api-access-xq9tg\") pod \"e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:25.771920 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.771800    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-credential-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.771920 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.771823    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-istio-data\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.771920 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.771854    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-model-cache\") pod \"e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:25.771920 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.771885    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-dshm\") pod \"e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:25.772181 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.771937    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c1509f-87be-4742-86ac-65c438f9523b-istio-token\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.772181 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.771970    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-tls-certs\") pod \"e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:25.772988 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.772959    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-workload-certs\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.773160 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.773138    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-credential-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.773592 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.773563    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-workload-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.774318 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.774294    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-istio-data\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.778682 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.778653    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c1509f-87be-4742-86ac-65c438f9523b-istiod-ca-cert\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.778781 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.778749    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-istio-envoy\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.779076 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.779058    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c1509f-87be-4742-86ac-65c438f9523b-istio-podinfo\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.805699 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.801773    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"]
Nov 03 17:31:25.872168 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.872072    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:25.872811 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.872780    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9tg\" (UniqueName: \"kubernetes.io/projected/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-kube-api-access-xq9tg\") pod \"e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:25.872983 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.872842    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-model-cache\") pod \"e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:25.872983 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.872873    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-dshm\") pod \"e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:25.872983 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.872915    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-tls-certs\") pod \"e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:25.872983 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.872979    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-home\") pod \"e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:25.873197 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.873050    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-tmp-dir\") pod \"e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:25.874469 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.874427    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-model-cache\") pod \"e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:25.874738 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.874708    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c1509f-87be-4742-86ac-65c438f9523b-istio-token\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.874836 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.874757    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-home\") pod \"e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:25.874952 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.874935    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-tmp-dir\") pod \"e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:25.876491 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.876445    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-tls-certs\") pod \"e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:25.876897 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.876849    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-dshm\") pod \"e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:25.877697 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.877672    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wbmt\" (UniqueName: \"kubernetes.io/projected/89c1509f-87be-4742-86ac-65c438f9523b-kube-api-access-8wbmt\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.954381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.953721    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:25.956795 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.956765    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq9tg\" (UniqueName: \"kubernetes.io/projected/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-kube-api-access-xq9tg\") pod \"e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:25.980276 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:25.980243    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:26.109408 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:26.109347    6059 patch_prober.go:29] interesting pod/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 17:31:26.109692 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:26.109653    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg" podUID="eb7314fe-6d72-4e51-8cdc-acfcce172a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 17:31:26.189865 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:26.189834    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:26.428904 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:31:26.428862    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c1509f_87be_4742_86ac_65c438f9523b.slice/crio-6f7470474be9dbdf584fc328a7e58381b13b6644dfa23410c3b6e1e0779035d9 WatchSource:0}: Error finding container 6f7470474be9dbdf584fc328a7e58381b13b6644dfa23410c3b6e1e0779035d9: Status 404 returned error can't find the container with id 6f7470474be9dbdf584fc328a7e58381b13b6644dfa23410c3b6e1e0779035d9
Nov 03 17:31:26.454531 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:26.452682    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"]
Nov 03 17:31:26.455431 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:26.455377    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:31:26.455593 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:26.455478    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:31:26.455593 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:26.455526    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:31:26.485825 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:31:26.485794    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod814e8ea8_2bea_40d3_a2a0_bfbf27d33503.slice/crio-9fff589c07d9d35477c35e7182f5a129683a8d163cdc8817a92fd30708c5f27c WatchSource:0}: Error finding container 9fff589c07d9d35477c35e7182f5a129683a8d163cdc8817a92fd30708c5f27c: Status 404 returned error can't find the container with id 9fff589c07d9d35477c35e7182f5a129683a8d163cdc8817a92fd30708c5f27c
Nov 03 17:31:26.487940 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:26.487918    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"]
Nov 03 17:31:26.586818 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:26.584981    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5" event={"ID":"814e8ea8-2bea-40d3-a2a0-bfbf27d33503","Type":"ContainerStarted","Data":"549e11a398cdc76df78a7948dc38e7a9606ffa58c72413a7e4db8d44aa27e4ad"}
Nov 03 17:31:26.586818 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:26.585019    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5" event={"ID":"814e8ea8-2bea-40d3-a2a0-bfbf27d33503","Type":"ContainerStarted","Data":"9fff589c07d9d35477c35e7182f5a129683a8d163cdc8817a92fd30708c5f27c"}
Nov 03 17:31:26.587132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:26.587110    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv" event={"ID":"89c1509f-87be-4742-86ac-65c438f9523b","Type":"ContainerStarted","Data":"58a5837abf1a2a9450b92e5456fb6055f3c46deb8b7039fa455b963943d056ec"}
Nov 03 17:31:26.587189 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:26.587159    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv" event={"ID":"89c1509f-87be-4742-86ac-65c438f9523b","Type":"ContainerStarted","Data":"6f7470474be9dbdf584fc328a7e58381b13b6644dfa23410c3b6e1e0779035d9"}
Nov 03 17:31:26.706802 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:26.706749    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5" podStartSLOduration=1.7067293540000001 podStartE2EDuration="1.706729354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:31:26.705832769 +0000 UTC m=+827.503545553" watchObservedRunningTime="2026-11-03 17:31:26.706729354 +0000 UTC m=+827.504442135"
Nov 03 17:31:26.838380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:26.833931    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv" podStartSLOduration=1.833909635 podStartE2EDuration="1.833909635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:31:26.830909409 +0000 UTC m=+827.628622194" watchObservedRunningTime="2026-11-03 17:31:26.833909635 +0000 UTC m=+827.631622419"
Nov 03 17:31:26.954754 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:26.954714    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:26.983845 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:26.982488    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:27.114109 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:27.114079    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:31:27.115187 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:27.115166    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:31:27.957463 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:27.957431    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 17:31:27.957830 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:27.957485    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv" podUID="89c1509f-87be-4742-86ac-65c438f9523b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 17:31:28.954985 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:28.954945    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 17:31:28.955216 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:28.955019    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv" podUID="89c1509f-87be-4742-86ac-65c438f9523b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 17:31:29.701406 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:29.701347    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:29.955054 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:29.954957    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 17:31:29.955054 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:29.955014    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv" podUID="89c1509f-87be-4742-86ac-65c438f9523b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 17:31:30.955848 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:30.955813    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 17:31:30.956398 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:30.956348    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv" podUID="89c1509f-87be-4742-86ac-65c438f9523b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 17:31:31.155826 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:31.155777    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:31.955783 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:31.955747    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 17:31:31.955959 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:31.955808    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv" podUID="89c1509f-87be-4742-86ac-65c438f9523b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 17:31:32.287206 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.287175    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6"]
Nov 03 17:31:32.291471 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.291447    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6"
Nov 03 17:31:32.313220 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.313187    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:32.351347 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.351310    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:31:32.380769 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.380726    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad6af55-71a2-4be1-aa27-07a1b0a4daff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7538718-nmxc6\" (UID: \"fad6af55-71a2-4be1-aa27-07a1b0a4daff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6"
Nov 03 17:31:32.380970 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.380776    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad6af55-71a2-4be1-aa27-07a1b0a4daff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7538718-nmxc6\" (UID: \"fad6af55-71a2-4be1-aa27-07a1b0a4daff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6"
Nov 03 17:31:32.380970 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.380816    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4txds\" (UniqueName: \"kubernetes.io/projected/fad6af55-71a2-4be1-aa27-07a1b0a4daff-kube-api-access-4txds\") pod \"maas-api-revoke-sub-c7538718-nmxc6\" (UID: \"fad6af55-71a2-4be1-aa27-07a1b0a4daff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6"
Nov 03 17:31:32.403390 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.402622    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:31:32.403390 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.402640    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:31:32.403390 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.402765    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:31:32.405833 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.405805    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6"]
Nov 03 17:31:32.482253 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.482213    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad6af55-71a2-4be1-aa27-07a1b0a4daff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7538718-nmxc6\" (UID: \"fad6af55-71a2-4be1-aa27-07a1b0a4daff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6"
Nov 03 17:31:32.482253 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.482254    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad6af55-71a2-4be1-aa27-07a1b0a4daff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7538718-nmxc6\" (UID: \"fad6af55-71a2-4be1-aa27-07a1b0a4daff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6"
Nov 03 17:31:32.482564 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.482287    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4txds\" (UniqueName: \"kubernetes.io/projected/fad6af55-71a2-4be1-aa27-07a1b0a4daff-kube-api-access-4txds\") pod \"maas-api-revoke-sub-c7538718-nmxc6\" (UID: \"fad6af55-71a2-4be1-aa27-07a1b0a4daff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6"
Nov 03 17:31:32.482960 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.482938    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad6af55-71a2-4be1-aa27-07a1b0a4daff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7538718-nmxc6\" (UID: \"fad6af55-71a2-4be1-aa27-07a1b0a4daff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6"
Nov 03 17:31:32.484805 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.484776    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad6af55-71a2-4be1-aa27-07a1b0a4daff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7538718-nmxc6\" (UID: \"fad6af55-71a2-4be1-aa27-07a1b0a4daff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6"
Nov 03 17:31:32.545887 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.545801    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4txds\" (UniqueName: \"kubernetes.io/projected/fad6af55-71a2-4be1-aa27-07a1b0a4daff-kube-api-access-4txds\") pod \"maas-api-revoke-sub-c7538718-nmxc6\" (UID: \"fad6af55-71a2-4be1-aa27-07a1b0a4daff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6"
Nov 03 17:31:32.605518 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.605485    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6"
Nov 03 17:31:32.679187 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.679118    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:32.860183 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.860061    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:32.955562 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.955515    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 17:31:32.955743 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.955585    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv" podUID="89c1509f-87be-4742-86ac-65c438f9523b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 17:31:32.984143 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:31:32.984104    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad6af55_71a2_4be1_aa27_07a1b0a4daff.slice/crio-6c95492485ea0b2fd9a20788340f7669884a8ebdb457d318c635d2dfe6a24bfc WatchSource:0}: Error finding container 6c95492485ea0b2fd9a20788340f7669884a8ebdb457d318c635d2dfe6a24bfc: Status 404 returned error can't find the container with id 6c95492485ea0b2fd9a20788340f7669884a8ebdb457d318c635d2dfe6a24bfc
Nov 03 17:31:32.986930 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:32.986900    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6"]
Nov 03 17:31:33.643504 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:33.643274    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6" event={"ID":"fad6af55-71a2-4be1-aa27-07a1b0a4daff","Type":"ContainerStarted","Data":"4e7bd80fa30c4315df002eae0abca15340f2d6c629aef68bd6de2c2484f3edb7"}
Nov 03 17:31:33.643833 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:33.643807    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6" event={"ID":"fad6af55-71a2-4be1-aa27-07a1b0a4daff","Type":"ContainerStarted","Data":"6c95492485ea0b2fd9a20788340f7669884a8ebdb457d318c635d2dfe6a24bfc"}
Nov 03 17:31:33.766736 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:33.766676    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6" podStartSLOduration=1.766656451 podStartE2EDuration="1.766656451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:31:33.764039401 +0000 UTC m=+834.561752194" watchObservedRunningTime="2026-11-03 17:31:33.766656451 +0000 UTC m=+834.564369232"
Nov 03 17:31:34.027421 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:34.024243    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:34.027421 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:34.024845    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:34.027421 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:34.026209    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:31:34.570027 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:34.569997    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh"
Nov 03 17:31:34.570462 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:34.570438    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x"
Nov 03 17:31:34.650902 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:34.650865    6059 generic.go:356] "Generic (PLEG): container finished" podID="fad6af55-71a2-4be1-aa27-07a1b0a4daff" containerID="4e7bd80fa30c4315df002eae0abca15340f2d6c629aef68bd6de2c2484f3edb7" exitCode=0
Nov 03 17:31:34.651104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:34.650970    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6" event={"ID":"fad6af55-71a2-4be1-aa27-07a1b0a4daff","Type":"ContainerDied","Data":"4e7bd80fa30c4315df002eae0abca15340f2d6c629aef68bd6de2c2484f3edb7"}
Nov 03 17:31:34.799304 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:34.799197    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:35.828242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:35.828211    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6"
Nov 03 17:31:35.947618 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:35.947292    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fad6af55-71a2-4be1-aa27-07a1b0a4daff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad6af55-71a2-4be1-aa27-07a1b0a4daff-maas-api-openshift-ca\") pod \"fad6af55-71a2-4be1-aa27-07a1b0a4daff\" (UID: \"fad6af55-71a2-4be1-aa27-07a1b0a4daff\") "
Nov 03 17:31:35.947618 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:35.947345    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad6af55-71a2-4be1-aa27-07a1b0a4daff-kube-api-access-4txds\" (UniqueName: \"kubernetes.io/projected/fad6af55-71a2-4be1-aa27-07a1b0a4daff-kube-api-access-4txds\") pod \"fad6af55-71a2-4be1-aa27-07a1b0a4daff\" (UID: \"fad6af55-71a2-4be1-aa27-07a1b0a4daff\") "
Nov 03 17:31:35.947618 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:35.947529    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fad6af55-71a2-4be1-aa27-07a1b0a4daff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad6af55-71a2-4be1-aa27-07a1b0a4daff-maas-api-xks-ca\") pod \"fad6af55-71a2-4be1-aa27-07a1b0a4daff\" (UID: \"fad6af55-71a2-4be1-aa27-07a1b0a4daff\") "
Nov 03 17:31:35.949050 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:35.949018    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad6af55-71a2-4be1-aa27-07a1b0a4daff-maas-api-openshift-ca" pod "fad6af55-71a2-4be1-aa27-07a1b0a4daff" (UID: "fad6af55-71a2-4be1-aa27-07a1b0a4daff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:31:35.950750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:35.950682    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fad6af55-71a2-4be1-aa27-07a1b0a4daff-maas-api-xks-ca" pod "fad6af55-71a2-4be1-aa27-07a1b0a4daff" (UID: "fad6af55-71a2-4be1-aa27-07a1b0a4daff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:31:35.951406 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:35.951380    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad6af55-71a2-4be1-aa27-07a1b0a4daff-kube-api-access-4txds" pod "fad6af55-71a2-4be1-aa27-07a1b0a4daff" (UID: "fad6af55-71a2-4be1-aa27-07a1b0a4daff"). InnerVolumeSpecName "kube-api-access-4txds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:31:35.980780 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:35.980747    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:35.993547 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:35.993515    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:36.048848 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:36.048813    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad6af55-71a2-4be1-aa27-07a1b0a4daff-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:31:36.048848 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:36.048844    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad6af55-71a2-4be1-aa27-07a1b0a4daff-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:31:36.049053 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:36.048862    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4txds\" (UniqueName: \"kubernetes.io/projected/fad6af55-71a2-4be1-aa27-07a1b0a4daff-kube-api-access-4txds\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:31:36.329040 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:36.329011    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:36.666216 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:36.666130    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6"
Nov 03 17:31:36.666216 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:36.666161    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6" event={"ID":"fad6af55-71a2-4be1-aa27-07a1b0a4daff","Type":"ContainerDied","Data":"6c95492485ea0b2fd9a20788340f7669884a8ebdb457d318c635d2dfe6a24bfc"}
Nov 03 17:31:36.666216 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:36.666212    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c95492485ea0b2fd9a20788340f7669884a8ebdb457d318c635d2dfe6a24bfc"
Nov 03 17:31:36.678026 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:36.677997    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:31:36.862307 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:31:36.862270    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:31:38.653310 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:38.653247    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:41.052762 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:41.052724    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:44.660557 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:44.660491    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:45.636676 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:45.636637    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:45.822649 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:45.822604    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:50.234778 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:50.234746    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:54.520856 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:54.520808    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e57883-75k8d"]
Nov 03 17:31:54.521792 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:54.521767    6059 cpu_manager.go:414] "containerMap: removing container" podUID="fad6af55-71a2-4be1-aa27-07a1b0a4daff" containerName="revoke-keys"
Nov 03 17:31:54.521909 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:54.521793    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad6af55-71a2-4be1-aa27-07a1b0a4daff" containerName="revoke-keys"
Nov 03 17:31:54.521988 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:54.521971    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad6af55-71a2-4be1-aa27-07a1b0a4daff" containerName="revoke-keys"
Nov 03 17:31:54.527595 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:54.527556    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e57883-75k8d"
Nov 03 17:31:54.583118 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:54.583045    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:31:54.583634 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:54.583569    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:31:54.586802 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:54.586774    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e57883-75k8d"]
Nov 03 17:31:54.634378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:54.634294    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:31:54.641801 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:54.641743    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:31:54.650393 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:54.645567    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683ae3d8-d754-4544-b8cc-342cff61f5f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e57883-75k8d\" (UID: \"683ae3d8-d754-4544-b8cc-342cff61f5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e57883-75k8d"
Nov 03 17:31:54.650393 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:54.645644    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683ae3d8-d754-4544-b8cc-342cff61f5f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e57883-75k8d\" (UID: \"683ae3d8-d754-4544-b8cc-342cff61f5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e57883-75k8d"
Nov 03 17:31:54.650393 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:54.645790    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db2mc\" (UniqueName: \"kubernetes.io/projected/683ae3d8-d754-4544-b8cc-342cff61f5f0-kube-api-access-db2mc\") pod \"maas-api-revoke-sub-85e57883-75k8d\" (UID: \"683ae3d8-d754-4544-b8cc-342cff61f5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e57883-75k8d"
Nov 03 17:31:54.678377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:54.678238    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:54.749381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:54.749145    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db2mc\" (UniqueName: \"kubernetes.io/projected/683ae3d8-d754-4544-b8cc-342cff61f5f0-kube-api-access-db2mc\") pod \"maas-api-revoke-sub-85e57883-75k8d\" (UID: \"683ae3d8-d754-4544-b8cc-342cff61f5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e57883-75k8d"
Nov 03 17:31:54.749381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:54.749382    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683ae3d8-d754-4544-b8cc-342cff61f5f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e57883-75k8d\" (UID: \"683ae3d8-d754-4544-b8cc-342cff61f5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e57883-75k8d"
Nov 03 17:31:54.749671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:54.749434    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683ae3d8-d754-4544-b8cc-342cff61f5f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e57883-75k8d\" (UID: \"683ae3d8-d754-4544-b8cc-342cff61f5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e57883-75k8d"
Nov 03 17:31:54.751186 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:54.751151    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683ae3d8-d754-4544-b8cc-342cff61f5f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e57883-75k8d\" (UID: \"683ae3d8-d754-4544-b8cc-342cff61f5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e57883-75k8d"
Nov 03 17:31:54.763147 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:54.763119    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683ae3d8-d754-4544-b8cc-342cff61f5f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e57883-75k8d\" (UID: \"683ae3d8-d754-4544-b8cc-342cff61f5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e57883-75k8d"
Nov 03 17:31:54.817737 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:54.817583    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db2mc\" (UniqueName: \"kubernetes.io/projected/683ae3d8-d754-4544-b8cc-342cff61f5f0-kube-api-access-db2mc\") pod \"maas-api-revoke-sub-85e57883-75k8d\" (UID: \"683ae3d8-d754-4544-b8cc-342cff61f5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e57883-75k8d"
Nov 03 17:31:54.854872 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:54.854573    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e57883-75k8d"
Nov 03 17:31:54.880380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:54.877226    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:55.066391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.065238    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:31:55.509492 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.509389    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"]
Nov 03 17:31:55.518328 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.518298    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.619468 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:31:55.618633    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683ae3d8_d754_4544_b8cc_342cff61f5f0.slice/crio-a93c158b5860697b4f3d4bf4c7fa52739539afd7141e5dea1c49818668bbb0e9 WatchSource:0}: Error finding container a93c158b5860697b4f3d4bf4c7fa52739539afd7141e5dea1c49818668bbb0e9: Status 404 returned error can't find the container with id a93c158b5860697b4f3d4bf4c7fa52739539afd7141e5dea1c49818668bbb0e9
Nov 03 17:31:55.638449 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.638413    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"]
Nov 03 17:31:55.640436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.640151    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e57883-75k8d"]
Nov 03 17:31:55.721002 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.720967    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcxnc\" (UniqueName: \"kubernetes.io/projected/1aec66b4-f062-41dc-b9e7-f69143f51117-kube-api-access-mcxnc\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.721192 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.721033    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-credential-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.721192 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.721099    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aec66b4-f062-41dc-b9e7-f69143f51117-istiod-ca-cert\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.721192 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.721155    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-workload-certs\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.721294 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.721201    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-data\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.721294 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.721229    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-podinfo\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.721294 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.721262    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-token\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.721413 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.721313    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-workload-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.721448 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.721416    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-envoy\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.806368 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.806258    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e57883-75k8d" event={"ID":"683ae3d8-d754-4544-b8cc-342cff61f5f0","Type":"ContainerStarted","Data":"a93c158b5860697b4f3d4bf4c7fa52739539afd7141e5dea1c49818668bbb0e9"}
Nov 03 17:31:55.822044 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.822007    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aec66b4-f062-41dc-b9e7-f69143f51117-istiod-ca-cert\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.822044 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.822047    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-workload-certs\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.822291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.822081    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-data\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.822291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.822103    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-podinfo\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.822291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.822127    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-token\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.822291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.822145    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-workload-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.822291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.822163    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-envoy\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.822602 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.822442    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcxnc\" (UniqueName: \"kubernetes.io/projected/1aec66b4-f062-41dc-b9e7-f69143f51117-kube-api-access-mcxnc\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.822602 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.822521    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-credential-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.822602 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.822568    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-workload-certs\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.822602 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.822583    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-data\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.822882 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.822837    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-workload-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.823016 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.822880    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aec66b4-f062-41dc-b9e7-f69143f51117-istiod-ca-cert\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.823113 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.823092    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-credential-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.824955 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.824925    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-podinfo\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.825070 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.825047    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-envoy\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.885455 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.885421    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-token\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:55.885626 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:55.885595    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcxnc\" (UniqueName: \"kubernetes.io/projected/1aec66b4-f062-41dc-b9e7-f69143f51117-kube-api-access-mcxnc\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:56.141777 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:56.141686    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:56.572933 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:56.572899    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"]
Nov 03 17:31:56.588596 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:31:56.588561    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aec66b4_f062_41dc_b9e7_f69143f51117.slice/crio-6457431bad9dd8a77ccad54343c8adf85b297e94d0cfec81d4791bebe8f2b6d9 WatchSource:0}: Error finding container 6457431bad9dd8a77ccad54343c8adf85b297e94d0cfec81d4791bebe8f2b6d9: Status 404 returned error can't find the container with id 6457431bad9dd8a77ccad54343c8adf85b297e94d0cfec81d4791bebe8f2b6d9
Nov 03 17:31:56.591303 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:56.591269    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:31:56.591414 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:56.591339    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:31:56.591414 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:56.591390    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:31:56.820419 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:56.820124    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj" event={"ID":"1aec66b4-f062-41dc-b9e7-f69143f51117","Type":"ContainerStarted","Data":"f18165f3089827e03034dd2ff4dec419d7659bfc9fcfa49ad0ce481b3026e895"}
Nov 03 17:31:56.820419 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:56.820193    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj" event={"ID":"1aec66b4-f062-41dc-b9e7-f69143f51117","Type":"ContainerStarted","Data":"6457431bad9dd8a77ccad54343c8adf85b297e94d0cfec81d4791bebe8f2b6d9"}
Nov 03 17:31:56.822088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:56.822058    6059 generic.go:356] "Generic (PLEG): container finished" podID="683ae3d8-d754-4544-b8cc-342cff61f5f0" containerID="67505fdafc3a2a7399c2ea51ab70d33b0d5baa1b633451de3020a99b408df9e0" exitCode=0
Nov 03 17:31:56.822235 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:56.822217    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e57883-75k8d" event={"ID":"683ae3d8-d754-4544-b8cc-342cff61f5f0","Type":"ContainerDied","Data":"67505fdafc3a2a7399c2ea51ab70d33b0d5baa1b633451de3020a99b408df9e0"}
Nov 03 17:31:56.946424 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:56.946335    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj" podStartSLOduration=1.946319904 podStartE2EDuration="1.946319904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:31:56.944886008 +0000 UTC m=+857.742598789" watchObservedRunningTime="2026-11-03 17:31:56.946319904 +0000 UTC m=+857.744032684"
Nov 03 17:31:57.142543 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:57.142449    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:31:57.144677 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:57.144649    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 17:31:57.144821 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:57.144699    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj" podUID="1aec66b4-f062-41dc-b9e7-f69143f51117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 17:31:58.079742 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:58.079710    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e57883-75k8d"
Nov 03 17:31:58.152446 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:58.145875    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 17:31:58.152446 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:58.145942    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj" podUID="1aec66b4-f062-41dc-b9e7-f69143f51117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 17:31:58.164076 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:58.164035    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/683ae3d8-d754-4544-b8cc-342cff61f5f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683ae3d8-d754-4544-b8cc-342cff61f5f0-maas-api-xks-ca\") pod \"683ae3d8-d754-4544-b8cc-342cff61f5f0\" (UID: \"683ae3d8-d754-4544-b8cc-342cff61f5f0\") "
Nov 03 17:31:58.164242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:58.164179    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/683ae3d8-d754-4544-b8cc-342cff61f5f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683ae3d8-d754-4544-b8cc-342cff61f5f0-maas-api-openshift-ca\") pod \"683ae3d8-d754-4544-b8cc-342cff61f5f0\" (UID: \"683ae3d8-d754-4544-b8cc-342cff61f5f0\") "
Nov 03 17:31:58.164284 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:58.164253    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/683ae3d8-d754-4544-b8cc-342cff61f5f0-kube-api-access-db2mc\" (UniqueName: \"kubernetes.io/projected/683ae3d8-d754-4544-b8cc-342cff61f5f0-kube-api-access-db2mc\") pod \"683ae3d8-d754-4544-b8cc-342cff61f5f0\" (UID: \"683ae3d8-d754-4544-b8cc-342cff61f5f0\") "
Nov 03 17:31:58.164520 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:58.164490    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/683ae3d8-d754-4544-b8cc-342cff61f5f0-maas-api-openshift-ca" pod "683ae3d8-d754-4544-b8cc-342cff61f5f0" (UID: "683ae3d8-d754-4544-b8cc-342cff61f5f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:31:58.164621 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:58.164608    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683ae3d8-d754-4544-b8cc-342cff61f5f0-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:31:58.166262 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:58.166233    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/683ae3d8-d754-4544-b8cc-342cff61f5f0-maas-api-xks-ca" pod "683ae3d8-d754-4544-b8cc-342cff61f5f0" (UID: "683ae3d8-d754-4544-b8cc-342cff61f5f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:31:58.166391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:58.166281    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683ae3d8-d754-4544-b8cc-342cff61f5f0-kube-api-access-db2mc" pod "683ae3d8-d754-4544-b8cc-342cff61f5f0" (UID: "683ae3d8-d754-4544-b8cc-342cff61f5f0"). InnerVolumeSpecName "kube-api-access-db2mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:31:58.265772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:58.265739    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db2mc\" (UniqueName: \"kubernetes.io/projected/683ae3d8-d754-4544-b8cc-342cff61f5f0-kube-api-access-db2mc\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:31:58.265772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:58.265771    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683ae3d8-d754-4544-b8cc-342cff61f5f0-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:31:58.861609 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:58.861578    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e57883-75k8d"
Nov 03 17:31:58.861803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:58.861606    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e57883-75k8d" event={"ID":"683ae3d8-d754-4544-b8cc-342cff61f5f0","Type":"ContainerDied","Data":"a93c158b5860697b4f3d4bf4c7fa52739539afd7141e5dea1c49818668bbb0e9"}
Nov 03 17:31:58.861803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:58.861664    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a93c158b5860697b4f3d4bf4c7fa52739539afd7141e5dea1c49818668bbb0e9"
Nov 03 17:31:59.149439 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:59.148849    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 17:31:59.149439 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:59.148928    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj" podUID="1aec66b4-f062-41dc-b9e7-f69143f51117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 17:31:59.604467 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:31:59.604426    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:00.150377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:00.146573    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 17:32:00.150377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:00.146641    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj" podUID="1aec66b4-f062-41dc-b9e7-f69143f51117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 17:32:01.148840 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:01.148796    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 17:32:01.149133 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:01.149099    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj" podUID="1aec66b4-f062-41dc-b9e7-f69143f51117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 17:32:02.150380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:02.149762    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 17:32:02.150380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:02.149826    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj" podUID="1aec66b4-f062-41dc-b9e7-f69143f51117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 17:32:03.142801 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:03.142759    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 17:32:03.142983 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:03.142830    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj" podUID="1aec66b4-f062-41dc-b9e7-f69143f51117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 17:32:04.139348 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:04.139305    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:04.487904 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:04.487871    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:32:04.488516 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:04.488484    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:32:04.489699 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:04.489678    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:32:05.313136 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:05.313099    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6e0d5b2-wzz6r"]
Nov 03 17:32:05.313727 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:05.313711    6059 cpu_manager.go:414] "containerMap: removing container" podUID="683ae3d8-d754-4544-b8cc-342cff61f5f0" containerName="revoke-keys"
Nov 03 17:32:05.313770 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:05.313728    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="683ae3d8-d754-4544-b8cc-342cff61f5f0" containerName="revoke-keys"
Nov 03 17:32:05.313821 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:05.313812    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="683ae3d8-d754-4544-b8cc-342cff61f5f0" containerName="revoke-keys"
Nov 03 17:32:05.322948 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:05.322915    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e0d5b2-wzz6r"
Nov 03 17:32:05.350721 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:05.350685    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d41f28-9e55-4c45-9fd7-275c2c00555a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6e0d5b2-wzz6r\" (UID: \"27d41f28-9e55-4c45-9fd7-275c2c00555a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e0d5b2-wzz6r"
Nov 03 17:32:05.350889 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:05.350842    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb84c\" (UniqueName: \"kubernetes.io/projected/27d41f28-9e55-4c45-9fd7-275c2c00555a-kube-api-access-mb84c\") pod \"maas-api-revoke-sub-a6e0d5b2-wzz6r\" (UID: \"27d41f28-9e55-4c45-9fd7-275c2c00555a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e0d5b2-wzz6r"
Nov 03 17:32:05.350931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:05.350907    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d41f28-9e55-4c45-9fd7-275c2c00555a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6e0d5b2-wzz6r\" (UID: \"27d41f28-9e55-4c45-9fd7-275c2c00555a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e0d5b2-wzz6r"
Nov 03 17:32:05.376754 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:05.376717    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6e0d5b2-wzz6r"]
Nov 03 17:32:05.378269 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:05.378247    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:32:05.378466 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:05.378446    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:32:05.378555 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:05.378511    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:32:05.378646 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:05.378625    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:32:05.452144 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:05.452109    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d41f28-9e55-4c45-9fd7-275c2c00555a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6e0d5b2-wzz6r\" (UID: \"27d41f28-9e55-4c45-9fd7-275c2c00555a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e0d5b2-wzz6r"
Nov 03 17:32:05.452337 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:05.452189    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb84c\" (UniqueName: \"kubernetes.io/projected/27d41f28-9e55-4c45-9fd7-275c2c00555a-kube-api-access-mb84c\") pod \"maas-api-revoke-sub-a6e0d5b2-wzz6r\" (UID: \"27d41f28-9e55-4c45-9fd7-275c2c00555a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e0d5b2-wzz6r"
Nov 03 17:32:05.452337 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:05.452244    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d41f28-9e55-4c45-9fd7-275c2c00555a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6e0d5b2-wzz6r\" (UID: \"27d41f28-9e55-4c45-9fd7-275c2c00555a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e0d5b2-wzz6r"
Nov 03 17:32:05.452826 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:05.452805    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d41f28-9e55-4c45-9fd7-275c2c00555a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6e0d5b2-wzz6r\" (UID: \"27d41f28-9e55-4c45-9fd7-275c2c00555a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e0d5b2-wzz6r"
Nov 03 17:32:05.454633 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:05.454616    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d41f28-9e55-4c45-9fd7-275c2c00555a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6e0d5b2-wzz6r\" (UID: \"27d41f28-9e55-4c45-9fd7-275c2c00555a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e0d5b2-wzz6r"
Nov 03 17:32:05.515725 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:05.515693    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb84c\" (UniqueName: \"kubernetes.io/projected/27d41f28-9e55-4c45-9fd7-275c2c00555a-kube-api-access-mb84c\") pod \"maas-api-revoke-sub-a6e0d5b2-wzz6r\" (UID: \"27d41f28-9e55-4c45-9fd7-275c2c00555a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e0d5b2-wzz6r"
Nov 03 17:32:05.638142 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:05.638043    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e0d5b2-wzz6r"
Nov 03 17:32:06.018329 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:32:06.018284    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d41f28_9e55_4c45_9fd7_275c2c00555a.slice/crio-355c523433c74afe0aceefd3209af661cdb69c9fb14c2d8d308b7cfd05889f34 WatchSource:0}: Error finding container 355c523433c74afe0aceefd3209af661cdb69c9fb14c2d8d308b7cfd05889f34: Status 404 returned error can't find the container with id 355c523433c74afe0aceefd3209af661cdb69c9fb14c2d8d308b7cfd05889f34
Nov 03 17:32:06.021636 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:06.021614    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6e0d5b2-wzz6r"]
Nov 03 17:32:06.862604 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:32:06.862567    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:32:06.926648 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:06.926615    6059 generic.go:356] "Generic (PLEG): container finished" podID="27d41f28-9e55-4c45-9fd7-275c2c00555a" containerID="4b3b1d45b6dd5cb2303c36bece8e390fcc3be874aaefd815b0f0f8093c2f0d89" exitCode=0
Nov 03 17:32:06.926823 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:06.926693    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e0d5b2-wzz6r" event={"ID":"27d41f28-9e55-4c45-9fd7-275c2c00555a","Type":"ContainerDied","Data":"4b3b1d45b6dd5cb2303c36bece8e390fcc3be874aaefd815b0f0f8093c2f0d89"}
Nov 03 17:32:06.926823 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:06.926743    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e0d5b2-wzz6r" event={"ID":"27d41f28-9e55-4c45-9fd7-275c2c00555a","Type":"ContainerStarted","Data":"355c523433c74afe0aceefd3209af661cdb69c9fb14c2d8d308b7cfd05889f34"}
Nov 03 17:32:08.097118 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:08.097083    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e0d5b2-wzz6r"
Nov 03 17:32:08.179323 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:08.179286    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:08.187403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:08.187313    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d41f28-9e55-4c45-9fd7-275c2c00555a-kube-api-access-mb84c\" (UniqueName: \"kubernetes.io/projected/27d41f28-9e55-4c45-9fd7-275c2c00555a-kube-api-access-mb84c\") pod \"27d41f28-9e55-4c45-9fd7-275c2c00555a\" (UID: \"27d41f28-9e55-4c45-9fd7-275c2c00555a\") "
Nov 03 17:32:08.187403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:08.187387    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27d41f28-9e55-4c45-9fd7-275c2c00555a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d41f28-9e55-4c45-9fd7-275c2c00555a-maas-api-xks-ca\") pod \"27d41f28-9e55-4c45-9fd7-275c2c00555a\" (UID: \"27d41f28-9e55-4c45-9fd7-275c2c00555a\") "
Nov 03 17:32:08.187629 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:08.187433    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27d41f28-9e55-4c45-9fd7-275c2c00555a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d41f28-9e55-4c45-9fd7-275c2c00555a-maas-api-openshift-ca\") pod \"27d41f28-9e55-4c45-9fd7-275c2c00555a\" (UID: \"27d41f28-9e55-4c45-9fd7-275c2c00555a\") "
Nov 03 17:32:08.187849 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:08.187820    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d41f28-9e55-4c45-9fd7-275c2c00555a-maas-api-openshift-ca" pod "27d41f28-9e55-4c45-9fd7-275c2c00555a" (UID: "27d41f28-9e55-4c45-9fd7-275c2c00555a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:32:08.187962 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:08.187948    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d41f28-9e55-4c45-9fd7-275c2c00555a-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:08.189667 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:08.189635    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d41f28-9e55-4c45-9fd7-275c2c00555a-kube-api-access-mb84c" pod "27d41f28-9e55-4c45-9fd7-275c2c00555a" (UID: "27d41f28-9e55-4c45-9fd7-275c2c00555a"). InnerVolumeSpecName "kube-api-access-mb84c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:32:08.189974 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:08.189944    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d41f28-9e55-4c45-9fd7-275c2c00555a-maas-api-xks-ca" pod "27d41f28-9e55-4c45-9fd7-275c2c00555a" (UID: "27d41f28-9e55-4c45-9fd7-275c2c00555a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:32:08.288850 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:08.288802    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb84c\" (UniqueName: \"kubernetes.io/projected/27d41f28-9e55-4c45-9fd7-275c2c00555a-kube-api-access-mb84c\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:08.288850 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:08.288833    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d41f28-9e55-4c45-9fd7-275c2c00555a-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:08.705907 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:08.705870    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"]
Nov 03 17:32:08.780395 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:08.780350    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:08.941789 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:08.941753    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e0d5b2-wzz6r" event={"ID":"27d41f28-9e55-4c45-9fd7-275c2c00555a","Type":"ContainerDied","Data":"355c523433c74afe0aceefd3209af661cdb69c9fb14c2d8d308b7cfd05889f34"}
Nov 03 17:32:08.941789 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:08.941787    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e0d5b2-wzz6r"
Nov 03 17:32:08.941986 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:08.941805    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="355c523433c74afe0aceefd3209af661cdb69c9fb14c2d8d308b7cfd05889f34"
Nov 03 17:32:09.853879 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:09.853846    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:14.590979 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:14.590939    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:15.155853 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:15.155817    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:15.356172 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:15.356085    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:16.754934 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:16.754870    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:17.263857 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:17.263816    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"]
Nov 03 17:32:17.264390 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:17.264316    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs" podUID="9a6b46be-ed11-4708-b1a9-4500870e4d2e" containerName="istio-proxy" containerID="cri-o://58ef2ec45f12d423530a12f40851cdff784f4dbeeadac2c5e3a75c65013b064d" gracePeriod=30
Nov 03 17:32:17.872321 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:17.872239    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:19.818202 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:19.818163    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:22.052984 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.052948    6059 generic.go:356] "Generic (PLEG): container finished" podID="3db8f565-8a57-45ff-b31c-aa5a4a7093fa" containerID="d1f7556315653fbd1a28346e5983a15d8ec09118a3ee7cd9543a599f8e0d004f" exitCode=0
Nov 03 17:32:22.053446 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.053022    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns" event={"ID":"3db8f565-8a57-45ff-b31c-aa5a4a7093fa","Type":"ContainerDied","Data":"d1f7556315653fbd1a28346e5983a15d8ec09118a3ee7cd9543a599f8e0d004f"}
Nov 03 17:32:22.053496 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.053465    6059 scope.go:122] "RemoveContainer" containerID="d1f7556315653fbd1a28346e5983a15d8ec09118a3ee7cd9543a599f8e0d004f"
Nov 03 17:32:22.055051 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.055019    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:32:22.055148 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.055108    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:32:22.055189 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.055151    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:32:22.541148 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.541121    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:32:22.659932 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.659821    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-workload-socket\") pod \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") "
Nov 03 17:32:22.660238 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.660214    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-podinfo\") pod \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") "
Nov 03 17:32:22.660418 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.660401    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-token\" (UniqueName: \"kubernetes.io/projected/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-token\") pod \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") "
Nov 03 17:32:22.660560 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.660546    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-workload-certs\") pod \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") "
Nov 03 17:32:22.660745 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.660728    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-data\") pod \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") "
Nov 03 17:32:22.660880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.660867    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-envoy\") pod \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") "
Nov 03 17:32:22.661006 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.660988    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istiod-ca-cert\") pod \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") "
Nov 03 17:32:22.661131 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.661118    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-credential-socket\") pod \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") "
Nov 03 17:32:22.661286 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.661271    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a6b46be-ed11-4708-b1a9-4500870e4d2e-kube-api-access-f87nf\" (UniqueName: \"kubernetes.io/projected/9a6b46be-ed11-4708-b1a9-4500870e4d2e-kube-api-access-f87nf\") pod \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\" (UID: \"9a6b46be-ed11-4708-b1a9-4500870e4d2e\") "
Nov 03 17:32:22.662546 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.661607    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-workload-certs" pod "9a6b46be-ed11-4708-b1a9-4500870e4d2e" (UID: "9a6b46be-ed11-4708-b1a9-4500870e4d2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:32:22.662546 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.661772    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-workload-socket" pod "9a6b46be-ed11-4708-b1a9-4500870e4d2e" (UID: "9a6b46be-ed11-4708-b1a9-4500870e4d2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:32:22.662546 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.662058    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:22.662546 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.662084    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:22.663554 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.663260    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istiod-ca-cert" pod "9a6b46be-ed11-4708-b1a9-4500870e4d2e" (UID: "9a6b46be-ed11-4708-b1a9-4500870e4d2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:32:22.665691 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.665638    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-credential-socket" pod "9a6b46be-ed11-4708-b1a9-4500870e4d2e" (UID: "9a6b46be-ed11-4708-b1a9-4500870e4d2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:32:22.667792 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.667755    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-data" pod "9a6b46be-ed11-4708-b1a9-4500870e4d2e" (UID: "9a6b46be-ed11-4708-b1a9-4500870e4d2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:32:22.667973 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.667948    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-podinfo" pod "9a6b46be-ed11-4708-b1a9-4500870e4d2e" (UID: "9a6b46be-ed11-4708-b1a9-4500870e4d2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:32:22.688905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.688860    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-envoy" pod "9a6b46be-ed11-4708-b1a9-4500870e4d2e" (UID: "9a6b46be-ed11-4708-b1a9-4500870e4d2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:32:22.689862 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.689828    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-token" pod "9a6b46be-ed11-4708-b1a9-4500870e4d2e" (UID: "9a6b46be-ed11-4708-b1a9-4500870e4d2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:32:22.690466 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.690437    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6b46be-ed11-4708-b1a9-4500870e4d2e-kube-api-access-f87nf" pod "9a6b46be-ed11-4708-b1a9-4500870e4d2e" (UID: "9a6b46be-ed11-4708-b1a9-4500870e4d2e"). InnerVolumeSpecName "kube-api-access-f87nf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:32:22.763667 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.763627    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:22.763667 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.763660    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:22.763667 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.763670    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:22.763667 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.763680    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:22.763954 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.763689    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6b46be-ed11-4708-b1a9-4500870e4d2e-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:22.763954 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.763697    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b46be-ed11-4708-b1a9-4500870e4d2e-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:22.763954 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:22.763726    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f87nf\" (UniqueName: \"kubernetes.io/projected/9a6b46be-ed11-4708-b1a9-4500870e4d2e-kube-api-access-f87nf\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:23.065427 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:23.065388    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns" event={"ID":"3db8f565-8a57-45ff-b31c-aa5a4a7093fa","Type":"ContainerStarted","Data":"96bfc86b7d81801f6bc1052bd0fd4b0e18042151e0b761f9d620f974dfab0438"}
Nov 03 17:32:23.067347 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:23.067319    6059 generic.go:356] "Generic (PLEG): container finished" podID="9a6b46be-ed11-4708-b1a9-4500870e4d2e" containerID="58ef2ec45f12d423530a12f40851cdff784f4dbeeadac2c5e3a75c65013b064d" exitCode=0
Nov 03 17:32:23.067478 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:23.067391    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs" event={"ID":"9a6b46be-ed11-4708-b1a9-4500870e4d2e","Type":"ContainerDied","Data":"58ef2ec45f12d423530a12f40851cdff784f4dbeeadac2c5e3a75c65013b064d"}
Nov 03 17:32:23.067478 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:23.067407    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"
Nov 03 17:32:23.067478 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:23.067444    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs" event={"ID":"9a6b46be-ed11-4708-b1a9-4500870e4d2e","Type":"ContainerDied","Data":"b9e526777553de18d2b083b25f94a2e6f2c6b4f8a7c8b03cf67bb23ff056b027"}
Nov 03 17:32:23.067478 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:23.067468    6059 scope.go:122] "RemoveContainer" containerID="58ef2ec45f12d423530a12f40851cdff784f4dbeeadac2c5e3a75c65013b064d"
Nov 03 17:32:23.078578 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:23.078552    6059 scope.go:122] "RemoveContainer" containerID="b1652b314b78cb0371f38ad4ca032887e99d03ec747732d6435a8959ae300773"
Nov 03 17:32:23.088965 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:23.088941    6059 scope.go:122] "RemoveContainer" containerID="58ef2ec45f12d423530a12f40851cdff784f4dbeeadac2c5e3a75c65013b064d"
Nov 03 17:32:23.089343 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:32:23.089320    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ef2ec45f12d423530a12f40851cdff784f4dbeeadac2c5e3a75c65013b064d\": container with ID starting with 58ef2ec45f12d423530a12f40851cdff784f4dbeeadac2c5e3a75c65013b064d not found: ID does not exist" containerID="58ef2ec45f12d423530a12f40851cdff784f4dbeeadac2c5e3a75c65013b064d"
Nov 03 17:32:23.089453 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:23.089365    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ef2ec45f12d423530a12f40851cdff784f4dbeeadac2c5e3a75c65013b064d"} err="failed to get container status \"58ef2ec45f12d423530a12f40851cdff784f4dbeeadac2c5e3a75c65013b064d\": rpc error: code = NotFound desc = could not find container \"58ef2ec45f12d423530a12f40851cdff784f4dbeeadac2c5e3a75c65013b064d\": container with ID starting with 58ef2ec45f12d423530a12f40851cdff784f4dbeeadac2c5e3a75c65013b064d not found: ID does not exist"
Nov 03 17:32:23.089453 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:23.089390    6059 scope.go:122] "RemoveContainer" containerID="b1652b314b78cb0371f38ad4ca032887e99d03ec747732d6435a8959ae300773"
Nov 03 17:32:23.089681 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:32:23.089658    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1652b314b78cb0371f38ad4ca032887e99d03ec747732d6435a8959ae300773\": container with ID starting with b1652b314b78cb0371f38ad4ca032887e99d03ec747732d6435a8959ae300773 not found: ID does not exist" containerID="b1652b314b78cb0371f38ad4ca032887e99d03ec747732d6435a8959ae300773"
Nov 03 17:32:23.089726 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:23.089688    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1652b314b78cb0371f38ad4ca032887e99d03ec747732d6435a8959ae300773"} err="failed to get container status \"b1652b314b78cb0371f38ad4ca032887e99d03ec747732d6435a8959ae300773\": rpc error: code = NotFound desc = could not find container \"b1652b314b78cb0371f38ad4ca032887e99d03ec747732d6435a8959ae300773\": container with ID starting with b1652b314b78cb0371f38ad4ca032887e99d03ec747732d6435a8959ae300773 not found: ID does not exist"
Nov 03 17:32:23.309731 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:23.309695    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"]
Nov 03 17:32:23.313693 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:23.313654    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-8fc5ec-openshift-default-7ddb776477-s5vhs"]
Nov 03 17:32:23.357760 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:23.357676    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a6b46be-ed11-4708-b1a9-4500870e4d2e" path="/var/lib/kubelet/pods/9a6b46be-ed11-4708-b1a9-4500870e4d2e/volumes"
Nov 03 17:32:23.554399 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:23.551380    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:23.612672 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:23.612588    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:32:23.612856 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:23.612833    6059 patch_prober.go:29] interesting pod/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 17:32:23.612918 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:23.612892    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns" podUID="3db8f565-8a57-45ff-b31c-aa5a4a7093fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 17:32:23.754382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:23.753537    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5" podUID="814e8ea8-2bea-40d3-a2a0-bfbf27d33503" containerName="main" containerID="cri-o://549e11a398cdc76df78a7948dc38e7a9606ffa58c72413a7e4db8d44aa27e4ad" gracePeriod=45
Nov 03 17:32:24.069455 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.069418    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:32:24.076404 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.076339    6059 generic.go:356] "Generic (PLEG): container finished" podID="814e8ea8-2bea-40d3-a2a0-bfbf27d33503" containerID="549e11a398cdc76df78a7948dc38e7a9606ffa58c72413a7e4db8d44aa27e4ad" exitCode=0
Nov 03 17:32:24.076564 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.076425    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"
Nov 03 17:32:24.076564 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.076431    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5" event={"ID":"814e8ea8-2bea-40d3-a2a0-bfbf27d33503","Type":"ContainerDied","Data":"549e11a398cdc76df78a7948dc38e7a9606ffa58c72413a7e4db8d44aa27e4ad"}
Nov 03 17:32:24.076564 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.076482    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5" event={"ID":"814e8ea8-2bea-40d3-a2a0-bfbf27d33503","Type":"ContainerDied","Data":"9fff589c07d9d35477c35e7182f5a129683a8d163cdc8817a92fd30708c5f27c"}
Nov 03 17:32:24.076564 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.076504    6059 scope.go:122] "RemoveContainer" containerID="549e11a398cdc76df78a7948dc38e7a9606ffa58c72413a7e4db8d44aa27e4ad"
Nov 03 17:32:24.086954 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.086932    6059 scope.go:122] "RemoveContainer" containerID="549e11a398cdc76df78a7948dc38e7a9606ffa58c72413a7e4db8d44aa27e4ad"
Nov 03 17:32:24.087268 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:32:24.087245    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"549e11a398cdc76df78a7948dc38e7a9606ffa58c72413a7e4db8d44aa27e4ad\": container with ID starting with 549e11a398cdc76df78a7948dc38e7a9606ffa58c72413a7e4db8d44aa27e4ad not found: ID does not exist" containerID="549e11a398cdc76df78a7948dc38e7a9606ffa58c72413a7e4db8d44aa27e4ad"
Nov 03 17:32:24.087310 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.087279    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"549e11a398cdc76df78a7948dc38e7a9606ffa58c72413a7e4db8d44aa27e4ad"} err="failed to get container status \"549e11a398cdc76df78a7948dc38e7a9606ffa58c72413a7e4db8d44aa27e4ad\": rpc error: code = NotFound desc = could not find container \"549e11a398cdc76df78a7948dc38e7a9606ffa58c72413a7e4db8d44aa27e4ad\": container with ID starting with 549e11a398cdc76df78a7948dc38e7a9606ffa58c72413a7e4db8d44aa27e4ad not found: ID does not exist"
Nov 03 17:32:24.179271 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.179186    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-kube-api-access-xq9tg\" (UniqueName: \"kubernetes.io/projected/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-kube-api-access-xq9tg\") pod \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") "
Nov 03 17:32:24.179271 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.179240    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-tmp-dir\") pod \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") "
Nov 03 17:32:24.179501 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.179317    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-home\" (UniqueName: \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-home\") pod \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") "
Nov 03 17:32:24.179501 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.179343    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-tls-certs\" (UniqueName: \"kubernetes.io/secret/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-tls-certs\") pod \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") "
Nov 03 17:32:24.179501 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.179414    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-dshm\" (UniqueName: \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-dshm\") pod \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") "
Nov 03 17:32:24.179501 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.179486    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-model-cache\") pod \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\" (UID: \"814e8ea8-2bea-40d3-a2a0-bfbf27d33503\") "
Nov 03 17:32:24.179751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.179525    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-tmp-dir" pod "814e8ea8-2bea-40d3-a2a0-bfbf27d33503" (UID: "814e8ea8-2bea-40d3-a2a0-bfbf27d33503"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:32:24.179751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.179648    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-home" pod "814e8ea8-2bea-40d3-a2a0-bfbf27d33503" (UID: "814e8ea8-2bea-40d3-a2a0-bfbf27d33503"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:32:24.179867 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.179790    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-model-cache" pod "814e8ea8-2bea-40d3-a2a0-bfbf27d33503" (UID: "814e8ea8-2bea-40d3-a2a0-bfbf27d33503"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:32:24.180123 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.180099    6059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-tmp-dir\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:24.180233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.180125    6059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-home\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:24.180233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.180140    6059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-model-cache\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:24.181670 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.181647    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-kube-api-access-xq9tg" pod "814e8ea8-2bea-40d3-a2a0-bfbf27d33503" (UID: "814e8ea8-2bea-40d3-a2a0-bfbf27d33503"). InnerVolumeSpecName "kube-api-access-xq9tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:32:24.181929 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.181909    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-dshm" pod "814e8ea8-2bea-40d3-a2a0-bfbf27d33503" (UID: "814e8ea8-2bea-40d3-a2a0-bfbf27d33503"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:32:24.181929 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.181919    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-tls-certs" pod "814e8ea8-2bea-40d3-a2a0-bfbf27d33503" (UID: "814e8ea8-2bea-40d3-a2a0-bfbf27d33503"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:32:24.281582 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.281542    6059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-tls-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:24.281582 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.281576    6059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-dshm\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:24.281582 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.281589    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq9tg\" (UniqueName: \"kubernetes.io/projected/814e8ea8-2bea-40d3-a2a0-bfbf27d33503-kube-api-access-xq9tg\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:24.424301 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.424268    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:24.507581 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.507542    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"]
Nov 03 17:32:24.512542 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.512508    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/e2e-conflict-b-0ff8cfd1-kserve-5695bf5d87-vc9h5"]
Nov 03 17:32:24.612545 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.612506    6059 patch_prober.go:29] interesting pod/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 17:32:24.612717 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:24.612565    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns" podUID="3db8f565-8a57-45ff-b31c-aa5a4a7093fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 17:32:25.361158 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:25.361123    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="814e8ea8-2bea-40d3-a2a0-bfbf27d33503" path="/var/lib/kubelet/pods/814e8ea8-2bea-40d3-a2a0-bfbf27d33503/volumes"
Nov 03 17:32:25.613223 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:25.613138    6059 patch_prober.go:29] interesting pod/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 17:32:25.613223 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:25.613201    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns" podUID="3db8f565-8a57-45ff-b31c-aa5a4a7093fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 17:32:26.613379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:26.612499    6059 patch_prober.go:29] interesting pod/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 17:32:26.613379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:26.612566    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns" podUID="3db8f565-8a57-45ff-b31c-aa5a4a7093fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 17:32:27.613060 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:27.613024    6059 patch_prober.go:29] interesting pod/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 17:32:27.613220 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:27.613087    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns" podUID="3db8f565-8a57-45ff-b31c-aa5a4a7093fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 17:32:28.617455 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:28.617425    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:32:28.617919 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:28.617733    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:32:28.618492 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:28.618473    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:32:28.929322 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:28.929209    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:28.953817 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:28.953784    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6"]
Nov 03 17:32:29.049102 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:29.049065    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr"]
Nov 03 17:32:29.312675 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:29.312638    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:29.520412 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:29.520317    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:29.822463 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:29.822425    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:33.719520 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:33.719481    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:33.954237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:33.954194    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6" podUID="0b201da2-6bd6-46c9-917b-03ab1da73865" containerName="payload-processing" containerID="cri-o://e05e24106f7d56d9bc2c219354cea208f2c5119d09fecf04197d8cacc1db17b6" gracePeriod=25
Nov 03 17:32:34.049891 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:34.049833    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr" podUID="a517f58a-ccdb-4629-bcd4-5aa9f7980cf3" containerName="payload-pre-processing" containerID="cri-o://daf57a41024abbb62a680985d5f05c419bfac1477e147e7cf7631e7f86c9987a" gracePeriod=25
Nov 03 17:32:34.168301 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:34.168269    6059 generic.go:356] "Generic (PLEG): container finished" podID="0b201da2-6bd6-46c9-917b-03ab1da73865" containerID="e05e24106f7d56d9bc2c219354cea208f2c5119d09fecf04197d8cacc1db17b6" exitCode=0
Nov 03 17:32:34.168571 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:34.168307    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6" event={"ID":"0b201da2-6bd6-46c9-917b-03ab1da73865","Type":"ContainerDied","Data":"e05e24106f7d56d9bc2c219354cea208f2c5119d09fecf04197d8cacc1db17b6"}
Nov 03 17:32:34.170728 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:34.170704    6059 generic.go:356] "Generic (PLEG): container finished" podID="a517f58a-ccdb-4629-bcd4-5aa9f7980cf3" containerID="daf57a41024abbb62a680985d5f05c419bfac1477e147e7cf7631e7f86c9987a" exitCode=0
Nov 03 17:32:34.170843 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:34.170764    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr" event={"ID":"a517f58a-ccdb-4629-bcd4-5aa9f7980cf3","Type":"ContainerDied","Data":"daf57a41024abbb62a680985d5f05c419bfac1477e147e7cf7631e7f86c9987a"}
Nov 03 17:32:34.309596 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:34.309567    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6"
Nov 03 17:32:34.338933 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:34.338908    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr"
Nov 03 17:32:34.457433 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:34.457400    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b201da2-6bd6-46c9-917b-03ab1da73865-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b201da2-6bd6-46c9-917b-03ab1da73865-plugins-config-volume\") pod \"0b201da2-6bd6-46c9-917b-03ab1da73865\" (UID: \"0b201da2-6bd6-46c9-917b-03ab1da73865\") "
Nov 03 17:32:34.457619 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:34.457449    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a517f58a-ccdb-4629-bcd4-5aa9f7980cf3-kube-api-access-wsvg2\" (UniqueName: \"kubernetes.io/projected/a517f58a-ccdb-4629-bcd4-5aa9f7980cf3-kube-api-access-wsvg2\") pod \"a517f58a-ccdb-4629-bcd4-5aa9f7980cf3\" (UID: \"a517f58a-ccdb-4629-bcd4-5aa9f7980cf3\") "
Nov 03 17:32:34.457619 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:34.457484    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a517f58a-ccdb-4629-bcd4-5aa9f7980cf3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a517f58a-ccdb-4629-bcd4-5aa9f7980cf3-plugins-config-volume\") pod \"a517f58a-ccdb-4629-bcd4-5aa9f7980cf3\" (UID: \"a517f58a-ccdb-4629-bcd4-5aa9f7980cf3\") "
Nov 03 17:32:34.457731 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:34.457656    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b201da2-6bd6-46c9-917b-03ab1da73865-kube-api-access-xffr2\" (UniqueName: \"kubernetes.io/projected/0b201da2-6bd6-46c9-917b-03ab1da73865-kube-api-access-xffr2\") pod \"0b201da2-6bd6-46c9-917b-03ab1da73865\" (UID: \"0b201da2-6bd6-46c9-917b-03ab1da73865\") "
Nov 03 17:32:34.458056 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:34.458022    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b201da2-6bd6-46c9-917b-03ab1da73865-plugins-config-volume" pod "0b201da2-6bd6-46c9-917b-03ab1da73865" (UID: "0b201da2-6bd6-46c9-917b-03ab1da73865"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:32:34.458056 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:34.458039    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a517f58a-ccdb-4629-bcd4-5aa9f7980cf3-plugins-config-volume" pod "a517f58a-ccdb-4629-bcd4-5aa9f7980cf3" (UID: "a517f58a-ccdb-4629-bcd4-5aa9f7980cf3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:32:34.458341 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:34.458320    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b201da2-6bd6-46c9-917b-03ab1da73865-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:34.458468 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:34.458345    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a517f58a-ccdb-4629-bcd4-5aa9f7980cf3-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:34.459779 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:34.459759    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b201da2-6bd6-46c9-917b-03ab1da73865-kube-api-access-xffr2" pod "0b201da2-6bd6-46c9-917b-03ab1da73865" (UID: "0b201da2-6bd6-46c9-917b-03ab1da73865"). InnerVolumeSpecName "kube-api-access-xffr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:32:34.460061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:34.460048    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a517f58a-ccdb-4629-bcd4-5aa9f7980cf3-kube-api-access-wsvg2" pod "a517f58a-ccdb-4629-bcd4-5aa9f7980cf3" (UID: "a517f58a-ccdb-4629-bcd4-5aa9f7980cf3"). InnerVolumeSpecName "kube-api-access-wsvg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:32:34.560327 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:34.560242    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsvg2\" (UniqueName: \"kubernetes.io/projected/a517f58a-ccdb-4629-bcd4-5aa9f7980cf3-kube-api-access-wsvg2\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:34.560327 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:34.560278    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xffr2\" (UniqueName: \"kubernetes.io/projected/0b201da2-6bd6-46c9-917b-03ab1da73865-kube-api-access-xffr2\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:35.135919 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:35.135888    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:35.179555 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:35.179526    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6"
Nov 03 17:32:35.179740 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:35.179566    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6" event={"ID":"0b201da2-6bd6-46c9-917b-03ab1da73865","Type":"ContainerDied","Data":"b5e58a531cf65855d572d0578b99be3529460e42b278c306d9feccd6d59782ab"}
Nov 03 17:32:35.179740 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:35.179681    6059 scope.go:122] "RemoveContainer" containerID="e05e24106f7d56d9bc2c219354cea208f2c5119d09fecf04197d8cacc1db17b6"
Nov 03 17:32:35.181153 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:35.181132    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr" event={"ID":"a517f58a-ccdb-4629-bcd4-5aa9f7980cf3","Type":"ContainerDied","Data":"03811d4c9bb654595a61607d8650fd77ad2b335f440fcb53338f8264bfd90bd3"}
Nov 03 17:32:35.181232 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:35.181167    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr"
Nov 03 17:32:35.194641 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:35.194618    6059 scope.go:122] "RemoveContainer" containerID="daf57a41024abbb62a680985d5f05c419bfac1477e147e7cf7631e7f86c9987a"
Nov 03 17:32:35.354529 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:35.354445    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6"]
Nov 03 17:32:35.406439 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:35.406403    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3e5c4d-5ccccf7b8f-p8kt6"]
Nov 03 17:32:35.470332 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:35.470295    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr"]
Nov 03 17:32:35.474752 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:35.474725    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3e5c4d-74d9c595c9-2x4vr"]
Nov 03 17:32:36.015633 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:36.015589    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:36.861476 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:32:36.861432    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:32:37.336952 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:37.336915    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b201da2-6bd6-46c9-917b-03ab1da73865" path="/var/lib/kubelet/pods/0b201da2-6bd6-46c9-917b-03ab1da73865/volumes"
Nov 03 17:32:37.337480 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:37.337459    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a517f58a-ccdb-4629-bcd4-5aa9f7980cf3" path="/var/lib/kubelet/pods/a517f58a-ccdb-4629-bcd4-5aa9f7980cf3/volumes"
Nov 03 17:32:38.135437 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.135402    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f"]
Nov 03 17:32:38.135975 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.135961    6059 cpu_manager.go:414] "containerMap: removing container" podUID="a517f58a-ccdb-4629-bcd4-5aa9f7980cf3" containerName="payload-pre-processing"
Nov 03 17:32:38.136026 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.135976    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a517f58a-ccdb-4629-bcd4-5aa9f7980cf3" containerName="payload-pre-processing"
Nov 03 17:32:38.136026 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.135986    6059 cpu_manager.go:414] "containerMap: removing container" podUID="0b201da2-6bd6-46c9-917b-03ab1da73865" containerName="payload-processing"
Nov 03 17:32:38.136026 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.135991    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b201da2-6bd6-46c9-917b-03ab1da73865" containerName="payload-processing"
Nov 03 17:32:38.136026 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.136002    6059 cpu_manager.go:414] "containerMap: removing container" podUID="9a6b46be-ed11-4708-b1a9-4500870e4d2e" containerName="istio-proxy"
Nov 03 17:32:38.136026 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.136007    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a6b46be-ed11-4708-b1a9-4500870e4d2e" containerName="istio-proxy"
Nov 03 17:32:38.136026 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.136017    6059 cpu_manager.go:414] "containerMap: removing container" podUID="27d41f28-9e55-4c45-9fd7-275c2c00555a" containerName="revoke-keys"
Nov 03 17:32:38.136026 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.136022    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27d41f28-9e55-4c45-9fd7-275c2c00555a" containerName="revoke-keys"
Nov 03 17:32:38.136255 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.136031    6059 cpu_manager.go:414] "containerMap: removing container" podUID="814e8ea8-2bea-40d3-a2a0-bfbf27d33503" containerName="main"
Nov 03 17:32:38.136255 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.136036    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="814e8ea8-2bea-40d3-a2a0-bfbf27d33503" containerName="main"
Nov 03 17:32:38.136255 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.136048    6059 cpu_manager.go:414] "containerMap: removing container" podUID="9a6b46be-ed11-4708-b1a9-4500870e4d2e" containerName="istio-proxy"
Nov 03 17:32:38.136255 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.136053    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a6b46be-ed11-4708-b1a9-4500870e4d2e" containerName="istio-proxy"
Nov 03 17:32:38.136255 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.136149    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a6b46be-ed11-4708-b1a9-4500870e4d2e" containerName="istio-proxy"
Nov 03 17:32:38.136255 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.136163    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a517f58a-ccdb-4629-bcd4-5aa9f7980cf3" containerName="payload-pre-processing"
Nov 03 17:32:38.136255 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.136172    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b201da2-6bd6-46c9-917b-03ab1da73865" containerName="payload-processing"
Nov 03 17:32:38.136255 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.136181    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="27d41f28-9e55-4c45-9fd7-275c2c00555a" containerName="revoke-keys"
Nov 03 17:32:38.136255 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.136187    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="814e8ea8-2bea-40d3-a2a0-bfbf27d33503" containerName="main"
Nov 03 17:32:38.140844 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.140824    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f"
Nov 03 17:32:38.198896 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.198864    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f"]
Nov 03 17:32:38.212954 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.212921    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e54841-75f1-4d4c-846f-c99ea9051b4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71750fe5-xnw9f\" (UID: \"76e54841-75f1-4d4c-846f-c99ea9051b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f"
Nov 03 17:32:38.213135 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.212960    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e54841-75f1-4d4c-846f-c99ea9051b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71750fe5-xnw9f\" (UID: \"76e54841-75f1-4d4c-846f-c99ea9051b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f"
Nov 03 17:32:38.213214 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.213136    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz6wv\" (UniqueName: \"kubernetes.io/projected/76e54841-75f1-4d4c-846f-c99ea9051b4e-kube-api-access-dz6wv\") pod \"maas-api-revoke-sub-71750fe5-xnw9f\" (UID: \"76e54841-75f1-4d4c-846f-c99ea9051b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f"
Nov 03 17:32:38.247073 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.247045    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:32:38.247278 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.247079    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:32:38.247278 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.247082    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:32:38.247278 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.247219    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:32:38.314280 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.314241    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz6wv\" (UniqueName: \"kubernetes.io/projected/76e54841-75f1-4d4c-846f-c99ea9051b4e-kube-api-access-dz6wv\") pod \"maas-api-revoke-sub-71750fe5-xnw9f\" (UID: \"76e54841-75f1-4d4c-846f-c99ea9051b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f"
Nov 03 17:32:38.314486 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.314389    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e54841-75f1-4d4c-846f-c99ea9051b4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71750fe5-xnw9f\" (UID: \"76e54841-75f1-4d4c-846f-c99ea9051b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f"
Nov 03 17:32:38.314486 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.314411    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e54841-75f1-4d4c-846f-c99ea9051b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71750fe5-xnw9f\" (UID: \"76e54841-75f1-4d4c-846f-c99ea9051b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f"
Nov 03 17:32:38.315067 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.315049    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e54841-75f1-4d4c-846f-c99ea9051b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71750fe5-xnw9f\" (UID: \"76e54841-75f1-4d4c-846f-c99ea9051b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f"
Nov 03 17:32:38.316823 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.316794    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e54841-75f1-4d4c-846f-c99ea9051b4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71750fe5-xnw9f\" (UID: \"76e54841-75f1-4d4c-846f-c99ea9051b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f"
Nov 03 17:32:38.344527 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.344497    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:38.379339 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.379280    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz6wv\" (UniqueName: \"kubernetes.io/projected/76e54841-75f1-4d4c-846f-c99ea9051b4e-kube-api-access-dz6wv\") pod \"maas-api-revoke-sub-71750fe5-xnw9f\" (UID: \"76e54841-75f1-4d4c-846f-c99ea9051b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f"
Nov 03 17:32:38.454822 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.454730    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f"
Nov 03 17:32:38.832900 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:32:38.832869    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e54841_75f1_4d4c_846f_c99ea9051b4e.slice/crio-483b7432dcece237b765b3d00b3be92b3286fc1858ad05b7f20c90875eebdc71 WatchSource:0}: Error finding container 483b7432dcece237b765b3d00b3be92b3286fc1858ad05b7f20c90875eebdc71: Status 404 returned error can't find the container with id 483b7432dcece237b765b3d00b3be92b3286fc1858ad05b7f20c90875eebdc71
Nov 03 17:32:38.837074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:38.837016    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f"]
Nov 03 17:32:39.218873 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:39.218827    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f" event={"ID":"76e54841-75f1-4d4c-846f-c99ea9051b4e","Type":"ContainerStarted","Data":"0fabdaf023c5a6c6382e145c0ad94e78c3e3123c8160c967e64df810be26db34"}
Nov 03 17:32:39.219404 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:39.218911    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f" event={"ID":"76e54841-75f1-4d4c-846f-c99ea9051b4e","Type":"ContainerStarted","Data":"483b7432dcece237b765b3d00b3be92b3286fc1858ad05b7f20c90875eebdc71"}
Nov 03 17:32:39.344796 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:39.344743    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f" podStartSLOduration=1.344729421 podStartE2EDuration="1.344729421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:32:39.343052782 +0000 UTC m=+900.140765564" watchObservedRunningTime="2026-11-03 17:32:39.344729421 +0000 UTC m=+900.142442202"
Nov 03 17:32:39.443138 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:39.443059    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 17:32:39.449766 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:39.449732    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 17:32:39.461316 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:39.461285    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:32:39.468575 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:39.468549    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:32:40.046066 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:40.046031    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:40.226712 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:40.226678    6059 generic.go:356] "Generic (PLEG): container finished" podID="76e54841-75f1-4d4c-846f-c99ea9051b4e" containerID="0fabdaf023c5a6c6382e145c0ad94e78c3e3123c8160c967e64df810be26db34" exitCode=0
Nov 03 17:32:40.227096 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:40.226762    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f" event={"ID":"76e54841-75f1-4d4c-846f-c99ea9051b4e","Type":"ContainerDied","Data":"0fabdaf023c5a6c6382e145c0ad94e78c3e3123c8160c967e64df810be26db34"}
Nov 03 17:32:40.854220 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:40.854176    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"]
Nov 03 17:32:40.854927 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:40.854903    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a6b46be-ed11-4708-b1a9-4500870e4d2e" containerName="istio-proxy"
Nov 03 17:32:40.858787 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:40.858763    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:40.944031 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:40.943989    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-token\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:40.944031 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:40.944035    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-workload-certs\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:40.944296 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:40.944064    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-credential-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:40.944296 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:40.944169    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-podinfo\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:40.944296 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:40.944217    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istiod-ca-cert\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:40.944496 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:40.944394    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh4bq\" (UniqueName: \"kubernetes.io/projected/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-kube-api-access-qh4bq\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:40.944496 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:40.944434    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-data\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:40.944562 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:40.944543    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-workload-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:40.944611 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:40.944581    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-envoy\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:40.968368 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:40.968318    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"]
Nov 03 17:32:41.046243 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.046211    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-token\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:41.046243 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.046250    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-workload-certs\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:41.046532 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.046269    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-credential-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:41.046532 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.046301    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-podinfo\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:41.046532 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.046335    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istiod-ca-cert\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:41.046532 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.046456    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh4bq\" (UniqueName: \"kubernetes.io/projected/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-kube-api-access-qh4bq\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:41.046532 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.046498    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-data\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:41.046803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.046575    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-workload-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:41.046803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.046599    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-envoy\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:41.046803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.046683    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-workload-certs\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:41.046803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.046750    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-credential-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:41.046946 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.046869    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-data\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:41.046993 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.046958    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-workload-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:41.047276 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.047256    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istiod-ca-cert\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:41.048678 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.048653    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-envoy\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:41.048935 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.048917    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-podinfo\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:41.109727 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.109655    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-token\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:41.109925 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.109901    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh4bq\" (UniqueName: \"kubernetes.io/projected/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-kube-api-access-qh4bq\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:41.173174 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.173110    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:41.390816 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.390790    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f"
Nov 03 17:32:41.460183 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.460134    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76e54841-75f1-4d4c-846f-c99ea9051b4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e54841-75f1-4d4c-846f-c99ea9051b4e-maas-api-openshift-ca\") pod \"76e54841-75f1-4d4c-846f-c99ea9051b4e\" (UID: \"76e54841-75f1-4d4c-846f-c99ea9051b4e\") "
Nov 03 17:32:41.460414 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.460314    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76e54841-75f1-4d4c-846f-c99ea9051b4e-kube-api-access-dz6wv\" (UniqueName: \"kubernetes.io/projected/76e54841-75f1-4d4c-846f-c99ea9051b4e-kube-api-access-dz6wv\") pod \"76e54841-75f1-4d4c-846f-c99ea9051b4e\" (UID: \"76e54841-75f1-4d4c-846f-c99ea9051b4e\") "
Nov 03 17:32:41.460414 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.460367    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76e54841-75f1-4d4c-846f-c99ea9051b4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e54841-75f1-4d4c-846f-c99ea9051b4e-maas-api-xks-ca\") pod \"76e54841-75f1-4d4c-846f-c99ea9051b4e\" (UID: \"76e54841-75f1-4d4c-846f-c99ea9051b4e\") "
Nov 03 17:32:41.460600 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.460574    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76e54841-75f1-4d4c-846f-c99ea9051b4e-maas-api-openshift-ca" pod "76e54841-75f1-4d4c-846f-c99ea9051b4e" (UID: "76e54841-75f1-4d4c-846f-c99ea9051b4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:32:41.460845 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.460828    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e54841-75f1-4d4c-846f-c99ea9051b4e-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:41.462601 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.462544    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76e54841-75f1-4d4c-846f-c99ea9051b4e-maas-api-xks-ca" pod "76e54841-75f1-4d4c-846f-c99ea9051b4e" (UID: "76e54841-75f1-4d4c-846f-c99ea9051b4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:32:41.462601 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.462582    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e54841-75f1-4d4c-846f-c99ea9051b4e-kube-api-access-dz6wv" pod "76e54841-75f1-4d4c-846f-c99ea9051b4e" (UID: "76e54841-75f1-4d4c-846f-c99ea9051b4e"). InnerVolumeSpecName "kube-api-access-dz6wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:32:41.561650 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.561612    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz6wv\" (UniqueName: \"kubernetes.io/projected/76e54841-75f1-4d4c-846f-c99ea9051b4e-kube-api-access-dz6wv\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:41.561650 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.561645    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e54841-75f1-4d4c-846f-c99ea9051b4e-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:32:41.602391 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:32:41.602334    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9ab6a0_a561_47fa_8cab_156ef9b8f331.slice/crio-dd6f501d577f25d8cb0e299a3b34e0d355e71cb7ba0c3f3e6c45d9f6a8fbdf56 WatchSource:0}: Error finding container dd6f501d577f25d8cb0e299a3b34e0d355e71cb7ba0c3f3e6c45d9f6a8fbdf56: Status 404 returned error can't find the container with id dd6f501d577f25d8cb0e299a3b34e0d355e71cb7ba0c3f3e6c45d9f6a8fbdf56
Nov 03 17:32:41.604889 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.604858    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:32:41.605004 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.604922    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:32:41.605004 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.604954    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:32:41.605458 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:41.605347    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"]
Nov 03 17:32:42.262775 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:42.262737    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds" event={"ID":"4f9ab6a0-a561-47fa-8cab-156ef9b8f331","Type":"ContainerStarted","Data":"0fc98d82881d86a9533ff3d0bfbef9ec3a6f9ac25f88ccda9f8c00d7907c8c3a"}
Nov 03 17:32:42.262954 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:42.262810    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds" event={"ID":"4f9ab6a0-a561-47fa-8cab-156ef9b8f331","Type":"ContainerStarted","Data":"dd6f501d577f25d8cb0e299a3b34e0d355e71cb7ba0c3f3e6c45d9f6a8fbdf56"}
Nov 03 17:32:42.264734 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:42.264712    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f"
Nov 03 17:32:42.264854 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:42.264754    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f" event={"ID":"76e54841-75f1-4d4c-846f-c99ea9051b4e","Type":"ContainerDied","Data":"483b7432dcece237b765b3d00b3be92b3286fc1858ad05b7f20c90875eebdc71"}
Nov 03 17:32:42.264854 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:42.264794    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="483b7432dcece237b765b3d00b3be92b3286fc1858ad05b7f20c90875eebdc71"
Nov 03 17:32:42.390775 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:42.390668    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds" podStartSLOduration=2.390654711 podStartE2EDuration="2.390654711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:32:42.389156156 +0000 UTC m=+903.186868939" watchObservedRunningTime="2026-11-03 17:32:42.390654711 +0000 UTC m=+903.188367491"
Nov 03 17:32:43.173525 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:43.173488    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:43.175399 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:43.175347    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 17:32:43.175498 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:43.175429    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds" podUID="4f9ab6a0-a561-47fa-8cab-156ef9b8f331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 17:32:43.254340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:43.254300    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:44.184378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:44.183795    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 17:32:44.184378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:44.183858    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds" podUID="4f9ab6a0-a561-47fa-8cab-156ef9b8f331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 17:32:45.173864 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:45.173821    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 17:32:45.174177 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:45.174141    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds" podUID="4f9ab6a0-a561-47fa-8cab-156ef9b8f331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 17:32:46.174497 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:46.174457    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 17:32:46.174902 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:46.174523    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds" podUID="4f9ab6a0-a561-47fa-8cab-156ef9b8f331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 17:32:47.174121 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:47.174075    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 17:32:47.174317 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:47.174140    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds" podUID="4f9ab6a0-a561-47fa-8cab-156ef9b8f331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 17:32:47.821344 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:47.821308    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:48.174320 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:48.174235    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 17:32:48.174320 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:48.174294    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds" podUID="4f9ab6a0-a561-47fa-8cab-156ef9b8f331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 17:32:49.178041 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:49.178010    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:49.178462 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:49.178277    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:49.179009 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:49.178993    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:32:50.074845 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.074810    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"]
Nov 03 17:32:50.076486 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.076456    6059 cpu_manager.go:414] "containerMap: removing container" podUID="76e54841-75f1-4d4c-846f-c99ea9051b4e" containerName="revoke-keys"
Nov 03 17:32:50.076632 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.076619    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e54841-75f1-4d4c-846f-c99ea9051b4e" containerName="revoke-keys"
Nov 03 17:32:50.077034 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.077017    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="76e54841-75f1-4d4c-846f-c99ea9051b4e" containerName="revoke-keys"
Nov 03 17:32:50.083417 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.082127    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:32:50.137924 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.137889    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"]
Nov 03 17:32:50.186632 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.186598    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2355f4\"/\"distinct-w3-2355f4-kserve-self-signed-certs\""
Nov 03 17:32:50.188617 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.188579    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2355f4\"/\"default-dockercfg-r6smg\""
Nov 03 17:32:50.188617 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.188578    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2355f4\"/\"kube-root-ca.crt\""
Nov 03 17:32:50.188617 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.188586    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2355f4\"/\"openshift-service-ca.crt\""
Nov 03 17:32:50.191656 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.191633    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:50.203037 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.203002    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvwkq\" (UniqueName: \"kubernetes.io/projected/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-kube-api-access-xvwkq\") pod \"distinct-w3-2355f4-kserve-6c8756668f-n6zcv\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:32:50.203171 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.203150    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-dshm\") pod \"distinct-w3-2355f4-kserve-6c8756668f-n6zcv\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:32:50.203242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.203229    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-model-cache\") pod \"distinct-w3-2355f4-kserve-6c8756668f-n6zcv\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:32:50.203412 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.203394    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-home\") pod \"distinct-w3-2355f4-kserve-6c8756668f-n6zcv\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:32:50.203519 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.203498    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-tls-certs\") pod \"distinct-w3-2355f4-kserve-6c8756668f-n6zcv\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:32:50.203589 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.203576    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-tmp-dir\") pod \"distinct-w3-2355f4-kserve-6c8756668f-n6zcv\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:32:50.304516 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.304475    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-home\") pod \"distinct-w3-2355f4-kserve-6c8756668f-n6zcv\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:32:50.304691 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.304541    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-tls-certs\") pod \"distinct-w3-2355f4-kserve-6c8756668f-n6zcv\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:32:50.304691 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.304611    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-tmp-dir\") pod \"distinct-w3-2355f4-kserve-6c8756668f-n6zcv\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:32:50.304691 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.304641    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvwkq\" (UniqueName: \"kubernetes.io/projected/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-kube-api-access-xvwkq\") pod \"distinct-w3-2355f4-kserve-6c8756668f-n6zcv\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:32:50.304828 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.304725    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-dshm\") pod \"distinct-w3-2355f4-kserve-6c8756668f-n6zcv\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:32:50.304828 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.304789    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-model-cache\") pod \"distinct-w3-2355f4-kserve-6c8756668f-n6zcv\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:32:50.304968 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.304943    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-home\") pod \"distinct-w3-2355f4-kserve-6c8756668f-n6zcv\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:32:50.305074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.305011    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-tmp-dir\") pod \"distinct-w3-2355f4-kserve-6c8756668f-n6zcv\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:32:50.305130 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.305117    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-model-cache\") pod \"distinct-w3-2355f4-kserve-6c8756668f-n6zcv\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:32:50.306892 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.306865    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-dshm\") pod \"distinct-w3-2355f4-kserve-6c8756668f-n6zcv\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:32:50.307259 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.307240    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-tls-certs\") pod \"distinct-w3-2355f4-kserve-6c8756668f-n6zcv\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:32:50.367790 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.367718    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvwkq\" (UniqueName: \"kubernetes.io/projected/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-kube-api-access-xvwkq\") pod \"distinct-w3-2355f4-kserve-6c8756668f-n6zcv\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:32:50.395952 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.395919    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:32:50.808137 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:50.808111    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"]
Nov 03 17:32:51.408449 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:51.408409    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv" event={"ID":"7942e9d1-597f-47a6-9c70-1d174fe4c8e6","Type":"ContainerStarted","Data":"2ed2e3f2640be105c953235c167382db1b0507b58d2313e7e5555999e1c2f16d"}
Nov 03 17:32:51.408873 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:51.408461    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv" event={"ID":"7942e9d1-597f-47a6-9c70-1d174fe4c8e6","Type":"ContainerStarted","Data":"d9718b309cee6579f5de0f11e025d668f9a8877027c900b76a8b7ca3b2902a52"}
Nov 03 17:32:51.531540 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:51.531483    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv" podStartSLOduration=1.531467643 podStartE2EDuration="1.531467643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:32:51.530005021 +0000 UTC m=+912.327717821" watchObservedRunningTime="2026-11-03 17:32:51.531467643 +0000 UTC m=+912.329180423"
Nov 03 17:32:52.396630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:52.396592    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:32:52.429581 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:52.429542    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:57.769071 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:57.769020    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:32:58.324063 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:58.324025    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68"]
Nov 03 17:32:58.328500 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:58.328478    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68"
Nov 03 17:32:58.388064 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:58.388025    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68"]
Nov 03 17:32:58.436509 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:58.436468    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:32:58.436509 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:58.436488    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:32:58.436509 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:58.436488    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:32:58.436750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:58.436517    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:32:58.465292 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:58.465255    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkq72\" (UniqueName: \"kubernetes.io/projected/31608ad3-ae05-4e9c-80ce-4bb324b45f0f-kube-api-access-rkq72\") pod \"maas-api-revoke-sub-9806e398-cbp68\" (UID: \"31608ad3-ae05-4e9c-80ce-4bb324b45f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68"
Nov 03 17:32:58.465530 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:58.465504    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31608ad3-ae05-4e9c-80ce-4bb324b45f0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9806e398-cbp68\" (UID: \"31608ad3-ae05-4e9c-80ce-4bb324b45f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68"
Nov 03 17:32:58.465652 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:58.465633    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31608ad3-ae05-4e9c-80ce-4bb324b45f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9806e398-cbp68\" (UID: \"31608ad3-ae05-4e9c-80ce-4bb324b45f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68"
Nov 03 17:32:58.566596 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:58.566557    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkq72\" (UniqueName: \"kubernetes.io/projected/31608ad3-ae05-4e9c-80ce-4bb324b45f0f-kube-api-access-rkq72\") pod \"maas-api-revoke-sub-9806e398-cbp68\" (UID: \"31608ad3-ae05-4e9c-80ce-4bb324b45f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68"
Nov 03 17:32:58.566789 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:58.566658    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31608ad3-ae05-4e9c-80ce-4bb324b45f0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9806e398-cbp68\" (UID: \"31608ad3-ae05-4e9c-80ce-4bb324b45f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68"
Nov 03 17:32:58.566789 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:58.566701    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31608ad3-ae05-4e9c-80ce-4bb324b45f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9806e398-cbp68\" (UID: \"31608ad3-ae05-4e9c-80ce-4bb324b45f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68"
Nov 03 17:32:58.567455 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:58.567426    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31608ad3-ae05-4e9c-80ce-4bb324b45f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9806e398-cbp68\" (UID: \"31608ad3-ae05-4e9c-80ce-4bb324b45f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68"
Nov 03 17:32:58.569169 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:58.569145    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31608ad3-ae05-4e9c-80ce-4bb324b45f0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9806e398-cbp68\" (UID: \"31608ad3-ae05-4e9c-80ce-4bb324b45f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68"
Nov 03 17:32:58.630227 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:58.630151    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkq72\" (UniqueName: \"kubernetes.io/projected/31608ad3-ae05-4e9c-80ce-4bb324b45f0f-kube-api-access-rkq72\") pod \"maas-api-revoke-sub-9806e398-cbp68\" (UID: \"31608ad3-ae05-4e9c-80ce-4bb324b45f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68"
Nov 03 17:32:58.642881 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:58.642844    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68"
Nov 03 17:32:59.020239 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:32:59.020204    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31608ad3_ae05_4e9c_80ce_4bb324b45f0f.slice/crio-9f0ecb5038e513832ab8922dcb525226b18eb84e9bdcda679c68735cc25c6e49 WatchSource:0}: Error finding container 9f0ecb5038e513832ab8922dcb525226b18eb84e9bdcda679c68735cc25c6e49: Status 404 returned error can't find the container with id 9f0ecb5038e513832ab8922dcb525226b18eb84e9bdcda679c68735cc25c6e49
Nov 03 17:32:59.023097 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:59.023014    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68"]
Nov 03 17:32:59.521195 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:59.521152    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68" event={"ID":"31608ad3-ae05-4e9c-80ce-4bb324b45f0f","Type":"ContainerStarted","Data":"473bb4c7d06c058400511be602750da812a217f1e8178c36bc9767a8e2903d73"}
Nov 03 17:32:59.521434 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:59.521216    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68" event={"ID":"31608ad3-ae05-4e9c-80ce-4bb324b45f0f","Type":"ContainerStarted","Data":"9f0ecb5038e513832ab8922dcb525226b18eb84e9bdcda679c68735cc25c6e49"}
Nov 03 17:32:59.644369 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:32:59.644246    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68" podStartSLOduration=1.644231878 podStartE2EDuration="1.644231878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:32:59.640770138 +0000 UTC m=+920.438482920" watchObservedRunningTime="2026-11-03 17:32:59.644231878 +0000 UTC m=+920.441944658"
Nov 03 17:33:00.397147 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:00.397110    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:33:00.409991 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:00.409962    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:33:00.532243 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:00.532207    6059 generic.go:356] "Generic (PLEG): container finished" podID="31608ad3-ae05-4e9c-80ce-4bb324b45f0f" containerID="473bb4c7d06c058400511be602750da812a217f1e8178c36bc9767a8e2903d73" exitCode=0
Nov 03 17:33:00.532477 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:00.532248    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68" event={"ID":"31608ad3-ae05-4e9c-80ce-4bb324b45f0f","Type":"ContainerDied","Data":"473bb4c7d06c058400511be602750da812a217f1e8178c36bc9767a8e2903d73"}
Nov 03 17:33:00.544167 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:00.544133    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:33:01.692753 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:01.692724    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68"
Nov 03 17:33:01.810948 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:01.810858    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31608ad3-ae05-4e9c-80ce-4bb324b45f0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31608ad3-ae05-4e9c-80ce-4bb324b45f0f-maas-api-openshift-ca\") pod \"31608ad3-ae05-4e9c-80ce-4bb324b45f0f\" (UID: \"31608ad3-ae05-4e9c-80ce-4bb324b45f0f\") "
Nov 03 17:33:01.811147 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:01.810967    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31608ad3-ae05-4e9c-80ce-4bb324b45f0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31608ad3-ae05-4e9c-80ce-4bb324b45f0f-maas-api-xks-ca\") pod \"31608ad3-ae05-4e9c-80ce-4bb324b45f0f\" (UID: \"31608ad3-ae05-4e9c-80ce-4bb324b45f0f\") "
Nov 03 17:33:01.811147 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:01.811047    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31608ad3-ae05-4e9c-80ce-4bb324b45f0f-kube-api-access-rkq72\" (UniqueName: \"kubernetes.io/projected/31608ad3-ae05-4e9c-80ce-4bb324b45f0f-kube-api-access-rkq72\") pod \"31608ad3-ae05-4e9c-80ce-4bb324b45f0f\" (UID: \"31608ad3-ae05-4e9c-80ce-4bb324b45f0f\") "
Nov 03 17:33:01.811313 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:01.811285    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31608ad3-ae05-4e9c-80ce-4bb324b45f0f-maas-api-openshift-ca" pod "31608ad3-ae05-4e9c-80ce-4bb324b45f0f" (UID: "31608ad3-ae05-4e9c-80ce-4bb324b45f0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:33:01.811512 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:01.811496    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31608ad3-ae05-4e9c-80ce-4bb324b45f0f-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:01.813316 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:01.813286    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31608ad3-ae05-4e9c-80ce-4bb324b45f0f-maas-api-xks-ca" pod "31608ad3-ae05-4e9c-80ce-4bb324b45f0f" (UID: "31608ad3-ae05-4e9c-80ce-4bb324b45f0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:33:01.818530 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:01.818494    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31608ad3-ae05-4e9c-80ce-4bb324b45f0f-kube-api-access-rkq72" pod "31608ad3-ae05-4e9c-80ce-4bb324b45f0f" (UID: "31608ad3-ae05-4e9c-80ce-4bb324b45f0f"). InnerVolumeSpecName "kube-api-access-rkq72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:33:01.912259 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:01.912207    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31608ad3-ae05-4e9c-80ce-4bb324b45f0f-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:01.912259 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:01.912251    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkq72\" (UniqueName: \"kubernetes.io/projected/31608ad3-ae05-4e9c-80ce-4bb324b45f0f-kube-api-access-rkq72\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:02.443018 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:02.442472    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:02.547212 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:02.547181    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68"
Nov 03 17:33:02.547391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:02.547215    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68" event={"ID":"31608ad3-ae05-4e9c-80ce-4bb324b45f0f","Type":"ContainerDied","Data":"9f0ecb5038e513832ab8922dcb525226b18eb84e9bdcda679c68735cc25c6e49"}
Nov 03 17:33:02.547391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:02.547263    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f0ecb5038e513832ab8922dcb525226b18eb84e9bdcda679c68735cc25c6e49"
Nov 03 17:33:05.249092 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.249055    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"]
Nov 03 17:33:05.249670 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.249655    6059 cpu_manager.go:414] "containerMap: removing container" podUID="31608ad3-ae05-4e9c-80ce-4bb324b45f0f" containerName="revoke-keys"
Nov 03 17:33:05.249714 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.249671    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31608ad3-ae05-4e9c-80ce-4bb324b45f0f" containerName="revoke-keys"
Nov 03 17:33:05.249758 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.249749    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="31608ad3-ae05-4e9c-80ce-4bb324b45f0f" containerName="revoke-keys"
Nov 03 17:33:05.253228 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.253213    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:05.353720 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.353686    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-dshm\") pod \"test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") " pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:05.353911 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.353762    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-tmp-dir\") pod \"test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") " pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:05.353911 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.353845    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlmvq\" (UniqueName: \"kubernetes.io/projected/83ca6484-7e38-4782-b3a9-57b58bdc3920-kube-api-access-zlmvq\") pod \"test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") " pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:05.353911 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.353891    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-model-cache\") pod \"test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") " pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:05.354030 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.353950    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-home\") pod \"test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") " pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:05.354030 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.354021    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83ca6484-7e38-4782-b3a9-57b58bdc3920-tls-certs\") pod \"test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") " pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:05.359725 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.359692    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"]
Nov 03 17:33:05.360982 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.360955    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3f1cf1\"/\"kube-root-ca.crt\""
Nov 03 17:33:05.361103 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.360991    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3f1cf1\"/\"openshift-service-ca.crt\""
Nov 03 17:33:05.361141 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.361102    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3f1cf1\"/\"test-model-3f1cf1-kserve-self-signed-certs\""
Nov 03 17:33:05.361141 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.361124    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3f1cf1\"/\"default-dockercfg-845xl\""
Nov 03 17:33:05.419020 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.418985    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:05.455276 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.455235    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-model-cache\") pod \"test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") " pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:05.455473 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.455295    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-home\") pod \"test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") " pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:05.455473 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.455349    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83ca6484-7e38-4782-b3a9-57b58bdc3920-tls-certs\") pod \"test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") " pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:05.455473 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.455446    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-dshm\") pod \"test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") " pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:05.455669 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.455500    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-tmp-dir\") pod \"test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") " pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:05.455669 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.455525    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlmvq\" (UniqueName: \"kubernetes.io/projected/83ca6484-7e38-4782-b3a9-57b58bdc3920-kube-api-access-zlmvq\") pod \"test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") " pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:05.455772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.455755    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-model-cache\") pod \"test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") " pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:05.455835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.455802    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-home\") pod \"test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") " pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:05.455891 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.455869    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-tmp-dir\") pod \"test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") " pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:05.457806 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.457785    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-dshm\") pod \"test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") " pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:05.458072 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.458052    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83ca6484-7e38-4782-b3a9-57b58bdc3920-tls-certs\") pod \"test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") " pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:05.519074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.519000    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlmvq\" (UniqueName: \"kubernetes.io/projected/83ca6484-7e38-4782-b3a9-57b58bdc3920-kube-api-access-zlmvq\") pod \"test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") " pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:05.565164 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.565129    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:05.942151 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:33:05.942118    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83ca6484_7e38_4782_b3a9_57b58bdc3920.slice/crio-aabd547ffc81af5fa5089d91bcb54eadd8462ae6476e383a62f9b1bdcbb95749 WatchSource:0}: Error finding container aabd547ffc81af5fa5089d91bcb54eadd8462ae6476e383a62f9b1bdcbb95749: Status 404 returned error can't find the container with id aabd547ffc81af5fa5089d91bcb54eadd8462ae6476e383a62f9b1bdcbb95749
Nov 03 17:33:05.945400 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:05.945340    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"]
Nov 03 17:33:06.387734 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:06.387700    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:06.574808 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:06.574772    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g" event={"ID":"83ca6484-7e38-4782-b3a9-57b58bdc3920","Type":"ContainerStarted","Data":"741eb234c9b2f81453242547032304249bf0fffc82cbaa6e8528343b15e55747"}
Nov 03 17:33:06.574985 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:06.574817    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g" event={"ID":"83ca6484-7e38-4782-b3a9-57b58bdc3920","Type":"ContainerStarted","Data":"aabd547ffc81af5fa5089d91bcb54eadd8462ae6476e383a62f9b1bdcbb95749"}
Nov 03 17:33:06.701253 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:06.701146    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g" podStartSLOduration=1.701132568 podStartE2EDuration="1.701132568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:06.698447443 +0000 UTC m=+927.496160224" watchObservedRunningTime="2026-11-03 17:33:06.701132568 +0000 UTC m=+927.498845349"
Nov 03 17:33:06.870050 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:33:06.870006    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:33:07.062799 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:07.062764    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v"]
Nov 03 17:33:07.065731 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:07.065703    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99aad8a2-z2t6v"]
Nov 03 17:33:07.336272 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:07.336192    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcef9d95-37e7-4d72-bd24-562f646c2ec9" path="/var/lib/kubelet/pods/dcef9d95-37e7-4d72-bd24-562f646c2ec9/volumes"
Nov 03 17:33:07.565893 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:07.565844    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:08.039131 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:08.039102    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:09.697464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:09.697432    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:09.877941 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:09.877909    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:11.113598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.113548    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"]
Nov 03 17:33:11.119183 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.119151    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.238499 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.238457    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49075270-6931-42bc-948c-d4cdfc72c5d0-istiod-ca-cert\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.238706 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.238519    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-podinfo\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.238706 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.238673    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-workload-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.238835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.238725    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-workload-certs\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.238835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.238757    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgtq7\" (UniqueName: \"kubernetes.io/projected/49075270-6931-42bc-948c-d4cdfc72c5d0-kube-api-access-tgtq7\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.238941 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.238924    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-data\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.239006 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.238975    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-envoy\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.239055 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.239023    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-token\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.239110 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.239068    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-credential-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.340636 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.340601    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49075270-6931-42bc-948c-d4cdfc72c5d0-istiod-ca-cert\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.340840 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.340653    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-podinfo\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.340840 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.340796    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-workload-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.340840 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.340833    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-workload-certs\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.341012 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.340862    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgtq7\" (UniqueName: \"kubernetes.io/projected/49075270-6931-42bc-948c-d4cdfc72c5d0-kube-api-access-tgtq7\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.341012 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.340958    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-data\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.341012 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.341007    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-envoy\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.341169 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.341058    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-token\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.341169 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.341113    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-credential-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.341523 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.341496    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49075270-6931-42bc-948c-d4cdfc72c5d0-istiod-ca-cert\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.341670 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.341520    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-credential-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.343108 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.343078    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-workload-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.343213 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.343130    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-data\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.345421 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.344405    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"]
Nov 03 17:33:11.345421 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.344591    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-workload-certs\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.365052 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.364970    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-envoy\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.366897 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.366870    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-podinfo\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.407146 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.407112    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-token\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.407317 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.407245    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgtq7\" (UniqueName: \"kubernetes.io/projected/49075270-6931-42bc-948c-d4cdfc72c5d0-kube-api-access-tgtq7\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.425221 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.425187    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:11.452812 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.452776    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:11.814200 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.814162    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"]
Nov 03 17:33:11.814472 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.814443    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2" podUID="20fc6389-2903-456b-89f3-f2dcda8db98a" containerName="istio-proxy" containerID="cri-o://165e105b5e8720fbf4d53e9aedd7708c731c59f53b6ae412b8e649dcce92eae0" gracePeriod=30
Nov 03 17:33:11.873315 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.873282    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:11.960150 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.960110    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"]
Nov 03 17:33:12.002782 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.991772    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:33:12.002782 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.991864    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:33:12.002782 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:11.991907    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:33:12.618991 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:12.618954    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925" event={"ID":"49075270-6931-42bc-948c-d4cdfc72c5d0","Type":"ContainerStarted","Data":"14a0f0f9e1947f122d8b78e76ea728129f0e762da1267733798a1d743b836bc0"}
Nov 03 17:33:12.619513 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:12.619022    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925" event={"ID":"49075270-6931-42bc-948c-d4cdfc72c5d0","Type":"ContainerStarted","Data":"e7254068f0b87a45c6e91805c5eb4fd71d19dbd60e3f177edace9b62c634e477"}
Nov 03 17:33:12.882619 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:12.882534    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:13.454379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:13.453755    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:13.456200 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:13.456173    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 17:33:13.456337 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:13.456254    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925" podUID="49075270-6931-42bc-948c-d4cdfc72c5d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 17:33:13.987230 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:13.987192    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:14.454440 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.453787    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 17:33:14.454440 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.453850    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925" podUID="49075270-6931-42bc-948c-d4cdfc72c5d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 17:33:14.500307 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.500247    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925" podStartSLOduration=3.500226361 podStartE2EDuration="3.500226361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:12.751731735 +0000 UTC m=+933.549444517" watchObservedRunningTime="2026-11-03 17:33:14.500226361 +0000 UTC m=+935.297939142"
Nov 03 17:33:14.501446 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.501423    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"]
Nov 03 17:33:14.505780 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.505761    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.569704 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.568467    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-029ca0-openshift-default-dockercfg-j5wx4\""
Nov 03 17:33:14.629281 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.628926    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"]
Nov 03 17:33:14.667442 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.667348    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:14.684741 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.684691    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44nm5\" (UniqueName: \"kubernetes.io/projected/3b813b37-7f95-4138-a42d-bb5af19d7e09-kube-api-access-44nm5\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.685090 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.685057    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-token\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.685191 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.685135    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-data\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.685316 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.685290    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-workload-socket\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.685433 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.685413    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-credential-socket\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.685596 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.685578    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b813b37-7f95-4138-a42d-bb5af19d7e09-istiod-ca-cert\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.685844 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.685826    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-workload-certs\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.686018 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.686000    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-podinfo\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.686056 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.686046    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-envoy\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.790383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.787934    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b813b37-7f95-4138-a42d-bb5af19d7e09-istiod-ca-cert\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.790383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.788054    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-workload-certs\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.790383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.788083    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-podinfo\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.790383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.788111    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-envoy\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.790383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.788193    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44nm5\" (UniqueName: \"kubernetes.io/projected/3b813b37-7f95-4138-a42d-bb5af19d7e09-kube-api-access-44nm5\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.790383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.788270    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-token\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.790383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.788325    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-data\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.790383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.788381    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-workload-socket\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.790383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.788454    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-credential-socket\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.790383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.788943    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-credential-socket\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.791221 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.791194    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b813b37-7f95-4138-a42d-bb5af19d7e09-istiod-ca-cert\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.791311 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.791241    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-workload-certs\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.791415 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.791403    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-data\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.791632 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.791610    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-workload-socket\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.799404 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.798930    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-envoy\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.799532 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.799482    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-podinfo\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.856479 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.856444    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-token\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.857022 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.856995    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44nm5\" (UniqueName: \"kubernetes.io/projected/3b813b37-7f95-4138-a42d-bb5af19d7e09-kube-api-access-44nm5\") pod \"e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") " pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:14.857830 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:14.857804    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:15.121167 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:15.121080    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:15.454899 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:15.454805    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 17:33:15.454899 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:15.454876    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925" podUID="49075270-6931-42bc-948c-d4cdfc72c5d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 17:33:15.566192 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:15.566152    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:15.577630 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:33:15.577599    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b813b37_7f95_4138_a42d_bb5af19d7e09.slice/crio-758ab79bff010b0cad8a0b6fe441cf5a40a37f857e745e4f69eb8d52cad49557 WatchSource:0}: Error finding container 758ab79bff010b0cad8a0b6fe441cf5a40a37f857e745e4f69eb8d52cad49557: Status 404 returned error can't find the container with id 758ab79bff010b0cad8a0b6fe441cf5a40a37f857e745e4f69eb8d52cad49557
Nov 03 17:33:15.580116 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:15.580086    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:33:15.580244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:15.580150    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:33:15.580244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:15.580183    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:33:15.581227 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:15.581206    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:15.581544 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:15.581524    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"]
Nov 03 17:33:15.695958 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:15.695916    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk" event={"ID":"3b813b37-7f95-4138-a42d-bb5af19d7e09","Type":"ContainerStarted","Data":"9a2e2e2e41accf8a19e9da57a36975a41c081d143904704ca8e494637fd408f0"}
Nov 03 17:33:15.696141 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:15.695977    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk" event={"ID":"3b813b37-7f95-4138-a42d-bb5af19d7e09","Type":"ContainerStarted","Data":"758ab79bff010b0cad8a0b6fe441cf5a40a37f857e745e4f69eb8d52cad49557"}
Nov 03 17:33:15.708350 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:15.708324    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:33:15.972265 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:15.972163    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk" podStartSLOduration=1.972147364 podStartE2EDuration="1.972147364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:15.969804324 +0000 UTC m=+936.767517102" watchObservedRunningTime="2026-11-03 17:33:15.972147364 +0000 UTC m=+936.769860144"
Nov 03 17:33:16.121603 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:16.121569    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:16.135382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:16.134739    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:16.454906 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:16.454814    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 17:33:16.454906 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:16.454880    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925" podUID="49075270-6931-42bc-948c-d4cdfc72c5d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 17:33:16.702816 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:16.702784    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:16.703842 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:16.703822    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:33:17.164942 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.164914    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:33:17.223542 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.223505    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-credential-socket\") pod \"20fc6389-2903-456b-89f3-f2dcda8db98a\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") "
Nov 03 17:33:17.223743 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.223610    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20fc6389-2903-456b-89f3-f2dcda8db98a-kube-api-access-wzs5h\" (UniqueName: \"kubernetes.io/projected/20fc6389-2903-456b-89f3-f2dcda8db98a-kube-api-access-wzs5h\") pod \"20fc6389-2903-456b-89f3-f2dcda8db98a\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") "
Nov 03 17:33:17.223743 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.223695    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-envoy\") pod \"20fc6389-2903-456b-89f3-f2dcda8db98a\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") "
Nov 03 17:33:17.223743 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.223726    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20fc6389-2903-456b-89f3-f2dcda8db98a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20fc6389-2903-456b-89f3-f2dcda8db98a-istiod-ca-cert\") pod \"20fc6389-2903-456b-89f3-f2dcda8db98a\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") "
Nov 03 17:33:17.223906 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.223780    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-data\") pod \"20fc6389-2903-456b-89f3-f2dcda8db98a\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") "
Nov 03 17:33:17.223906 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.223815    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-credential-socket" pod "20fc6389-2903-456b-89f3-f2dcda8db98a" (UID: "20fc6389-2903-456b-89f3-f2dcda8db98a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:17.223906 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.223869    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-workload-socket\") pod \"20fc6389-2903-456b-89f3-f2dcda8db98a\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") "
Nov 03 17:33:17.224058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.223917    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-podinfo\") pod \"20fc6389-2903-456b-89f3-f2dcda8db98a\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") "
Nov 03 17:33:17.224058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.223965    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-workload-certs\") pod \"20fc6389-2903-456b-89f3-f2dcda8db98a\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") "
Nov 03 17:33:17.224058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.224037    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-token\" (UniqueName: \"kubernetes.io/projected/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-token\") pod \"20fc6389-2903-456b-89f3-f2dcda8db98a\" (UID: \"20fc6389-2903-456b-89f3-f2dcda8db98a\") "
Nov 03 17:33:17.224201 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.224089    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-data" pod "20fc6389-2903-456b-89f3-f2dcda8db98a" (UID: "20fc6389-2903-456b-89f3-f2dcda8db98a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:17.224201 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.224102    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20fc6389-2903-456b-89f3-f2dcda8db98a-istiod-ca-cert" pod "20fc6389-2903-456b-89f3-f2dcda8db98a" (UID: "20fc6389-2903-456b-89f3-f2dcda8db98a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:33:17.224201 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.224185    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-workload-socket" pod "20fc6389-2903-456b-89f3-f2dcda8db98a" (UID: "20fc6389-2903-456b-89f3-f2dcda8db98a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:17.224415 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.224392    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-workload-certs" pod "20fc6389-2903-456b-89f3-f2dcda8db98a" (UID: "20fc6389-2903-456b-89f3-f2dcda8db98a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:17.224610 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.224586    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:17.224689 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.224616    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20fc6389-2903-456b-89f3-f2dcda8db98a-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:17.224689 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.224633    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:17.224689 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.224647    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:17.224689 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.224660    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:17.226057 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.226033    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20fc6389-2903-456b-89f3-f2dcda8db98a-kube-api-access-wzs5h" pod "20fc6389-2903-456b-89f3-f2dcda8db98a" (UID: "20fc6389-2903-456b-89f3-f2dcda8db98a"). InnerVolumeSpecName "kube-api-access-wzs5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:33:17.226149 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.226029    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-envoy" pod "20fc6389-2903-456b-89f3-f2dcda8db98a" (UID: "20fc6389-2903-456b-89f3-f2dcda8db98a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:17.226271 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.226253    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-podinfo" pod "20fc6389-2903-456b-89f3-f2dcda8db98a" (UID: "20fc6389-2903-456b-89f3-f2dcda8db98a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:33:17.226333 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.226306    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-token" pod "20fc6389-2903-456b-89f3-f2dcda8db98a" (UID: "20fc6389-2903-456b-89f3-f2dcda8db98a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:33:17.326241 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.326202    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzs5h\" (UniqueName: \"kubernetes.io/projected/20fc6389-2903-456b-89f3-f2dcda8db98a-kube-api-access-wzs5h\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:17.326241 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.326244    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:17.326489 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.326261    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:17.326489 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.326278    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20fc6389-2903-456b-89f3-f2dcda8db98a-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:17.457923 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.457821    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 17:33:17.457923 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.457898    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925" podUID="49075270-6931-42bc-948c-d4cdfc72c5d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 17:33:17.711068 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.711036    6059 generic.go:356] "Generic (PLEG): container finished" podID="20fc6389-2903-456b-89f3-f2dcda8db98a" containerID="165e105b5e8720fbf4d53e9aedd7708c731c59f53b6ae412b8e649dcce92eae0" exitCode=0
Nov 03 17:33:17.711244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.711086    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"
Nov 03 17:33:17.711244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.711119    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2" event={"ID":"20fc6389-2903-456b-89f3-f2dcda8db98a","Type":"ContainerDied","Data":"165e105b5e8720fbf4d53e9aedd7708c731c59f53b6ae412b8e649dcce92eae0"}
Nov 03 17:33:17.711244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.711176    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2" event={"ID":"20fc6389-2903-456b-89f3-f2dcda8db98a","Type":"ContainerDied","Data":"c2005e722da505ca89da4fde2fa0a17e81a71efb5a85c391ae44a5ef6ed10fdc"}
Nov 03 17:33:17.711244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.711203    6059 scope.go:122] "RemoveContainer" containerID="165e105b5e8720fbf4d53e9aedd7708c731c59f53b6ae412b8e649dcce92eae0"
Nov 03 17:33:17.721499 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.721476    6059 scope.go:122] "RemoveContainer" containerID="a4a3b867238c8ade5daed58061b7b4b9681aef9ad064f4a2ffd73d925efd6b24"
Nov 03 17:33:17.757795 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.757764    6059 scope.go:122] "RemoveContainer" containerID="165e105b5e8720fbf4d53e9aedd7708c731c59f53b6ae412b8e649dcce92eae0"
Nov 03 17:33:17.758302 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:33:17.758268    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165e105b5e8720fbf4d53e9aedd7708c731c59f53b6ae412b8e649dcce92eae0\": container with ID starting with 165e105b5e8720fbf4d53e9aedd7708c731c59f53b6ae412b8e649dcce92eae0 not found: ID does not exist" containerID="165e105b5e8720fbf4d53e9aedd7708c731c59f53b6ae412b8e649dcce92eae0"
Nov 03 17:33:17.758449 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.758312    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165e105b5e8720fbf4d53e9aedd7708c731c59f53b6ae412b8e649dcce92eae0"} err="failed to get container status \"165e105b5e8720fbf4d53e9aedd7708c731c59f53b6ae412b8e649dcce92eae0\": rpc error: code = NotFound desc = could not find container \"165e105b5e8720fbf4d53e9aedd7708c731c59f53b6ae412b8e649dcce92eae0\": container with ID starting with 165e105b5e8720fbf4d53e9aedd7708c731c59f53b6ae412b8e649dcce92eae0 not found: ID does not exist"
Nov 03 17:33:17.758449 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.758337    6059 scope.go:122] "RemoveContainer" containerID="a4a3b867238c8ade5daed58061b7b4b9681aef9ad064f4a2ffd73d925efd6b24"
Nov 03 17:33:17.758702 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:33:17.758677    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4a3b867238c8ade5daed58061b7b4b9681aef9ad064f4a2ffd73d925efd6b24\": container with ID starting with a4a3b867238c8ade5daed58061b7b4b9681aef9ad064f4a2ffd73d925efd6b24 not found: ID does not exist" containerID="a4a3b867238c8ade5daed58061b7b4b9681aef9ad064f4a2ffd73d925efd6b24"
Nov 03 17:33:17.758775 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.758710    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4a3b867238c8ade5daed58061b7b4b9681aef9ad064f4a2ffd73d925efd6b24"} err="failed to get container status \"a4a3b867238c8ade5daed58061b7b4b9681aef9ad064f4a2ffd73d925efd6b24\": rpc error: code = NotFound desc = could not find container \"a4a3b867238c8ade5daed58061b7b4b9681aef9ad064f4a2ffd73d925efd6b24\": container with ID starting with a4a3b867238c8ade5daed58061b7b4b9681aef9ad064f4a2ffd73d925efd6b24 not found: ID does not exist"
Nov 03 17:33:17.818218 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.818179    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:17.886209 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.886154    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"]
Nov 03 17:33:17.891197 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:17.891168    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-3e5c4d-openshift-default-6c4d677745-89dx2"]
Nov 03 17:33:18.453208 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:18.453166    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 17:33:18.453600 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:18.453226    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925" podUID="49075270-6931-42bc-948c-d4cdfc72c5d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 17:33:18.903677 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:18.903628    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:19.173838 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:19.173754    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:19.336507 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:19.336469    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20fc6389-2903-456b-89f3-f2dcda8db98a" path="/var/lib/kubelet/pods/20fc6389-2903-456b-89f3-f2dcda8db98a/volumes"
Nov 03 17:33:19.457998 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:19.457967    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:19.458416 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:19.458276    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:19.458966 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:19.458949    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:33:20.713316 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:20.713284    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:22.814351 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:22.814315    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:24.046597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:24.046565    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:24.246026 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:24.245985    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:24.425110 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:24.425016    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp"]
Nov 03 17:33:24.425694 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:24.425673    6059 cpu_manager.go:414] "containerMap: removing container" podUID="20fc6389-2903-456b-89f3-f2dcda8db98a" containerName="istio-proxy"
Nov 03 17:33:24.425694 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:24.425694    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20fc6389-2903-456b-89f3-f2dcda8db98a" containerName="istio-proxy"
Nov 03 17:33:24.425854 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:24.425746    6059 cpu_manager.go:414] "containerMap: removing container" podUID="20fc6389-2903-456b-89f3-f2dcda8db98a" containerName="istio-proxy"
Nov 03 17:33:24.425854 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:24.425755    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20fc6389-2903-456b-89f3-f2dcda8db98a" containerName="istio-proxy"
Nov 03 17:33:24.425948 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:24.425889    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="20fc6389-2903-456b-89f3-f2dcda8db98a" containerName="istio-proxy"
Nov 03 17:33:24.425948 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:24.425910    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="20fc6389-2903-456b-89f3-f2dcda8db98a" containerName="istio-proxy"
Nov 03 17:33:24.429635 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:24.429612    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp"
Nov 03 17:33:24.489230 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:24.489194    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp"]
Nov 03 17:33:24.536378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:24.535892    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be0c0d23-e8b2-43aa-a7a2-921718dde1b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp\" (UID: \"be0c0d23-e8b2-43aa-a7a2-921718dde1b1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp"
Nov 03 17:33:24.536378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:24.535992    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd88r\" (UniqueName: \"kubernetes.io/projected/be0c0d23-e8b2-43aa-a7a2-921718dde1b1-kube-api-access-xd88r\") pod \"payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp\" (UID: \"be0c0d23-e8b2-43aa-a7a2-921718dde1b1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp"
Nov 03 17:33:24.544378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:24.538538    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-029ca0\""
Nov 03 17:33:24.544378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:24.538718    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-029ca0-dockercfg-cd8zn\""
Nov 03 17:33:24.637812 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:24.637767    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be0c0d23-e8b2-43aa-a7a2-921718dde1b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp\" (UID: \"be0c0d23-e8b2-43aa-a7a2-921718dde1b1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp"
Nov 03 17:33:24.638002 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:24.637872    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd88r\" (UniqueName: \"kubernetes.io/projected/be0c0d23-e8b2-43aa-a7a2-921718dde1b1-kube-api-access-xd88r\") pod \"payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp\" (UID: \"be0c0d23-e8b2-43aa-a7a2-921718dde1b1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp"
Nov 03 17:33:24.638750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:24.638720    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be0c0d23-e8b2-43aa-a7a2-921718dde1b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp\" (UID: \"be0c0d23-e8b2-43aa-a7a2-921718dde1b1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp"
Nov 03 17:33:24.701471 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:24.701397    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd88r\" (UniqueName: \"kubernetes.io/projected/be0c0d23-e8b2-43aa-a7a2-921718dde1b1-kube-api-access-xd88r\") pod \"payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp\" (UID: \"be0c0d23-e8b2-43aa-a7a2-921718dde1b1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp"
Nov 03 17:33:24.743415 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:24.743346    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp"
Nov 03 17:33:25.123014 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:33:25.122980    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0c0d23_e8b2_43aa_a7a2_921718dde1b1.slice/crio-5c370a91ad0206d5388bad2b49528ed91462a4757fc8c23a20861c67f0eda568 WatchSource:0}: Error finding container 5c370a91ad0206d5388bad2b49528ed91462a4757fc8c23a20861c67f0eda568: Status 404 returned error can't find the container with id 5c370a91ad0206d5388bad2b49528ed91462a4757fc8c23a20861c67f0eda568
Nov 03 17:33:25.126147 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:25.126121    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp"]
Nov 03 17:33:25.806795 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:25.806761    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp" event={"ID":"be0c0d23-e8b2-43aa-a7a2-921718dde1b1","Type":"ContainerStarted","Data":"ddd5efa584a6f5effac4ccf7cf21bf988e88d6ae2c87fce3a871144fac29a206"}
Nov 03 17:33:25.807013 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:25.806815    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp" event={"ID":"be0c0d23-e8b2-43aa-a7a2-921718dde1b1","Type":"ContainerStarted","Data":"5c370a91ad0206d5388bad2b49528ed91462a4757fc8c23a20861c67f0eda568"}
Nov 03 17:33:25.807013 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:25.806854    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp"
Nov 03 17:33:25.927647 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:25.927579    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp" podStartSLOduration=1.927560124 podStartE2EDuration="1.927560124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:25.927029588 +0000 UTC m=+946.724742370" watchObservedRunningTime="2026-11-03 17:33:25.927560124 +0000 UTC m=+946.725272908"
Nov 03 17:33:27.436688 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:27.436653    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:31.083215 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:31.083183    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68"]
Nov 03 17:33:31.124380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:31.120665    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68"
Nov 03 17:33:31.172380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:31.169698    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68"]
Nov 03 17:33:31.203452 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:31.202936    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:33:31.203452 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:31.202939    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:33:31.203452 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:31.203060    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:33:31.239421 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:31.238478    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29wz4\" (UniqueName: \"kubernetes.io/projected/c8e67b74-9b61-4832-9e2d-f79287894fa9-kube-api-access-29wz4\") pod \"maas-api-revoke-sub-2ebb2cfe-ssk68\" (UID: \"c8e67b74-9b61-4832-9e2d-f79287894fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68"
Nov 03 17:33:31.239421 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:31.238586    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e67b74-9b61-4832-9e2d-f79287894fa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ebb2cfe-ssk68\" (UID: \"c8e67b74-9b61-4832-9e2d-f79287894fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68"
Nov 03 17:33:31.239421 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:31.238641    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e67b74-9b61-4832-9e2d-f79287894fa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ebb2cfe-ssk68\" (UID: \"c8e67b74-9b61-4832-9e2d-f79287894fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68"
Nov 03 17:33:31.248979 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:31.248946    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:33:31.359439 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:31.359158    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e67b74-9b61-4832-9e2d-f79287894fa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ebb2cfe-ssk68\" (UID: \"c8e67b74-9b61-4832-9e2d-f79287894fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68"
Nov 03 17:33:31.359439 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:31.359233    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e67b74-9b61-4832-9e2d-f79287894fa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ebb2cfe-ssk68\" (UID: \"c8e67b74-9b61-4832-9e2d-f79287894fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68"
Nov 03 17:33:31.359688 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:31.359509    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29wz4\" (UniqueName: \"kubernetes.io/projected/c8e67b74-9b61-4832-9e2d-f79287894fa9-kube-api-access-29wz4\") pod \"maas-api-revoke-sub-2ebb2cfe-ssk68\" (UID: \"c8e67b74-9b61-4832-9e2d-f79287894fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68"
Nov 03 17:33:31.377390 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:31.360738    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e67b74-9b61-4832-9e2d-f79287894fa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ebb2cfe-ssk68\" (UID: \"c8e67b74-9b61-4832-9e2d-f79287894fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68"
Nov 03 17:33:31.401014 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:31.398724    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e67b74-9b61-4832-9e2d-f79287894fa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ebb2cfe-ssk68\" (UID: \"c8e67b74-9b61-4832-9e2d-f79287894fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68"
Nov 03 17:33:31.429536 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:31.429455    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29wz4\" (UniqueName: \"kubernetes.io/projected/c8e67b74-9b61-4832-9e2d-f79287894fa9-kube-api-access-29wz4\") pod \"maas-api-revoke-sub-2ebb2cfe-ssk68\" (UID: \"c8e67b74-9b61-4832-9e2d-f79287894fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68"
Nov 03 17:33:31.502986 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:31.502652    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68"
Nov 03 17:33:32.195367 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:33:32.195323    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e67b74_9b61_4832_9e2d_f79287894fa9.slice/crio-422fed00ebc23d0f3537ccef490c52c6c255761cb58962cf3a19cd65e3f50f72 WatchSource:0}: Error finding container 422fed00ebc23d0f3537ccef490c52c6c255761cb58962cf3a19cd65e3f50f72: Status 404 returned error can't find the container with id 422fed00ebc23d0f3537ccef490c52c6c255761cb58962cf3a19cd65e3f50f72
Nov 03 17:33:32.198599 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:32.198570    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68"]
Nov 03 17:33:32.646584 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:32.646551    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:33.015483 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:33.015446    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68" event={"ID":"c8e67b74-9b61-4832-9e2d-f79287894fa9","Type":"ContainerStarted","Data":"371a38a396a387004193ebfe30fc564eb08975e2821b9d8f5a97fac34b154c84"}
Nov 03 17:33:33.015654 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:33.015501    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68" event={"ID":"c8e67b74-9b61-4832-9e2d-f79287894fa9","Type":"ContainerStarted","Data":"422fed00ebc23d0f3537ccef490c52c6c255761cb58962cf3a19cd65e3f50f72"}
Nov 03 17:33:33.136749 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:33.136698    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68" podStartSLOduration=2.136684008 podStartE2EDuration="2.136684008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:33.135718528 +0000 UTC m=+953.933431311" watchObservedRunningTime="2026-11-03 17:33:33.136684008 +0000 UTC m=+953.934396789"
Nov 03 17:33:34.024126 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:34.024095    6059 generic.go:356] "Generic (PLEG): container finished" podID="c8e67b74-9b61-4832-9e2d-f79287894fa9" containerID="371a38a396a387004193ebfe30fc564eb08975e2821b9d8f5a97fac34b154c84" exitCode=0
Nov 03 17:33:34.024611 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:34.024159    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68" event={"ID":"c8e67b74-9b61-4832-9e2d-f79287894fa9","Type":"ContainerDied","Data":"371a38a396a387004193ebfe30fc564eb08975e2821b9d8f5a97fac34b154c84"}
Nov 03 17:33:35.193974 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:35.193947    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68"
Nov 03 17:33:35.281202 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:35.281168    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8e67b74-9b61-4832-9e2d-f79287894fa9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e67b74-9b61-4832-9e2d-f79287894fa9-maas-api-xks-ca\") pod \"c8e67b74-9b61-4832-9e2d-f79287894fa9\" (UID: \"c8e67b74-9b61-4832-9e2d-f79287894fa9\") "
Nov 03 17:33:35.281401 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:35.281311    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8e67b74-9b61-4832-9e2d-f79287894fa9-kube-api-access-29wz4\" (UniqueName: \"kubernetes.io/projected/c8e67b74-9b61-4832-9e2d-f79287894fa9-kube-api-access-29wz4\") pod \"c8e67b74-9b61-4832-9e2d-f79287894fa9\" (UID: \"c8e67b74-9b61-4832-9e2d-f79287894fa9\") "
Nov 03 17:33:35.281503 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:35.281491    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8e67b74-9b61-4832-9e2d-f79287894fa9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e67b74-9b61-4832-9e2d-f79287894fa9-maas-api-openshift-ca\") pod \"c8e67b74-9b61-4832-9e2d-f79287894fa9\" (UID: \"c8e67b74-9b61-4832-9e2d-f79287894fa9\") "
Nov 03 17:33:35.281848 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:35.281819    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e67b74-9b61-4832-9e2d-f79287894fa9-maas-api-openshift-ca" pod "c8e67b74-9b61-4832-9e2d-f79287894fa9" (UID: "c8e67b74-9b61-4832-9e2d-f79287894fa9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:33:35.283269 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:35.283246    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8e67b74-9b61-4832-9e2d-f79287894fa9-maas-api-xks-ca" pod "c8e67b74-9b61-4832-9e2d-f79287894fa9" (UID: "c8e67b74-9b61-4832-9e2d-f79287894fa9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:33:35.283348 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:35.283299    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e67b74-9b61-4832-9e2d-f79287894fa9-kube-api-access-29wz4" pod "c8e67b74-9b61-4832-9e2d-f79287894fa9" (UID: "c8e67b74-9b61-4832-9e2d-f79287894fa9"). InnerVolumeSpecName "kube-api-access-29wz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:33:35.383242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:35.383210    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29wz4\" (UniqueName: \"kubernetes.io/projected/c8e67b74-9b61-4832-9e2d-f79287894fa9-kube-api-access-29wz4\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:35.383242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:35.383237    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e67b74-9b61-4832-9e2d-f79287894fa9-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:35.383242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:35.383250    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e67b74-9b61-4832-9e2d-f79287894fa9-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:35.488974 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:35.488921    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:36.039936 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:36.039899    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68" event={"ID":"c8e67b74-9b61-4832-9e2d-f79287894fa9","Type":"ContainerDied","Data":"422fed00ebc23d0f3537ccef490c52c6c255761cb58962cf3a19cd65e3f50f72"}
Nov 03 17:33:36.039936 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:36.039938    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68"
Nov 03 17:33:36.040118 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:36.039955    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="422fed00ebc23d0f3537ccef490c52c6c255761cb58962cf3a19cd65e3f50f72"
Nov 03 17:33:36.816872 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:36.816842    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp"
Nov 03 17:33:36.861466 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:33:36.861430    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:33:37.238757 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:37.238708    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:39.720342 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.720310    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"]
Nov 03 17:33:39.720882 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.720868    6059 cpu_manager.go:414] "containerMap: removing container" podUID="c8e67b74-9b61-4832-9e2d-f79287894fa9" containerName="revoke-keys"
Nov 03 17:33:39.720928 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.720883    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e67b74-9b61-4832-9e2d-f79287894fa9" containerName="revoke-keys"
Nov 03 17:33:39.720991 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.720981    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e67b74-9b61-4832-9e2d-f79287894fa9" containerName="revoke-keys"
Nov 03 17:33:39.728153 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.728131    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:39.785675 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.785604    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"]
Nov 03 17:33:39.832641 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.832607    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2355f4\"/\"distinct-2-w3-2355f4-kserve-self-signed-certs\""
Nov 03 17:33:39.839343 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.839317    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmwrl\" (UniqueName: \"kubernetes.io/projected/d6c1263b-e648-4f10-af7d-86397d6b207a-kube-api-access-kmwrl\") pod \"distinct-2-w3-2355f4-kserve-59958c9658-vtz5n\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:39.839484 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.839407    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-model-cache\") pod \"distinct-2-w3-2355f4-kserve-59958c9658-vtz5n\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:39.839527 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.839486    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-tmp-dir\") pod \"distinct-2-w3-2355f4-kserve-59958c9658-vtz5n\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:39.839566 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.839532    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-dshm\") pod \"distinct-2-w3-2355f4-kserve-59958c9658-vtz5n\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:39.839649 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.839632    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-home\") pod \"distinct-2-w3-2355f4-kserve-59958c9658-vtz5n\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:39.839709 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.839697    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c1263b-e648-4f10-af7d-86397d6b207a-tls-certs\") pod \"distinct-2-w3-2355f4-kserve-59958c9658-vtz5n\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:39.842670 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.842637    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:39.941023 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.940989    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-tmp-dir\") pod \"distinct-2-w3-2355f4-kserve-59958c9658-vtz5n\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:39.941203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.941056    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-dshm\") pod \"distinct-2-w3-2355f4-kserve-59958c9658-vtz5n\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:39.941203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.941097    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-home\") pod \"distinct-2-w3-2355f4-kserve-59958c9658-vtz5n\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:39.941203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.941149    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c1263b-e648-4f10-af7d-86397d6b207a-tls-certs\") pod \"distinct-2-w3-2355f4-kserve-59958c9658-vtz5n\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:39.941342 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.941239    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmwrl\" (UniqueName: \"kubernetes.io/projected/d6c1263b-e648-4f10-af7d-86397d6b207a-kube-api-access-kmwrl\") pod \"distinct-2-w3-2355f4-kserve-59958c9658-vtz5n\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:39.941342 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.941284    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-model-cache\") pod \"distinct-2-w3-2355f4-kserve-59958c9658-vtz5n\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:39.941503 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.941413    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-tmp-dir\") pod \"distinct-2-w3-2355f4-kserve-59958c9658-vtz5n\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:39.941580 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.941541    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-home\") pod \"distinct-2-w3-2355f4-kserve-59958c9658-vtz5n\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:39.941670 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.941635    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-model-cache\") pod \"distinct-2-w3-2355f4-kserve-59958c9658-vtz5n\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:39.943451 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.943430    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-dshm\") pod \"distinct-2-w3-2355f4-kserve-59958c9658-vtz5n\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:39.943728 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:39.943705    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c1263b-e648-4f10-af7d-86397d6b207a-tls-certs\") pod \"distinct-2-w3-2355f4-kserve-59958c9658-vtz5n\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:40.007796 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:40.007764    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmwrl\" (UniqueName: \"kubernetes.io/projected/d6c1263b-e648-4f10-af7d-86397d6b207a-kube-api-access-kmwrl\") pod \"distinct-2-w3-2355f4-kserve-59958c9658-vtz5n\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") " pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:40.045285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:40.045210    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:40.482447 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:33:40.482415    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6c1263b_e648_4f10_af7d_86397d6b207a.slice/crio-90582841d9011b5798b2d29a43f8241444f63895004383b6f3f4478bdb4c9a5b WatchSource:0}: Error finding container 90582841d9011b5798b2d29a43f8241444f63895004383b6f3f4478bdb4c9a5b: Status 404 returned error can't find the container with id 90582841d9011b5798b2d29a43f8241444f63895004383b6f3f4478bdb4c9a5b
Nov 03 17:33:40.486069 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:40.486044    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"]
Nov 03 17:33:41.096200 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:41.096166    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n" event={"ID":"d6c1263b-e648-4f10-af7d-86397d6b207a","Type":"ContainerStarted","Data":"a656aff129f93a9d7d50f083585d75cad313c85465b5153c0707c840968a35e6"}
Nov 03 17:33:41.096633 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:41.096208    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n" event={"ID":"d6c1263b-e648-4f10-af7d-86397d6b207a","Type":"ContainerStarted","Data":"90582841d9011b5798b2d29a43f8241444f63895004383b6f3f4478bdb4c9a5b"}
Nov 03 17:33:41.222460 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:41.222404    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n" podStartSLOduration=2.222389297 podStartE2EDuration="2.222389297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:41.221164592 +0000 UTC m=+962.018877373" watchObservedRunningTime="2026-11-03 17:33:41.222389297 +0000 UTC m=+962.020102078"
Nov 03 17:33:41.829289 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:41.829251    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:42.045660 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:42.045627    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:45.105444 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:45.105408    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"]
Nov 03 17:33:45.105907 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:45.105597    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw" podUID="04181a54-5cd2-4dee-b102-5955e3e9b5a3" containerName="istio-proxy" containerID="cri-o://2a26b9b5de878006e2c7dbcec5a94d728792d35258240e7a9d3ff4c5c076287d" gracePeriod=30
Nov 03 17:33:45.157083 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:45.157043    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"]
Nov 03 17:33:45.157270 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:45.157253    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d" podUID="c097188b-4b64-4636-822b-ab2c8817dfa0" containerName="istio-proxy" containerID="cri-o://58f02de98c6f04c8694d8c7f340a484d92974d0e41245245afbb5c44e810e122" gracePeriod=30
Nov 03 17:33:45.159994 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:45.159956    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"]
Nov 03 17:33:45.160163 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:45.160140    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9" podUID="1db59e38-c479-48cd-b641-48c6e44a0b23" containerName="istio-proxy" containerID="cri-o://d6217027333c556c73a797c90cdd7e0f1aa6801a16eab7c1dd7e17c853582af5" gracePeriod=30
Nov 03 17:33:45.209593 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:45.209555    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"]
Nov 03 17:33:45.209800 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:45.209774    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn" podUID="e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb" containerName="istio-proxy" containerID="cri-o://5e1cf1c5fe254aedbb4f43169db791e3199427bc008159b5858a57683616f0b1" gracePeriod=30
Nov 03 17:33:46.455991 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:46.455958    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:48.061779 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:48.061734    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:48.876669 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:48.876628    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a92b5d-bvl8d"]
Nov 03 17:33:48.891031 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:48.890997    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a92b5d-bvl8d"
Nov 03 17:33:48.938381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:48.925187    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a92b5d-bvl8d"]
Nov 03 17:33:48.983491 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:48.983453    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:33:48.984343 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:48.984233    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:33:48.984822 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:48.984801    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:33:49.031897 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:49.022741    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:33:49.086898 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:49.086447    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l24w\" (UniqueName: \"kubernetes.io/projected/4e3b666b-5066-4524-a552-cbc829bd11b4-kube-api-access-2l24w\") pod \"maas-api-revoke-sub-40a92b5d-bvl8d\" (UID: \"4e3b666b-5066-4524-a552-cbc829bd11b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a92b5d-bvl8d"
Nov 03 17:33:49.086898 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:49.086634    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e3b666b-5066-4524-a552-cbc829bd11b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40a92b5d-bvl8d\" (UID: \"4e3b666b-5066-4524-a552-cbc829bd11b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a92b5d-bvl8d"
Nov 03 17:33:49.086898 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:49.086669    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e3b666b-5066-4524-a552-cbc829bd11b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40a92b5d-bvl8d\" (UID: \"4e3b666b-5066-4524-a552-cbc829bd11b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a92b5d-bvl8d"
Nov 03 17:33:49.194808 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:49.194712    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e3b666b-5066-4524-a552-cbc829bd11b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40a92b5d-bvl8d\" (UID: \"4e3b666b-5066-4524-a552-cbc829bd11b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a92b5d-bvl8d"
Nov 03 17:33:49.218527 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:49.218486    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e3b666b-5066-4524-a552-cbc829bd11b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40a92b5d-bvl8d\" (UID: \"4e3b666b-5066-4524-a552-cbc829bd11b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a92b5d-bvl8d"
Nov 03 17:33:49.218953 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:49.218931    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l24w\" (UniqueName: \"kubernetes.io/projected/4e3b666b-5066-4524-a552-cbc829bd11b4-kube-api-access-2l24w\") pod \"maas-api-revoke-sub-40a92b5d-bvl8d\" (UID: \"4e3b666b-5066-4524-a552-cbc829bd11b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a92b5d-bvl8d"
Nov 03 17:33:49.221260 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:49.221232    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e3b666b-5066-4524-a552-cbc829bd11b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40a92b5d-bvl8d\" (UID: \"4e3b666b-5066-4524-a552-cbc829bd11b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a92b5d-bvl8d"
Nov 03 17:33:49.253381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:49.253220    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e3b666b-5066-4524-a552-cbc829bd11b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40a92b5d-bvl8d\" (UID: \"4e3b666b-5066-4524-a552-cbc829bd11b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a92b5d-bvl8d"
Nov 03 17:33:49.313390 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:49.301031    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l24w\" (UniqueName: \"kubernetes.io/projected/4e3b666b-5066-4524-a552-cbc829bd11b4-kube-api-access-2l24w\") pod \"maas-api-revoke-sub-40a92b5d-bvl8d\" (UID: \"4e3b666b-5066-4524-a552-cbc829bd11b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a92b5d-bvl8d"
Nov 03 17:33:49.529172 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:49.529139    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a92b5d-bvl8d"
Nov 03 17:33:49.745215 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:49.745175    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:49.946872 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:33:49.946830    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e3b666b_5066_4524_a552_cbc829bd11b4.slice/crio-2ba561e4576c61b6477bb6a6daf15086109df91767ae7f06883c0ae5b3acff2c WatchSource:0}: Error finding container 2ba561e4576c61b6477bb6a6daf15086109df91767ae7f06883c0ae5b3acff2c: Status 404 returned error can't find the container with id 2ba561e4576c61b6477bb6a6daf15086109df91767ae7f06883c0ae5b3acff2c
Nov 03 17:33:49.951266 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:49.951241    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a92b5d-bvl8d"]
Nov 03 17:33:50.045959 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.045920    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:50.058647 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.058612    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:50.246086 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.246048    6059 generic.go:356] "Generic (PLEG): container finished" podID="04181a54-5cd2-4dee-b102-5955e3e9b5a3" containerID="2a26b9b5de878006e2c7dbcec5a94d728792d35258240e7a9d3ff4c5c076287d" exitCode=0
Nov 03 17:33:50.246504 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.246167    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw" event={"ID":"04181a54-5cd2-4dee-b102-5955e3e9b5a3","Type":"ContainerDied","Data":"2a26b9b5de878006e2c7dbcec5a94d728792d35258240e7a9d3ff4c5c076287d"}
Nov 03 17:33:50.247701 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.247679    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a92b5d-bvl8d" event={"ID":"4e3b666b-5066-4524-a552-cbc829bd11b4","Type":"ContainerStarted","Data":"2ba561e4576c61b6477bb6a6daf15086109df91767ae7f06883c0ae5b3acff2c"}
Nov 03 17:33:50.259295 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.259274    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:33:50.475371 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.475326    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:33:50.552730 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.552693    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04181a54-5cd2-4dee-b102-5955e3e9b5a3-kube-api-access-flsls\" (UniqueName: \"kubernetes.io/projected/04181a54-5cd2-4dee-b102-5955e3e9b5a3-kube-api-access-flsls\") pod \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") "
Nov 03 17:33:50.552917 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.552810    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-podinfo\") pod \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") "
Nov 03 17:33:50.552917 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.552879    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-workload-socket\") pod \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") "
Nov 03 17:33:50.552917 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.552907    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-envoy\") pod \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") "
Nov 03 17:33:50.553093 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.552955    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-workload-certs\") pod \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") "
Nov 03 17:33:50.553093 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.553067    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-data\") pod \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") "
Nov 03 17:33:50.553205 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.553144    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-credential-socket\") pod \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") "
Nov 03 17:33:50.553266 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.553220    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-token\" (UniqueName: \"kubernetes.io/projected/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-token\") pod \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") "
Nov 03 17:33:50.553325 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.553295    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istiod-ca-cert\") pod \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\" (UID: \"04181a54-5cd2-4dee-b102-5955e3e9b5a3\") "
Nov 03 17:33:50.554225 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.553692    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-workload-certs" pod "04181a54-5cd2-4dee-b102-5955e3e9b5a3" (UID: "04181a54-5cd2-4dee-b102-5955e3e9b5a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:50.554225 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.553909    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.554225 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.554176    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-workload-socket" pod "04181a54-5cd2-4dee-b102-5955e3e9b5a3" (UID: "04181a54-5cd2-4dee-b102-5955e3e9b5a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:50.554225 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.554190    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-data" pod "04181a54-5cd2-4dee-b102-5955e3e9b5a3" (UID: "04181a54-5cd2-4dee-b102-5955e3e9b5a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:50.554545 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.554267    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-credential-socket" pod "04181a54-5cd2-4dee-b102-5955e3e9b5a3" (UID: "04181a54-5cd2-4dee-b102-5955e3e9b5a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:50.555635 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.555609    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istiod-ca-cert" pod "04181a54-5cd2-4dee-b102-5955e3e9b5a3" (UID: "04181a54-5cd2-4dee-b102-5955e3e9b5a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:33:50.556949 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.556925    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04181a54-5cd2-4dee-b102-5955e3e9b5a3-kube-api-access-flsls" pod "04181a54-5cd2-4dee-b102-5955e3e9b5a3" (UID: "04181a54-5cd2-4dee-b102-5955e3e9b5a3"). InnerVolumeSpecName "kube-api-access-flsls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:33:50.561081 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.561032    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-envoy" pod "04181a54-5cd2-4dee-b102-5955e3e9b5a3" (UID: "04181a54-5cd2-4dee-b102-5955e3e9b5a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:50.561245 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.561219    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-podinfo" pod "04181a54-5cd2-4dee-b102-5955e3e9b5a3" (UID: "04181a54-5cd2-4dee-b102-5955e3e9b5a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:33:50.574410 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.574312    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-token" pod "04181a54-5cd2-4dee-b102-5955e3e9b5a3" (UID: "04181a54-5cd2-4dee-b102-5955e3e9b5a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:33:50.586508 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.586481    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:33:50.589158 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.589138    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:33:50.595558 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.595539    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:33:50.655014 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.654981    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-credential-socket\") pod \"c097188b-4b64-4636-822b-ab2c8817dfa0\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") "
Nov 03 17:33:50.655216 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655032    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-data\") pod \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") "
Nov 03 17:33:50.655216 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655070    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db59e38-c479-48cd-b641-48c6e44a0b23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1db59e38-c479-48cd-b641-48c6e44a0b23-istiod-ca-cert\") pod \"1db59e38-c479-48cd-b641-48c6e44a0b23\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") "
Nov 03 17:33:50.655216 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655089    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-credential-socket\") pod \"1db59e38-c479-48cd-b641-48c6e44a0b23\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") "
Nov 03 17:33:50.655216 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655122    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-podinfo\") pod \"1db59e38-c479-48cd-b641-48c6e44a0b23\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") "
Nov 03 17:33:50.655216 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655147    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istiod-ca-cert\") pod \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") "
Nov 03 17:33:50.655216 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655204    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-podinfo\") pod \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") "
Nov 03 17:33:50.655542 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655235    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-credential-socket\") pod \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") "
Nov 03 17:33:50.655542 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655280    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-workload-certs\") pod \"1db59e38-c479-48cd-b641-48c6e44a0b23\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") "
Nov 03 17:33:50.655542 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655321    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-workload-socket\") pod \"1db59e38-c479-48cd-b641-48c6e44a0b23\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") "
Nov 03 17:33:50.655542 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655346    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-workload-certs\") pod \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") "
Nov 03 17:33:50.655542 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655341    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-credential-socket" pod "c097188b-4b64-4636-822b-ab2c8817dfa0" (UID: "c097188b-4b64-4636-822b-ab2c8817dfa0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:50.655542 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655395    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-workload-socket\") pod \"c097188b-4b64-4636-822b-ab2c8817dfa0\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") "
Nov 03 17:33:50.655542 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655429    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-data\") pod \"c097188b-4b64-4636-822b-ab2c8817dfa0\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") "
Nov 03 17:33:50.655542 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655476    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-envoy\") pod \"1db59e38-c479-48cd-b641-48c6e44a0b23\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") "
Nov 03 17:33:50.655542 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655486    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-credential-socket" pod "1db59e38-c479-48cd-b641-48c6e44a0b23" (UID: "1db59e38-c479-48cd-b641-48c6e44a0b23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:50.655542 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655512    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db59e38-c479-48cd-b641-48c6e44a0b23-kube-api-access-dmj8t\" (UniqueName: \"kubernetes.io/projected/1db59e38-c479-48cd-b641-48c6e44a0b23-kube-api-access-dmj8t\") pod \"1db59e38-c479-48cd-b641-48c6e44a0b23\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") "
Nov 03 17:33:50.656061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655554    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-workload-certs\") pod \"c097188b-4b64-4636-822b-ab2c8817dfa0\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") "
Nov 03 17:33:50.656061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655608    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-podinfo\") pod \"c097188b-4b64-4636-822b-ab2c8817dfa0\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") "
Nov 03 17:33:50.656061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655630    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db59e38-c479-48cd-b641-48c6e44a0b23-istiod-ca-cert" pod "1db59e38-c479-48cd-b641-48c6e44a0b23" (UID: "1db59e38-c479-48cd-b641-48c6e44a0b23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:33:50.656061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655665    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-kube-api-access-9z6jm\" (UniqueName: \"kubernetes.io/projected/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-kube-api-access-9z6jm\") pod \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") "
Nov 03 17:33:50.656061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655716    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istiod-ca-cert" pod "e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb" (UID: "e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:33:50.656061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655722    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-token\" (UniqueName: \"kubernetes.io/projected/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-token\") pod \"1db59e38-c479-48cd-b641-48c6e44a0b23\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") "
Nov 03 17:33:50.656061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655852    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-credential-socket" pod "e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb" (UID: "e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:50.656061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655863    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-data" pod "c097188b-4b64-4636-822b-ab2c8817dfa0" (UID: "c097188b-4b64-4636-822b-ab2c8817dfa0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:50.656061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655782    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-envoy\") pod \"c097188b-4b64-4636-822b-ab2c8817dfa0\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") "
Nov 03 17:33:50.656061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.655959    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-data\") pod \"1db59e38-c479-48cd-b641-48c6e44a0b23\" (UID: \"1db59e38-c479-48cd-b641-48c6e44a0b23\") "
Nov 03 17:33:50.656061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.656015    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c097188b-4b64-4636-822b-ab2c8817dfa0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c097188b-4b64-4636-822b-ab2c8817dfa0-istiod-ca-cert\") pod \"c097188b-4b64-4636-822b-ab2c8817dfa0\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") "
Nov 03 17:33:50.656061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.656043    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-envoy\") pod \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") "
Nov 03 17:33:50.656686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.656053    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-workload-certs" pod "1db59e38-c479-48cd-b641-48c6e44a0b23" (UID: "1db59e38-c479-48cd-b641-48c6e44a0b23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:50.656686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.656091    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-workload-certs" pod "c097188b-4b64-4636-822b-ab2c8817dfa0" (UID: "c097188b-4b64-4636-822b-ab2c8817dfa0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:50.656686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.656112    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c097188b-4b64-4636-822b-ab2c8817dfa0-kube-api-access-7r95l\" (UniqueName: \"kubernetes.io/projected/c097188b-4b64-4636-822b-ab2c8817dfa0-kube-api-access-7r95l\") pod \"c097188b-4b64-4636-822b-ab2c8817dfa0\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") "
Nov 03 17:33:50.656686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.656168    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-workload-socket\") pod \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") "
Nov 03 17:33:50.656686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.656223    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-token\" (UniqueName: \"kubernetes.io/projected/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-token\") pod \"c097188b-4b64-4636-822b-ab2c8817dfa0\" (UID: \"c097188b-4b64-4636-822b-ab2c8817dfa0\") "
Nov 03 17:33:50.656686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.656235    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-workload-socket" pod "1db59e38-c479-48cd-b641-48c6e44a0b23" (UID: "1db59e38-c479-48cd-b641-48c6e44a0b23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:50.656686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.656255    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-token\" (UniqueName: \"kubernetes.io/projected/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-token\") pod \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\" (UID: \"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb\") "
Nov 03 17:33:50.656686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.656432    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-workload-certs" pod "e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb" (UID: "e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:50.656686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.656509    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-data" pod "e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb" (UID: "e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:50.656686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.656613    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-workload-socket" pod "c097188b-4b64-4636-822b-ab2c8817dfa0" (UID: "c097188b-4b64-4636-822b-ab2c8817dfa0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:50.657159 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.657010    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.657159 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.657032    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.657159 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.657048    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.657159 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.657062    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.657159 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.657077    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.657159 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.657090    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1db59e38-c479-48cd-b641-48c6e44a0b23-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.657159 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.657106    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.657159 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.657120    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.657159 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.657135    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.657159 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.657149    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.657681 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.657163    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.657681 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.657176    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.657681 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.657190    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.657681 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.657205    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04181a54-5cd2-4dee-b102-5955e3e9b5a3-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.657681 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.657219    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.657681 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.657232    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.657681 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.657246    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.657681 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.657261    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.657681 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.657275    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04181a54-5cd2-4dee-b102-5955e3e9b5a3-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.657681 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.657291    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flsls\" (UniqueName: \"kubernetes.io/projected/04181a54-5cd2-4dee-b102-5955e3e9b5a3-kube-api-access-flsls\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.658750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.658198    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-podinfo" pod "1db59e38-c479-48cd-b641-48c6e44a0b23" (UID: "1db59e38-c479-48cd-b641-48c6e44a0b23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:33:50.658750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.658481    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-token" pod "1db59e38-c479-48cd-b641-48c6e44a0b23" (UID: "1db59e38-c479-48cd-b641-48c6e44a0b23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:33:50.658750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.658501    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-workload-socket" pod "e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb" (UID: "e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:50.659824 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.659797    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-token" pod "e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb" (UID: "e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:33:50.660227 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.659974    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-envoy" pod "e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb" (UID: "e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:50.661382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.661329    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-envoy" pod "1db59e38-c479-48cd-b641-48c6e44a0b23" (UID: "1db59e38-c479-48cd-b641-48c6e44a0b23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:50.661631 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.661389    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-data" pod "1db59e38-c479-48cd-b641-48c6e44a0b23" (UID: "1db59e38-c479-48cd-b641-48c6e44a0b23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:50.661631 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.661487    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-podinfo" pod "c097188b-4b64-4636-822b-ab2c8817dfa0" (UID: "c097188b-4b64-4636-822b-ab2c8817dfa0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:33:50.661786 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.661651    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c097188b-4b64-4636-822b-ab2c8817dfa0-istiod-ca-cert" pod "c097188b-4b64-4636-822b-ab2c8817dfa0" (UID: "c097188b-4b64-4636-822b-ab2c8817dfa0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:33:50.661786 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.661691    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db59e38-c479-48cd-b641-48c6e44a0b23-kube-api-access-dmj8t" pod "1db59e38-c479-48cd-b641-48c6e44a0b23" (UID: "1db59e38-c479-48cd-b641-48c6e44a0b23"). InnerVolumeSpecName "kube-api-access-dmj8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:33:50.661786 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.661760    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c097188b-4b64-4636-822b-ab2c8817dfa0-kube-api-access-7r95l" pod "c097188b-4b64-4636-822b-ab2c8817dfa0" (UID: "c097188b-4b64-4636-822b-ab2c8817dfa0"). InnerVolumeSpecName "kube-api-access-7r95l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:33:50.662158 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.662131    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-kube-api-access-9z6jm" pod "e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb" (UID: "e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb"). InnerVolumeSpecName "kube-api-access-9z6jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:33:50.662440 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.662419    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-token" pod "c097188b-4b64-4636-822b-ab2c8817dfa0" (UID: "c097188b-4b64-4636-822b-ab2c8817dfa0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:33:50.662440 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.662434    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-envoy" pod "c097188b-4b64-4636-822b-ab2c8817dfa0" (UID: "c097188b-4b64-4636-822b-ab2c8817dfa0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:33:50.662609 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.662593    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-podinfo" pod "e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb" (UID: "e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:33:50.758721 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.758648    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r95l\" (UniqueName: \"kubernetes.io/projected/c097188b-4b64-4636-822b-ab2c8817dfa0-kube-api-access-7r95l\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.758721 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.758676    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.758721 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.758686    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.758721 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.758696    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.758721 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.758704    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.758721 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.758714    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.758721 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.758723    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.759038 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.758732    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmj8t\" (UniqueName: \"kubernetes.io/projected/1db59e38-c479-48cd-b641-48c6e44a0b23-kube-api-access-dmj8t\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.759038 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.758741    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.759038 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.758751    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z6jm\" (UniqueName: \"kubernetes.io/projected/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-kube-api-access-9z6jm\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.759038 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.758760    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.759038 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.758769    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c097188b-4b64-4636-822b-ab2c8817dfa0-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.759038 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.758777    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1db59e38-c479-48cd-b641-48c6e44a0b23-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.759038 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.758785    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c097188b-4b64-4636-822b-ab2c8817dfa0-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:50.759038 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:50.758793    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:51.017140 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.017034    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:51.215176 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.215145    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:51.258496 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.258463    6059 generic.go:356] "Generic (PLEG): container finished" podID="e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb" containerID="5e1cf1c5fe254aedbb4f43169db791e3199427bc008159b5858a57683616f0b1" exitCode=0
Nov 03 17:33:51.258886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.258522    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"
Nov 03 17:33:51.258886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.258546    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn" event={"ID":"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb","Type":"ContainerDied","Data":"5e1cf1c5fe254aedbb4f43169db791e3199427bc008159b5858a57683616f0b1"}
Nov 03 17:33:51.258886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.258611    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn" event={"ID":"e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb","Type":"ContainerDied","Data":"0e77b9788d9234a9449759f34092ea749845b5741a752805e652071017dbaf8d"}
Nov 03 17:33:51.258886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.258650    6059 scope.go:122] "RemoveContainer" containerID="5e1cf1c5fe254aedbb4f43169db791e3199427bc008159b5858a57683616f0b1"
Nov 03 17:33:51.260958 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.260936    6059 generic.go:356] "Generic (PLEG): container finished" podID="c097188b-4b64-4636-822b-ab2c8817dfa0" containerID="58f02de98c6f04c8694d8c7f340a484d92974d0e41245245afbb5c44e810e122" exitCode=0
Nov 03 17:33:51.261072 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.260994    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"
Nov 03 17:33:51.261072 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.261021    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d" event={"ID":"c097188b-4b64-4636-822b-ab2c8817dfa0","Type":"ContainerDied","Data":"58f02de98c6f04c8694d8c7f340a484d92974d0e41245245afbb5c44e810e122"}
Nov 03 17:33:51.261242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.261080    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d" event={"ID":"c097188b-4b64-4636-822b-ab2c8817dfa0","Type":"ContainerDied","Data":"630a0dc32c2dfc1973fed8972d1f81cbb8b3b9e8aafcf30cfd8334f00a79938f"}
Nov 03 17:33:51.262921 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.262895    6059 generic.go:356] "Generic (PLEG): container finished" podID="1db59e38-c479-48cd-b641-48c6e44a0b23" containerID="d6217027333c556c73a797c90cdd7e0f1aa6801a16eab7c1dd7e17c853582af5" exitCode=0
Nov 03 17:33:51.263041 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.262932    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"
Nov 03 17:33:51.263041 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.262982    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9" event={"ID":"1db59e38-c479-48cd-b641-48c6e44a0b23","Type":"ContainerDied","Data":"d6217027333c556c73a797c90cdd7e0f1aa6801a16eab7c1dd7e17c853582af5"}
Nov 03 17:33:51.263041 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.263030    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9" event={"ID":"1db59e38-c479-48cd-b641-48c6e44a0b23","Type":"ContainerDied","Data":"a38a017659010860eea7940ff07d8f594738a730bef8f2ad8c48f6a1b1f844e8"}
Nov 03 17:33:51.264590 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.264572    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"
Nov 03 17:33:51.264708 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.264655    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw" event={"ID":"04181a54-5cd2-4dee-b102-5955e3e9b5a3","Type":"ContainerDied","Data":"343b2a071abb9bc4fb53c60c3d683865a2ed3ff77ef3c9727f3f78d0ff177dfc"}
Nov 03 17:33:51.270472 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.270378    6059 generic.go:356] "Generic (PLEG): container finished" podID="4e3b666b-5066-4524-a552-cbc829bd11b4" containerID="30f37b5f8372446715faa033514c9cd1b762f5a829e2adffc8f04425f129e5b5" exitCode=0
Nov 03 17:33:51.270569 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.270475    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a92b5d-bvl8d" event={"ID":"4e3b666b-5066-4524-a552-cbc829bd11b4","Type":"ContainerDied","Data":"30f37b5f8372446715faa033514c9cd1b762f5a829e2adffc8f04425f129e5b5"}
Nov 03 17:33:51.278340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.278234    6059 scope.go:122] "RemoveContainer" containerID="5e1cf1c5fe254aedbb4f43169db791e3199427bc008159b5858a57683616f0b1"
Nov 03 17:33:51.278573 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:33:51.278544    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e1cf1c5fe254aedbb4f43169db791e3199427bc008159b5858a57683616f0b1\": container with ID starting with 5e1cf1c5fe254aedbb4f43169db791e3199427bc008159b5858a57683616f0b1 not found: ID does not exist" containerID="5e1cf1c5fe254aedbb4f43169db791e3199427bc008159b5858a57683616f0b1"
Nov 03 17:33:51.278630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.278585    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e1cf1c5fe254aedbb4f43169db791e3199427bc008159b5858a57683616f0b1"} err="failed to get container status \"5e1cf1c5fe254aedbb4f43169db791e3199427bc008159b5858a57683616f0b1\": rpc error: code = NotFound desc = could not find container \"5e1cf1c5fe254aedbb4f43169db791e3199427bc008159b5858a57683616f0b1\": container with ID starting with 5e1cf1c5fe254aedbb4f43169db791e3199427bc008159b5858a57683616f0b1 not found: ID does not exist"
Nov 03 17:33:51.278630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.278609    6059 scope.go:122] "RemoveContainer" containerID="58f02de98c6f04c8694d8c7f340a484d92974d0e41245245afbb5c44e810e122"
Nov 03 17:33:51.288213 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.288187    6059 scope.go:122] "RemoveContainer" containerID="c982d01afb4ddc77ddff2d184fbae564c62dada763de0a7204ca4dc30b39c590"
Nov 03 17:33:51.297839 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.297823    6059 scope.go:122] "RemoveContainer" containerID="58f02de98c6f04c8694d8c7f340a484d92974d0e41245245afbb5c44e810e122"
Nov 03 17:33:51.298090 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:33:51.298073    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58f02de98c6f04c8694d8c7f340a484d92974d0e41245245afbb5c44e810e122\": container with ID starting with 58f02de98c6f04c8694d8c7f340a484d92974d0e41245245afbb5c44e810e122 not found: ID does not exist" containerID="58f02de98c6f04c8694d8c7f340a484d92974d0e41245245afbb5c44e810e122"
Nov 03 17:33:51.298136 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.298106    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58f02de98c6f04c8694d8c7f340a484d92974d0e41245245afbb5c44e810e122"} err="failed to get container status \"58f02de98c6f04c8694d8c7f340a484d92974d0e41245245afbb5c44e810e122\": rpc error: code = NotFound desc = could not find container \"58f02de98c6f04c8694d8c7f340a484d92974d0e41245245afbb5c44e810e122\": container with ID starting with 58f02de98c6f04c8694d8c7f340a484d92974d0e41245245afbb5c44e810e122 not found: ID does not exist"
Nov 03 17:33:51.298136 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.298122    6059 scope.go:122] "RemoveContainer" containerID="c982d01afb4ddc77ddff2d184fbae564c62dada763de0a7204ca4dc30b39c590"
Nov 03 17:33:51.298342 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:33:51.298327    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c982d01afb4ddc77ddff2d184fbae564c62dada763de0a7204ca4dc30b39c590\": container with ID starting with c982d01afb4ddc77ddff2d184fbae564c62dada763de0a7204ca4dc30b39c590 not found: ID does not exist" containerID="c982d01afb4ddc77ddff2d184fbae564c62dada763de0a7204ca4dc30b39c590"
Nov 03 17:33:51.298403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.298345    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c982d01afb4ddc77ddff2d184fbae564c62dada763de0a7204ca4dc30b39c590"} err="failed to get container status \"c982d01afb4ddc77ddff2d184fbae564c62dada763de0a7204ca4dc30b39c590\": rpc error: code = NotFound desc = could not find container \"c982d01afb4ddc77ddff2d184fbae564c62dada763de0a7204ca4dc30b39c590\": container with ID starting with c982d01afb4ddc77ddff2d184fbae564c62dada763de0a7204ca4dc30b39c590 not found: ID does not exist"
Nov 03 17:33:51.298403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.298373    6059 scope.go:122] "RemoveContainer" containerID="d6217027333c556c73a797c90cdd7e0f1aa6801a16eab7c1dd7e17c853582af5"
Nov 03 17:33:51.307305 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.307290    6059 scope.go:122] "RemoveContainer" containerID="d6217027333c556c73a797c90cdd7e0f1aa6801a16eab7c1dd7e17c853582af5"
Nov 03 17:33:51.307614 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:33:51.307583    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6217027333c556c73a797c90cdd7e0f1aa6801a16eab7c1dd7e17c853582af5\": container with ID starting with d6217027333c556c73a797c90cdd7e0f1aa6801a16eab7c1dd7e17c853582af5 not found: ID does not exist" containerID="d6217027333c556c73a797c90cdd7e0f1aa6801a16eab7c1dd7e17c853582af5"
Nov 03 17:33:51.307703 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.307625    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6217027333c556c73a797c90cdd7e0f1aa6801a16eab7c1dd7e17c853582af5"} err="failed to get container status \"d6217027333c556c73a797c90cdd7e0f1aa6801a16eab7c1dd7e17c853582af5\": rpc error: code = NotFound desc = could not find container \"d6217027333c556c73a797c90cdd7e0f1aa6801a16eab7c1dd7e17c853582af5\": container with ID starting with d6217027333c556c73a797c90cdd7e0f1aa6801a16eab7c1dd7e17c853582af5 not found: ID does not exist"
Nov 03 17:33:51.307703 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.307645    6059 scope.go:122] "RemoveContainer" containerID="2a26b9b5de878006e2c7dbcec5a94d728792d35258240e7a9d3ff4c5c076287d"
Nov 03 17:33:51.597178 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.597102    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"]
Nov 03 17:33:51.600075 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.600045    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wgvjn"]
Nov 03 17:33:51.712170 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.712140    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"]
Nov 03 17:33:51.726802 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.726772    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-s48fw"]
Nov 03 17:33:51.841868 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.841836    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"]
Nov 03 17:33:51.846860 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.846836    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-x276d"]
Nov 03 17:33:51.963901 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.963868    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"]
Nov 03 17:33:51.966806 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:51.966785    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-679z9"]
Nov 03 17:33:52.422422 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:52.422397    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a92b5d-bvl8d"
Nov 03 17:33:52.479445 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:52.479419    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e3b666b-5066-4524-a552-cbc829bd11b4-kube-api-access-2l24w\" (UniqueName: \"kubernetes.io/projected/4e3b666b-5066-4524-a552-cbc829bd11b4-kube-api-access-2l24w\") pod \"4e3b666b-5066-4524-a552-cbc829bd11b4\" (UID: \"4e3b666b-5066-4524-a552-cbc829bd11b4\") "
Nov 03 17:33:52.479617 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:52.479502    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e3b666b-5066-4524-a552-cbc829bd11b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e3b666b-5066-4524-a552-cbc829bd11b4-maas-api-openshift-ca\") pod \"4e3b666b-5066-4524-a552-cbc829bd11b4\" (UID: \"4e3b666b-5066-4524-a552-cbc829bd11b4\") "
Nov 03 17:33:52.479617 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:52.479585    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e3b666b-5066-4524-a552-cbc829bd11b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e3b666b-5066-4524-a552-cbc829bd11b4-maas-api-xks-ca\") pod \"4e3b666b-5066-4524-a552-cbc829bd11b4\" (UID: \"4e3b666b-5066-4524-a552-cbc829bd11b4\") "
Nov 03 17:33:52.479854 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:52.479827    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e3b666b-5066-4524-a552-cbc829bd11b4-maas-api-openshift-ca" pod "4e3b666b-5066-4524-a552-cbc829bd11b4" (UID: "4e3b666b-5066-4524-a552-cbc829bd11b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:33:52.481574 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:52.481551    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3b666b-5066-4524-a552-cbc829bd11b4-kube-api-access-2l24w" pod "4e3b666b-5066-4524-a552-cbc829bd11b4" (UID: "4e3b666b-5066-4524-a552-cbc829bd11b4"). InnerVolumeSpecName "kube-api-access-2l24w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:33:52.481642 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:52.481617    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e3b666b-5066-4524-a552-cbc829bd11b4-maas-api-xks-ca" pod "4e3b666b-5066-4524-a552-cbc829bd11b4" (UID: "4e3b666b-5066-4524-a552-cbc829bd11b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:33:52.581040 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:52.580967    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e3b666b-5066-4524-a552-cbc829bd11b4-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:52.581040 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:52.580993    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l24w\" (UniqueName: \"kubernetes.io/projected/4e3b666b-5066-4524-a552-cbc829bd11b4-kube-api-access-2l24w\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:52.581040 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:52.581005    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e3b666b-5066-4524-a552-cbc829bd11b4-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:33:53.288628 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:53.288601    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a92b5d-bvl8d"
Nov 03 17:33:53.288815 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:53.288598    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a92b5d-bvl8d" event={"ID":"4e3b666b-5066-4524-a552-cbc829bd11b4","Type":"ContainerDied","Data":"2ba561e4576c61b6477bb6a6daf15086109df91767ae7f06883c0ae5b3acff2c"}
Nov 03 17:33:53.288815 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:53.288732    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ba561e4576c61b6477bb6a6daf15086109df91767ae7f06883c0ae5b3acff2c"
Nov 03 17:33:53.335605 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:53.335568    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04181a54-5cd2-4dee-b102-5955e3e9b5a3" path="/var/lib/kubelet/pods/04181a54-5cd2-4dee-b102-5955e3e9b5a3/volumes"
Nov 03 17:33:53.336062 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:53.336049    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1db59e38-c479-48cd-b641-48c6e44a0b23" path="/var/lib/kubelet/pods/1db59e38-c479-48cd-b641-48c6e44a0b23/volumes"
Nov 03 17:33:53.336500 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:53.336487    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c097188b-4b64-4636-822b-ab2c8817dfa0" path="/var/lib/kubelet/pods/c097188b-4b64-4636-822b-ab2c8817dfa0/volumes"
Nov 03 17:33:53.337213 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:53.337198    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb" path="/var/lib/kubelet/pods/e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb/volumes"
Nov 03 17:33:55.852382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:55.850262    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:33:59.612788 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.612754    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf"]
Nov 03 17:33:59.613266 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.613252    6059 cpu_manager.go:414] "containerMap: removing container" podUID="c097188b-4b64-4636-822b-ab2c8817dfa0" containerName="istio-proxy"
Nov 03 17:33:59.613306 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.613266    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c097188b-4b64-4636-822b-ab2c8817dfa0" containerName="istio-proxy"
Nov 03 17:33:59.613306 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.613277    6059 cpu_manager.go:414] "containerMap: removing container" podUID="04181a54-5cd2-4dee-b102-5955e3e9b5a3" containerName="istio-proxy"
Nov 03 17:33:59.613306 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.613282    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04181a54-5cd2-4dee-b102-5955e3e9b5a3" containerName="istio-proxy"
Nov 03 17:33:59.613306 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.613290    6059 cpu_manager.go:414] "containerMap: removing container" podUID="e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb" containerName="istio-proxy"
Nov 03 17:33:59.613306 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.613295    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb" containerName="istio-proxy"
Nov 03 17:33:59.613502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.613310    6059 cpu_manager.go:414] "containerMap: removing container" podUID="c097188b-4b64-4636-822b-ab2c8817dfa0" containerName="istio-proxy"
Nov 03 17:33:59.613502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.613315    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c097188b-4b64-4636-822b-ab2c8817dfa0" containerName="istio-proxy"
Nov 03 17:33:59.613502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.613343    6059 cpu_manager.go:414] "containerMap: removing container" podUID="4e3b666b-5066-4524-a552-cbc829bd11b4" containerName="revoke-keys"
Nov 03 17:33:59.613502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.613349    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e3b666b-5066-4524-a552-cbc829bd11b4" containerName="revoke-keys"
Nov 03 17:33:59.613502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.613374    6059 cpu_manager.go:414] "containerMap: removing container" podUID="1db59e38-c479-48cd-b641-48c6e44a0b23" containerName="istio-proxy"
Nov 03 17:33:59.613502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.613381    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db59e38-c479-48cd-b641-48c6e44a0b23" containerName="istio-proxy"
Nov 03 17:33:59.613502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.613480    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c097188b-4b64-4636-822b-ab2c8817dfa0" containerName="istio-proxy"
Nov 03 17:33:59.613502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.613489    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="04181a54-5cd2-4dee-b102-5955e3e9b5a3" containerName="istio-proxy"
Nov 03 17:33:59.613502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.613495    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e42e8a1e-2f87-4d9d-bbc4-c76d84d84dbb" containerName="istio-proxy"
Nov 03 17:33:59.613502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.613502    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c097188b-4b64-4636-822b-ab2c8817dfa0" containerName="istio-proxy"
Nov 03 17:33:59.613806 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.613514    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e3b666b-5066-4524-a552-cbc829bd11b4" containerName="revoke-keys"
Nov 03 17:33:59.613806 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.613521    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db59e38-c479-48cd-b641-48c6e44a0b23" containerName="istio-proxy"
Nov 03 17:33:59.618348 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.618325    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf"
Nov 03 17:33:59.679575 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.679539    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf"]
Nov 03 17:33:59.726644 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.726607    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:33:59.726644 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.726636    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:33:59.726644 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.726641    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:33:59.726872 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.726644    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:33:59.765639 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.765613    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/031c7311-179c-4fdc-9911-fd6c0743ce8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce8f3dc8-l7pzf\" (UID: \"031c7311-179c-4fdc-9911-fd6c0743ce8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf"
Nov 03 17:33:59.765806 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.765659    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86ncn\" (UniqueName: \"kubernetes.io/projected/031c7311-179c-4fdc-9911-fd6c0743ce8a-kube-api-access-86ncn\") pod \"maas-api-revoke-sub-ce8f3dc8-l7pzf\" (UID: \"031c7311-179c-4fdc-9911-fd6c0743ce8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf"
Nov 03 17:33:59.765806 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.765684    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/031c7311-179c-4fdc-9911-fd6c0743ce8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce8f3dc8-l7pzf\" (UID: \"031c7311-179c-4fdc-9911-fd6c0743ce8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf"
Nov 03 17:33:59.866774 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.866691    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/031c7311-179c-4fdc-9911-fd6c0743ce8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce8f3dc8-l7pzf\" (UID: \"031c7311-179c-4fdc-9911-fd6c0743ce8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf"
Nov 03 17:33:59.866774 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.866746    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86ncn\" (UniqueName: \"kubernetes.io/projected/031c7311-179c-4fdc-9911-fd6c0743ce8a-kube-api-access-86ncn\") pod \"maas-api-revoke-sub-ce8f3dc8-l7pzf\" (UID: \"031c7311-179c-4fdc-9911-fd6c0743ce8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf"
Nov 03 17:33:59.867019 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.866793    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/031c7311-179c-4fdc-9911-fd6c0743ce8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce8f3dc8-l7pzf\" (UID: \"031c7311-179c-4fdc-9911-fd6c0743ce8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf"
Nov 03 17:33:59.867411 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.867348    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/031c7311-179c-4fdc-9911-fd6c0743ce8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce8f3dc8-l7pzf\" (UID: \"031c7311-179c-4fdc-9911-fd6c0743ce8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf"
Nov 03 17:33:59.869224 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.869202    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/031c7311-179c-4fdc-9911-fd6c0743ce8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce8f3dc8-l7pzf\" (UID: \"031c7311-179c-4fdc-9911-fd6c0743ce8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf"
Nov 03 17:33:59.946940 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:33:59.946905    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86ncn\" (UniqueName: \"kubernetes.io/projected/031c7311-179c-4fdc-9911-fd6c0743ce8a-kube-api-access-86ncn\") pod \"maas-api-revoke-sub-ce8f3dc8-l7pzf\" (UID: \"031c7311-179c-4fdc-9911-fd6c0743ce8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf"
Nov 03 17:34:00.231702 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:00.231673    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf"
Nov 03 17:34:00.552815 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:00.552719    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:00.614703 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:34:00.614669    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031c7311_179c_4fdc_9911_fd6c0743ce8a.slice/crio-73eb3e072da659d902ab1ecc4c628f74b6ff292e53088893578b719e5e81b87d WatchSource:0}: Error finding container 73eb3e072da659d902ab1ecc4c628f74b6ff292e53088893578b719e5e81b87d: Status 404 returned error can't find the container with id 73eb3e072da659d902ab1ecc4c628f74b6ff292e53088893578b719e5e81b87d
Nov 03 17:34:00.617070 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:00.617038    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf"]
Nov 03 17:34:01.357175 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:01.357138    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf" event={"ID":"031c7311-179c-4fdc-9911-fd6c0743ce8a","Type":"ContainerStarted","Data":"fe4786836e46cf4a9ecad018848f5e360a14dd7bdfdcff44fe9ed1f8d2a56f7b"}
Nov 03 17:34:01.357320 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:01.357196    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf" event={"ID":"031c7311-179c-4fdc-9911-fd6c0743ce8a","Type":"ContainerStarted","Data":"73eb3e072da659d902ab1ecc4c628f74b6ff292e53088893578b719e5e81b87d"}
Nov 03 17:34:01.481336 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:01.481273    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf" podStartSLOduration=2.481256239 podStartE2EDuration="2.481256239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:01.47861113 +0000 UTC m=+982.276323912" watchObservedRunningTime="2026-11-03 17:34:01.481256239 +0000 UTC m=+982.278969021"
Nov 03 17:34:02.365733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:02.365699    6059 generic.go:356] "Generic (PLEG): container finished" podID="031c7311-179c-4fdc-9911-fd6c0743ce8a" containerID="fe4786836e46cf4a9ecad018848f5e360a14dd7bdfdcff44fe9ed1f8d2a56f7b" exitCode=0
Nov 03 17:34:02.366119 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:02.365783    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf" event={"ID":"031c7311-179c-4fdc-9911-fd6c0743ce8a","Type":"ContainerDied","Data":"fe4786836e46cf4a9ecad018848f5e360a14dd7bdfdcff44fe9ed1f8d2a56f7b"}
Nov 03 17:34:03.523012 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:03.522987    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf"
Nov 03 17:34:03.611866 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:03.611828    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/031c7311-179c-4fdc-9911-fd6c0743ce8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/031c7311-179c-4fdc-9911-fd6c0743ce8a-maas-api-xks-ca\") pod \"031c7311-179c-4fdc-9911-fd6c0743ce8a\" (UID: \"031c7311-179c-4fdc-9911-fd6c0743ce8a\") "
Nov 03 17:34:03.612014 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:03.611899    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/031c7311-179c-4fdc-9911-fd6c0743ce8a-kube-api-access-86ncn\" (UniqueName: \"kubernetes.io/projected/031c7311-179c-4fdc-9911-fd6c0743ce8a-kube-api-access-86ncn\") pod \"031c7311-179c-4fdc-9911-fd6c0743ce8a\" (UID: \"031c7311-179c-4fdc-9911-fd6c0743ce8a\") "
Nov 03 17:34:03.612014 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:03.611952    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/031c7311-179c-4fdc-9911-fd6c0743ce8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/031c7311-179c-4fdc-9911-fd6c0743ce8a-maas-api-openshift-ca\") pod \"031c7311-179c-4fdc-9911-fd6c0743ce8a\" (UID: \"031c7311-179c-4fdc-9911-fd6c0743ce8a\") "
Nov 03 17:34:03.612309 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:03.612277    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/031c7311-179c-4fdc-9911-fd6c0743ce8a-maas-api-openshift-ca" pod "031c7311-179c-4fdc-9911-fd6c0743ce8a" (UID: "031c7311-179c-4fdc-9911-fd6c0743ce8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:34:03.612465 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:03.612448    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/031c7311-179c-4fdc-9911-fd6c0743ce8a-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:34:03.614051 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:03.614029    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/031c7311-179c-4fdc-9911-fd6c0743ce8a-maas-api-xks-ca" pod "031c7311-179c-4fdc-9911-fd6c0743ce8a" (UID: "031c7311-179c-4fdc-9911-fd6c0743ce8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:34:03.614109 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:03.614090    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031c7311-179c-4fdc-9911-fd6c0743ce8a-kube-api-access-86ncn" pod "031c7311-179c-4fdc-9911-fd6c0743ce8a" (UID: "031c7311-179c-4fdc-9911-fd6c0743ce8a"). InnerVolumeSpecName "kube-api-access-86ncn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:34:03.713889 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:03.713852    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/031c7311-179c-4fdc-9911-fd6c0743ce8a-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:34:03.713889 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:03.713887    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86ncn\" (UniqueName: \"kubernetes.io/projected/031c7311-179c-4fdc-9911-fd6c0743ce8a-kube-api-access-86ncn\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:34:04.379608 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:04.379567    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf" event={"ID":"031c7311-179c-4fdc-9911-fd6c0743ce8a","Type":"ContainerDied","Data":"73eb3e072da659d902ab1ecc4c628f74b6ff292e53088893578b719e5e81b87d"}
Nov 03 17:34:04.379608 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:04.379603    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf"
Nov 03 17:34:04.379811 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:04.379627    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73eb3e072da659d902ab1ecc4c628f74b6ff292e53088893578b719e5e81b87d"
Nov 03 17:34:05.120653 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:05.120614    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:06.998383 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:34:06.991631    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:34:09.958145 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:09.957922    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:14.523455 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:14.523422    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:15.510065 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:15.510035    6059 generic.go:356] "Generic (PLEG): container finished" podID="3b813b37-7f95-4138-a42d-bb5af19d7e09" containerID="9a2e2e2e41accf8a19e9da57a36975a41c081d143904704ca8e494637fd408f0" exitCode=0
Nov 03 17:34:15.510281 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:15.510113    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk" event={"ID":"3b813b37-7f95-4138-a42d-bb5af19d7e09","Type":"ContainerDied","Data":"9a2e2e2e41accf8a19e9da57a36975a41c081d143904704ca8e494637fd408f0"}
Nov 03 17:34:15.510517 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:15.510503    6059 scope.go:122] "RemoveContainer" containerID="9a2e2e2e41accf8a19e9da57a36975a41c081d143904704ca8e494637fd408f0"
Nov 03 17:34:15.511895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:15.511870    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:34:15.511974 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:15.511934    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:34:15.511974 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:15.511966    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:34:15.982394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:15.982337    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd"]
Nov 03 17:34:15.983242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:15.983218    6059 cpu_manager.go:414] "containerMap: removing container" podUID="031c7311-179c-4fdc-9911-fd6c0743ce8a" containerName="revoke-keys"
Nov 03 17:34:15.983408 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:15.983243    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="031c7311-179c-4fdc-9911-fd6c0743ce8a" containerName="revoke-keys"
Nov 03 17:34:15.983543 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:15.983521    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="031c7311-179c-4fdc-9911-fd6c0743ce8a" containerName="revoke-keys"
Nov 03 17:34:15.987755 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:15.987731    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd"
Nov 03 17:34:16.053488 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.053454    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd"]
Nov 03 17:34:16.056255 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.055299    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:34:16.060615 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.056044    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:34:16.060615 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.056098    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:34:16.060615 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.056157    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:34:16.155106 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.155066    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdch4\" (UniqueName: \"kubernetes.io/projected/018a023a-a6d2-4189-9d3f-4c9820addd27-kube-api-access-bdch4\") pod \"maas-api-revoke-sub-422063f1-f4ctd\" (UID: \"018a023a-a6d2-4189-9d3f-4c9820addd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd"
Nov 03 17:34:16.155315 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.155211    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/018a023a-a6d2-4189-9d3f-4c9820addd27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-422063f1-f4ctd\" (UID: \"018a023a-a6d2-4189-9d3f-4c9820addd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd"
Nov 03 17:34:16.155315 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.155282    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/018a023a-a6d2-4189-9d3f-4c9820addd27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-422063f1-f4ctd\" (UID: \"018a023a-a6d2-4189-9d3f-4c9820addd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd"
Nov 03 17:34:16.256256 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.256162    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdch4\" (UniqueName: \"kubernetes.io/projected/018a023a-a6d2-4189-9d3f-4c9820addd27-kube-api-access-bdch4\") pod \"maas-api-revoke-sub-422063f1-f4ctd\" (UID: \"018a023a-a6d2-4189-9d3f-4c9820addd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd"
Nov 03 17:34:16.256256 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.256230    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/018a023a-a6d2-4189-9d3f-4c9820addd27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-422063f1-f4ctd\" (UID: \"018a023a-a6d2-4189-9d3f-4c9820addd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd"
Nov 03 17:34:16.256519 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.256272    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/018a023a-a6d2-4189-9d3f-4c9820addd27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-422063f1-f4ctd\" (UID: \"018a023a-a6d2-4189-9d3f-4c9820addd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd"
Nov 03 17:34:16.256947 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.256927    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/018a023a-a6d2-4189-9d3f-4c9820addd27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-422063f1-f4ctd\" (UID: \"018a023a-a6d2-4189-9d3f-4c9820addd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd"
Nov 03 17:34:16.258682 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.258663    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/018a023a-a6d2-4189-9d3f-4c9820addd27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-422063f1-f4ctd\" (UID: \"018a023a-a6d2-4189-9d3f-4c9820addd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd"
Nov 03 17:34:16.322236 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.322208    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdch4\" (UniqueName: \"kubernetes.io/projected/018a023a-a6d2-4189-9d3f-4c9820addd27-kube-api-access-bdch4\") pod \"maas-api-revoke-sub-422063f1-f4ctd\" (UID: \"018a023a-a6d2-4189-9d3f-4c9820addd27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd"
Nov 03 17:34:16.519235 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.519149    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk" event={"ID":"3b813b37-7f95-4138-a42d-bb5af19d7e09","Type":"ContainerStarted","Data":"b08716f414dd05f8458ae06a0a338f6cb3d95b906d01c0391411ef870bded263"}
Nov 03 17:34:16.602168 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.602139    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd"
Nov 03 17:34:16.705476 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.705451    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:34:16.776749 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.776670    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7484-03ebc6"]
Nov 03 17:34:16.781849 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.781828    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-03ebc6"
Nov 03 17:34:16.860281 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.860228    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7484-03ebc6"]
Nov 03 17:34:16.860597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.860573    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hjccf\""
Nov 03 17:34:16.863619 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.863586    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnr9j\" (UniqueName: \"kubernetes.io/projected/3bad05ec-aa3f-4101-adb8-e690e06d7c67-kube-api-access-dnr9j\") pod \"test-curl-7484-03ebc6\" (UID: \"3bad05ec-aa3f-4101-adb8-e690e06d7c67\") " pod="opendatahub/test-curl-7484-03ebc6"
Nov 03 17:34:16.964386 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:16.964323    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnr9j\" (UniqueName: \"kubernetes.io/projected/3bad05ec-aa3f-4101-adb8-e690e06d7c67-kube-api-access-dnr9j\") pod \"test-curl-7484-03ebc6\" (UID: \"3bad05ec-aa3f-4101-adb8-e690e06d7c67\") " pod="opendatahub/test-curl-7484-03ebc6"
Nov 03 17:34:17.061199 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:17.061167    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd"]
Nov 03 17:34:17.062057 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:34:17.062028    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018a023a_a6d2_4189_9d3f_4c9820addd27.slice/crio-2a5326d50ad4f41320e060353ea22f0393043fef470e9675edd49ccd36e00a95 WatchSource:0}: Error finding container 2a5326d50ad4f41320e060353ea22f0393043fef470e9675edd49ccd36e00a95: Status 404 returned error can't find the container with id 2a5326d50ad4f41320e060353ea22f0393043fef470e9675edd49ccd36e00a95
Nov 03 17:34:17.064003 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:17.063981    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnr9j\" (UniqueName: \"kubernetes.io/projected/3bad05ec-aa3f-4101-adb8-e690e06d7c67-kube-api-access-dnr9j\") pod \"test-curl-7484-03ebc6\" (UID: \"3bad05ec-aa3f-4101-adb8-e690e06d7c67\") " pod="opendatahub/test-curl-7484-03ebc6"
Nov 03 17:34:17.096013 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:17.095988    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-03ebc6"
Nov 03 17:34:17.121551 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:17.121518    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:34:17.123113 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:17.123087    6059 patch_prober.go:29] interesting pod/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 17:34:17.123247 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:17.123133    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk" podUID="3b813b37-7f95-4138-a42d-bb5af19d7e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 17:34:17.500800 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:34:17.500771    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bad05ec_aa3f_4101_adb8_e690e06d7c67.slice/crio-790451a6e969b8861ec73ce6fc645417a8388c241ce7807f50e7bd177c07412d WatchSource:0}: Error finding container 790451a6e969b8861ec73ce6fc645417a8388c241ce7807f50e7bd177c07412d: Status 404 returned error can't find the container with id 790451a6e969b8861ec73ce6fc645417a8388c241ce7807f50e7bd177c07412d
Nov 03 17:34:17.502845 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:17.502828    6059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:34:17.503980 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:17.503941    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7484-03ebc6"]
Nov 03 17:34:17.527194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:17.527162    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd" event={"ID":"018a023a-a6d2-4189-9d3f-4c9820addd27","Type":"ContainerStarted","Data":"375ffdac5ca9cafdcf30c62c70f6a99878ae4ecb728adb279462b3edd7703be5"}
Nov 03 17:34:17.527302 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:17.527249    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd" event={"ID":"018a023a-a6d2-4189-9d3f-4c9820addd27","Type":"ContainerStarted","Data":"2a5326d50ad4f41320e060353ea22f0393043fef470e9675edd49ccd36e00a95"}
Nov 03 17:34:17.528524 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:17.528498    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7484-03ebc6" event={"ID":"3bad05ec-aa3f-4101-adb8-e690e06d7c67","Type":"ContainerStarted","Data":"790451a6e969b8861ec73ce6fc645417a8388c241ce7807f50e7bd177c07412d"}
Nov 03 17:34:17.661108 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:17.661038    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd" podStartSLOduration=2.661019959 podStartE2EDuration="2.661019959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:17.656777234 +0000 UTC m=+998.454490018" watchObservedRunningTime="2026-11-03 17:34:17.661019959 +0000 UTC m=+998.458732743"
Nov 03 17:34:18.121659 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:18.121627    6059 patch_prober.go:29] interesting pod/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 17:34:18.122059 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:18.121683    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk" podUID="3b813b37-7f95-4138-a42d-bb5af19d7e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 17:34:18.558974 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:18.558940    6059 generic.go:356] "Generic (PLEG): container finished" podID="018a023a-a6d2-4189-9d3f-4c9820addd27" containerID="375ffdac5ca9cafdcf30c62c70f6a99878ae4ecb728adb279462b3edd7703be5" exitCode=0
Nov 03 17:34:18.559999 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:18.559972    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd" event={"ID":"018a023a-a6d2-4189-9d3f-4c9820addd27","Type":"ContainerDied","Data":"375ffdac5ca9cafdcf30c62c70f6a99878ae4ecb728adb279462b3edd7703be5"}
Nov 03 17:34:19.048941 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:19.048877    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:19.122478 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:19.122436    6059 patch_prober.go:29] interesting pod/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 17:34:19.122861 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:19.122521    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk" podUID="3b813b37-7f95-4138-a42d-bb5af19d7e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 17:34:19.762316 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:19.762287    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd"
Nov 03 17:34:19.903443 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:19.903365    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/018a023a-a6d2-4189-9d3f-4c9820addd27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/018a023a-a6d2-4189-9d3f-4c9820addd27-maas-api-openshift-ca\") pod \"018a023a-a6d2-4189-9d3f-4c9820addd27\" (UID: \"018a023a-a6d2-4189-9d3f-4c9820addd27\") "
Nov 03 17:34:19.903620 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:19.903493    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/018a023a-a6d2-4189-9d3f-4c9820addd27-kube-api-access-bdch4\" (UniqueName: \"kubernetes.io/projected/018a023a-a6d2-4189-9d3f-4c9820addd27-kube-api-access-bdch4\") pod \"018a023a-a6d2-4189-9d3f-4c9820addd27\" (UID: \"018a023a-a6d2-4189-9d3f-4c9820addd27\") "
Nov 03 17:34:19.903620 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:19.903579    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/018a023a-a6d2-4189-9d3f-4c9820addd27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/018a023a-a6d2-4189-9d3f-4c9820addd27-maas-api-xks-ca\") pod \"018a023a-a6d2-4189-9d3f-4c9820addd27\" (UID: \"018a023a-a6d2-4189-9d3f-4c9820addd27\") "
Nov 03 17:34:19.904470 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:19.904430    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/018a023a-a6d2-4189-9d3f-4c9820addd27-maas-api-openshift-ca" pod "018a023a-a6d2-4189-9d3f-4c9820addd27" (UID: "018a023a-a6d2-4189-9d3f-4c9820addd27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:34:19.906132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:19.906109    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/018a023a-a6d2-4189-9d3f-4c9820addd27-maas-api-xks-ca" pod "018a023a-a6d2-4189-9d3f-4c9820addd27" (UID: "018a023a-a6d2-4189-9d3f-4c9820addd27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:34:19.907122 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:19.907090    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/018a023a-a6d2-4189-9d3f-4c9820addd27-kube-api-access-bdch4" pod "018a023a-a6d2-4189-9d3f-4c9820addd27" (UID: "018a023a-a6d2-4189-9d3f-4c9820addd27"). InnerVolumeSpecName "kube-api-access-bdch4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:34:20.005153 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:20.005118    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdch4\" (UniqueName: \"kubernetes.io/projected/018a023a-a6d2-4189-9d3f-4c9820addd27-kube-api-access-bdch4\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:34:20.005153 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:20.005148    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/018a023a-a6d2-4189-9d3f-4c9820addd27-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:34:20.005153 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:20.005158    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/018a023a-a6d2-4189-9d3f-4c9820addd27-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:34:20.123384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:20.122128    6059 patch_prober.go:29] interesting pod/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 17:34:20.123384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:20.122190    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk" podUID="3b813b37-7f95-4138-a42d-bb5af19d7e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 17:34:20.628004 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:20.627950    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd" event={"ID":"018a023a-a6d2-4189-9d3f-4c9820addd27","Type":"ContainerDied","Data":"2a5326d50ad4f41320e060353ea22f0393043fef470e9675edd49ccd36e00a95"}
Nov 03 17:34:20.628210 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:20.628022    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a5326d50ad4f41320e060353ea22f0393043fef470e9675edd49ccd36e00a95"
Nov 03 17:34:20.628210 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:20.628096    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd"
Nov 03 17:34:21.121669 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:21.121636    6059 patch_prober.go:29] interesting pod/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 17:34:21.121866 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:21.121691    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk" podUID="3b813b37-7f95-4138-a42d-bb5af19d7e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 17:34:22.127101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:22.127066    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:34:22.128472 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:22.128452    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 17:34:23.657506 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:23.657469    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7484-03ebc6" event={"ID":"3bad05ec-aa3f-4101-adb8-e690e06d7c67","Type":"ContainerStarted","Data":"9d6db13ed5d97ef398d302b61c72bf54357be3d27488c20a248dd731e0ed0f13"}
Nov 03 17:34:23.723702 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:23.723668    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:23.779781 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:23.779679    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7484-03ebc6" podStartSLOduration=2.616362219 podStartE2EDuration="7.779651933s" totalImagesPullingTime="5.163289714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:23.775787516 +0000 UTC m=+1004.573500297" watchObservedRunningTime="2026-11-03 17:34:23.779651933 +0000 UTC m=+1004.577364706"
Nov 03 17:34:25.261380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.261313    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7484-03ebc6"]
Nov 03 17:34:25.265076 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.265048    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7484-03ebc6"]
Nov 03 17:34:25.556007 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.555928    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll"]
Nov 03 17:34:25.556512 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.556496    6059 cpu_manager.go:414] "containerMap: removing container" podUID="018a023a-a6d2-4189-9d3f-4c9820addd27" containerName="revoke-keys"
Nov 03 17:34:25.556577 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.556513    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="018a023a-a6d2-4189-9d3f-4c9820addd27" containerName="revoke-keys"
Nov 03 17:34:25.556577 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.556536    6059 cpu_manager.go:414] "containerMap: removing container" podUID="3bad05ec-aa3f-4101-adb8-e690e06d7c67" containerName="test-curl-7484-03ebc6"
Nov 03 17:34:25.556577 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.556542    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bad05ec-aa3f-4101-adb8-e690e06d7c67" containerName="test-curl-7484-03ebc6"
Nov 03 17:34:25.556668 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.556625    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="018a023a-a6d2-4189-9d3f-4c9820addd27" containerName="revoke-keys"
Nov 03 17:34:25.556668 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.556638    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bad05ec-aa3f-4101-adb8-e690e06d7c67" containerName="test-curl-7484-03ebc6"
Nov 03 17:34:25.560255 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.560238    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll"
Nov 03 17:34:25.618530 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.618501    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:34:25.618530 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.618523    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:34:25.620753 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.620728    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll"]
Nov 03 17:34:25.623037 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.623016    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cc27644-ghpll\" (UID: \"4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll"
Nov 03 17:34:25.623149 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.623133    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng64k\" (UniqueName: \"kubernetes.io/projected/4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d-kube-api-access-ng64k\") pod \"maas-api-revoke-sub-6cc27644-ghpll\" (UID: \"4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll"
Nov 03 17:34:25.623203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.623186    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cc27644-ghpll\" (UID: \"4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll"
Nov 03 17:34:25.668517 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.668485    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:34:25.668517 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.668485    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:34:25.676841 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.676806    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7484-03ebc6" podUID="3bad05ec-aa3f-4101-adb8-e690e06d7c67" containerName="test-curl-7484-03ebc6" containerID="cri-o://9d6db13ed5d97ef398d302b61c72bf54357be3d27488c20a248dd731e0ed0f13" gracePeriod=2
Nov 03 17:34:25.724746 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.724712    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cc27644-ghpll\" (UID: \"4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll"
Nov 03 17:34:25.724900 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.724807    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng64k\" (UniqueName: \"kubernetes.io/projected/4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d-kube-api-access-ng64k\") pod \"maas-api-revoke-sub-6cc27644-ghpll\" (UID: \"4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll"
Nov 03 17:34:25.724900 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.724861    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cc27644-ghpll\" (UID: \"4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll"
Nov 03 17:34:25.725344 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.725324    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cc27644-ghpll\" (UID: \"4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll"
Nov 03 17:34:25.727197 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.727174    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cc27644-ghpll\" (UID: \"4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll"
Nov 03 17:34:25.788064 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.788035    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng64k\" (UniqueName: \"kubernetes.io/projected/4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d-kube-api-access-ng64k\") pod \"maas-api-revoke-sub-6cc27644-ghpll\" (UID: \"4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll"
Nov 03 17:34:25.873970 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:25.873869    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll"
Nov 03 17:34:26.248886 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:34:26.248851    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec91acf_4d9b_43cd_a9f4_f8ce43212b9d.slice/crio-885ac54802de91a09f2f047cd3070b73a252478095f294b2dfc87ab194eb7fd8 WatchSource:0}: Error finding container 885ac54802de91a09f2f047cd3070b73a252478095f294b2dfc87ab194eb7fd8: Status 404 returned error can't find the container with id 885ac54802de91a09f2f047cd3070b73a252478095f294b2dfc87ab194eb7fd8
Nov 03 17:34:26.252730 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:26.252700    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll"]
Nov 03 17:34:26.685202 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:26.685165    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll" event={"ID":"4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d","Type":"ContainerStarted","Data":"bf101c32c926ff4037f57d5ec3911e54d53378379617637d1b1424b61963244c"}
Nov 03 17:34:26.685591 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:26.685227    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll" event={"ID":"4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d","Type":"ContainerStarted","Data":"885ac54802de91a09f2f047cd3070b73a252478095f294b2dfc87ab194eb7fd8"}
Nov 03 17:34:26.740966 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:26.740930    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c904b31-926pc"]
Nov 03 17:34:26.744985 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:26.744968    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c904b31-926pc"
Nov 03 17:34:26.804911 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:26.804840    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c904b31-926pc"]
Nov 03 17:34:26.838415 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:26.838376    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22zqh\" (UniqueName: \"kubernetes.io/projected/d7676565-4063-4272-8929-6e25a934ec5d-kube-api-access-22zqh\") pod \"maas-api-revoke-sub-6c904b31-926pc\" (UID: \"d7676565-4063-4272-8929-6e25a934ec5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c904b31-926pc"
Nov 03 17:34:26.838583 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:26.838496    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7676565-4063-4272-8929-6e25a934ec5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c904b31-926pc\" (UID: \"d7676565-4063-4272-8929-6e25a934ec5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c904b31-926pc"
Nov 03 17:34:26.838695 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:26.838676    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7676565-4063-4272-8929-6e25a934ec5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c904b31-926pc\" (UID: \"d7676565-4063-4272-8929-6e25a934ec5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c904b31-926pc"
Nov 03 17:34:26.857033 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:26.856980    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll" podStartSLOduration=1.8569656540000001 podStartE2EDuration="1.856965654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:26.855404851 +0000 UTC m=+1007.653117631" watchObservedRunningTime="2026-11-03 17:34:26.856965654 +0000 UTC m=+1007.654678435"
Nov 03 17:34:26.939856 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:26.939823    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7676565-4063-4272-8929-6e25a934ec5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c904b31-926pc\" (UID: \"d7676565-4063-4272-8929-6e25a934ec5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c904b31-926pc"
Nov 03 17:34:26.940023 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:26.939967    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22zqh\" (UniqueName: \"kubernetes.io/projected/d7676565-4063-4272-8929-6e25a934ec5d-kube-api-access-22zqh\") pod \"maas-api-revoke-sub-6c904b31-926pc\" (UID: \"d7676565-4063-4272-8929-6e25a934ec5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c904b31-926pc"
Nov 03 17:34:26.940023 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:26.940018    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7676565-4063-4272-8929-6e25a934ec5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c904b31-926pc\" (UID: \"d7676565-4063-4272-8929-6e25a934ec5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c904b31-926pc"
Nov 03 17:34:26.940607 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:26.940582    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7676565-4063-4272-8929-6e25a934ec5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c904b31-926pc\" (UID: \"d7676565-4063-4272-8929-6e25a934ec5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c904b31-926pc"
Nov 03 17:34:26.942225 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:26.942207    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7676565-4063-4272-8929-6e25a934ec5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c904b31-926pc\" (UID: \"d7676565-4063-4272-8929-6e25a934ec5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c904b31-926pc"
Nov 03 17:34:27.003778 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:27.003740    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22zqh\" (UniqueName: \"kubernetes.io/projected/d7676565-4063-4272-8929-6e25a934ec5d-kube-api-access-22zqh\") pod \"maas-api-revoke-sub-6c904b31-926pc\" (UID: \"d7676565-4063-4272-8929-6e25a934ec5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c904b31-926pc"
Nov 03 17:34:27.059073 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:27.058997    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c904b31-926pc"
Nov 03 17:34:27.438614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:34:27.438574    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7676565_4063_4272_8929_6e25a934ec5d.slice/crio-e3d3a93cb66dfa5de7a784fa8357b6af2fc1d32c8ca38778683593a5a34cb835 WatchSource:0}: Error finding container e3d3a93cb66dfa5de7a784fa8357b6af2fc1d32c8ca38778683593a5a34cb835: Status 404 returned error can't find the container with id e3d3a93cb66dfa5de7a784fa8357b6af2fc1d32c8ca38778683593a5a34cb835
Nov 03 17:34:27.442093 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:27.442067    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c904b31-926pc"]
Nov 03 17:34:27.694282 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:27.694221    6059 generic.go:356] "Generic (PLEG): container finished" podID="4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d" containerID="bf101c32c926ff4037f57d5ec3911e54d53378379617637d1b1424b61963244c" exitCode=0
Nov 03 17:34:27.694677 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:27.694302    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll" event={"ID":"4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d","Type":"ContainerDied","Data":"bf101c32c926ff4037f57d5ec3911e54d53378379617637d1b1424b61963244c"}
Nov 03 17:34:27.695843 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:27.695815    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c904b31-926pc" event={"ID":"d7676565-4063-4272-8929-6e25a934ec5d","Type":"ContainerStarted","Data":"e3d3a93cb66dfa5de7a784fa8357b6af2fc1d32c8ca38778683593a5a34cb835"}
Nov 03 17:34:27.924092 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:27.924060    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-03ebc6"
Nov 03 17:34:27.950313 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:27.950235    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bad05ec-aa3f-4101-adb8-e690e06d7c67-kube-api-access-dnr9j\" (UniqueName: \"kubernetes.io/projected/3bad05ec-aa3f-4101-adb8-e690e06d7c67-kube-api-access-dnr9j\") pod \"3bad05ec-aa3f-4101-adb8-e690e06d7c67\" (UID: \"3bad05ec-aa3f-4101-adb8-e690e06d7c67\") "
Nov 03 17:34:27.952262 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:27.952233    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bad05ec-aa3f-4101-adb8-e690e06d7c67-kube-api-access-dnr9j" pod "3bad05ec-aa3f-4101-adb8-e690e06d7c67" (UID: "3bad05ec-aa3f-4101-adb8-e690e06d7c67"). InnerVolumeSpecName "kube-api-access-dnr9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:34:28.057392 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:28.053407    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnr9j\" (UniqueName: \"kubernetes.io/projected/3bad05ec-aa3f-4101-adb8-e690e06d7c67-kube-api-access-dnr9j\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:34:28.232233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:28.232201    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:28.704053 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:28.703955    6059 generic.go:356] "Generic (PLEG): container finished" podID="3bad05ec-aa3f-4101-adb8-e690e06d7c67" containerID="9d6db13ed5d97ef398d302b61c72bf54357be3d27488c20a248dd731e0ed0f13" exitCode=137
Nov 03 17:34:28.704053 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:28.704006    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-03ebc6"
Nov 03 17:34:28.704053 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:28.704047    6059 scope.go:122] "RemoveContainer" containerID="9d6db13ed5d97ef398d302b61c72bf54357be3d27488c20a248dd731e0ed0f13"
Nov 03 17:34:28.709305 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:28.709279    6059 generic.go:356] "Generic (PLEG): container finished" podID="d7676565-4063-4272-8929-6e25a934ec5d" containerID="77d4243ce418b3bb4f6eb65df4dd304d381fe7fe1f80b8d02687c382c26342c3" exitCode=0
Nov 03 17:34:28.709459 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:28.709387    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c904b31-926pc" event={"ID":"d7676565-4063-4272-8929-6e25a934ec5d","Type":"ContainerDied","Data":"77d4243ce418b3bb4f6eb65df4dd304d381fe7fe1f80b8d02687c382c26342c3"}
Nov 03 17:34:28.718313 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:28.718295    6059 scope.go:122] "RemoveContainer" containerID="9d6db13ed5d97ef398d302b61c72bf54357be3d27488c20a248dd731e0ed0f13"
Nov 03 17:34:28.718701 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:34:28.718681    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d6db13ed5d97ef398d302b61c72bf54357be3d27488c20a248dd731e0ed0f13\": container with ID starting with 9d6db13ed5d97ef398d302b61c72bf54357be3d27488c20a248dd731e0ed0f13 not found: ID does not exist" containerID="9d6db13ed5d97ef398d302b61c72bf54357be3d27488c20a248dd731e0ed0f13"
Nov 03 17:34:28.718762 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:28.718710    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6db13ed5d97ef398d302b61c72bf54357be3d27488c20a248dd731e0ed0f13"} err="failed to get container status \"9d6db13ed5d97ef398d302b61c72bf54357be3d27488c20a248dd731e0ed0f13\": rpc error: code = NotFound desc = could not find container \"9d6db13ed5d97ef398d302b61c72bf54357be3d27488c20a248dd731e0ed0f13\": container with ID starting with 9d6db13ed5d97ef398d302b61c72bf54357be3d27488c20a248dd731e0ed0f13 not found: ID does not exist"
Nov 03 17:34:28.849237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:28.849205    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll"
Nov 03 17:34:28.882860 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:34:28.882828    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-03ebc6\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="3bad05ec-aa3f-4101-adb8-e690e06d7c67" pod="opendatahub/test-curl-7484-03ebc6"
Nov 03 17:34:28.965637 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:28.965603    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d-maas-api-openshift-ca\") pod \"4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d\" (UID: \"4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d\") "
Nov 03 17:34:28.965809 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:28.965692    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d-maas-api-xks-ca\") pod \"4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d\" (UID: \"4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d\") "
Nov 03 17:34:28.965889 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:28.965852    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d-kube-api-access-ng64k\" (UniqueName: \"kubernetes.io/projected/4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d-kube-api-access-ng64k\") pod \"4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d\" (UID: \"4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d\") "
Nov 03 17:34:28.965961 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:28.965932    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d-maas-api-openshift-ca" pod "4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d" (UID: "4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:34:28.966291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:28.966272    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:34:28.967774 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:28.967751    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d-maas-api-xks-ca" pod "4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d" (UID: "4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:34:28.967886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:28.967813    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d-kube-api-access-ng64k" pod "4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d" (UID: "4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d"). InnerVolumeSpecName "kube-api-access-ng64k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:34:29.056637 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:34:29.056596    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-03ebc6\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="3bad05ec-aa3f-4101-adb8-e690e06d7c67" pod="opendatahub/test-curl-7484-03ebc6"
Nov 03 17:34:29.067455 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:29.067433    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:34:29.067455 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:29.067457    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng64k\" (UniqueName: \"kubernetes.io/projected/4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d-kube-api-access-ng64k\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:34:29.336185 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:29.336103    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bad05ec-aa3f-4101-adb8-e690e06d7c67" path="/var/lib/kubelet/pods/3bad05ec-aa3f-4101-adb8-e690e06d7c67/volumes"
Nov 03 17:34:29.385844 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:34:29.385803    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-03ebc6\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="3bad05ec-aa3f-4101-adb8-e690e06d7c67" pod="opendatahub/test-curl-7484-03ebc6"
Nov 03 17:34:29.718279 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:29.718237    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll"
Nov 03 17:34:29.718279 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:29.718269    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll" event={"ID":"4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d","Type":"ContainerDied","Data":"885ac54802de91a09f2f047cd3070b73a252478095f294b2dfc87ab194eb7fd8"}
Nov 03 17:34:29.718825 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:29.718327    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="885ac54802de91a09f2f047cd3070b73a252478095f294b2dfc87ab194eb7fd8"
Nov 03 17:34:29.874745 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:29.874717    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c904b31-926pc"
Nov 03 17:34:29.980807 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:29.980721    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7676565-4063-4272-8929-6e25a934ec5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7676565-4063-4272-8929-6e25a934ec5d-maas-api-openshift-ca\") pod \"d7676565-4063-4272-8929-6e25a934ec5d\" (UID: \"d7676565-4063-4272-8929-6e25a934ec5d\") "
Nov 03 17:34:29.980973 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:29.980889    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7676565-4063-4272-8929-6e25a934ec5d-kube-api-access-22zqh\" (UniqueName: \"kubernetes.io/projected/d7676565-4063-4272-8929-6e25a934ec5d-kube-api-access-22zqh\") pod \"d7676565-4063-4272-8929-6e25a934ec5d\" (UID: \"d7676565-4063-4272-8929-6e25a934ec5d\") "
Nov 03 17:34:29.980973 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:29.980935    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7676565-4063-4272-8929-6e25a934ec5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7676565-4063-4272-8929-6e25a934ec5d-maas-api-xks-ca\") pod \"d7676565-4063-4272-8929-6e25a934ec5d\" (UID: \"d7676565-4063-4272-8929-6e25a934ec5d\") "
Nov 03 17:34:29.981158 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:29.981126    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7676565-4063-4272-8929-6e25a934ec5d-maas-api-openshift-ca" pod "d7676565-4063-4272-8929-6e25a934ec5d" (UID: "d7676565-4063-4272-8929-6e25a934ec5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:34:29.981513 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:29.981490    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7676565-4063-4272-8929-6e25a934ec5d-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:34:29.983031 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:29.983003    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7676565-4063-4272-8929-6e25a934ec5d-maas-api-xks-ca" pod "d7676565-4063-4272-8929-6e25a934ec5d" (UID: "d7676565-4063-4272-8929-6e25a934ec5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:34:29.983103 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:29.983046    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7676565-4063-4272-8929-6e25a934ec5d-kube-api-access-22zqh" pod "d7676565-4063-4272-8929-6e25a934ec5d" (UID: "d7676565-4063-4272-8929-6e25a934ec5d"). InnerVolumeSpecName "kube-api-access-22zqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:34:30.083184 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:30.083147    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22zqh\" (UniqueName: \"kubernetes.io/projected/d7676565-4063-4272-8929-6e25a934ec5d-kube-api-access-22zqh\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:34:30.083326 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:30.083188    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7676565-4063-4272-8929-6e25a934ec5d-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:34:30.728538 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:30.728504    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c904b31-926pc" event={"ID":"d7676565-4063-4272-8929-6e25a934ec5d","Type":"ContainerDied","Data":"e3d3a93cb66dfa5de7a784fa8357b6af2fc1d32c8ca38778683593a5a34cb835"}
Nov 03 17:34:30.728538 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:30.728538    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c904b31-926pc"
Nov 03 17:34:30.728948 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:30.728553    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3d3a93cb66dfa5de7a784fa8357b6af2fc1d32c8ca38778683593a5a34cb835"
Nov 03 17:34:32.890725 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:32.890673    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:34.535382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:34.534983    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv" podUID="7942e9d1-597f-47a6-9c70-1d174fe4c8e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:34:34.707381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:34.706457    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g" podUID="83ca6484-7e38-4782-b3a9-57b58bdc3920" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:34:35.229512 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:35.229468    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:34:35.229819 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:35.229785    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds" podUID="4f9ab6a0-a561-47fa-8cab-156ef9b8f331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:34:35.457033 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:35.456974    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:35.856452 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:35.856417    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:36.308576 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.308541    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q"]
Nov 03 17:34:36.309080 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.309067    6059 cpu_manager.go:414] "containerMap: removing container" podUID="d7676565-4063-4272-8929-6e25a934ec5d" containerName="revoke-keys"
Nov 03 17:34:36.309126 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.309082    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7676565-4063-4272-8929-6e25a934ec5d" containerName="revoke-keys"
Nov 03 17:34:36.309126 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.309099    6059 cpu_manager.go:414] "containerMap: removing container" podUID="4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d" containerName="revoke-keys"
Nov 03 17:34:36.309126 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.309104    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d" containerName="revoke-keys"
Nov 03 17:34:36.309211 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.309193    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7676565-4063-4272-8929-6e25a934ec5d" containerName="revoke-keys"
Nov 03 17:34:36.309211 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.309203    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d" containerName="revoke-keys"
Nov 03 17:34:36.312908 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.312886    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q"
Nov 03 17:34:36.368151 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.368106    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:34:36.419727 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.419695    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:34:36.419913 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.419760    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:34:36.419913 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.419831    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:34:36.423345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.423324    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q"]
Nov 03 17:34:36.460942 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.460910    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e21c192-qsd9q\" (UID: \"cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q"
Nov 03 17:34:36.461096 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.460963    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjbx2\" (UniqueName: \"kubernetes.io/projected/cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd-kube-api-access-fjbx2\") pod \"maas-api-revoke-sub-9e21c192-qsd9q\" (UID: \"cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q"
Nov 03 17:34:36.461200 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.461174    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e21c192-qsd9q\" (UID: \"cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q"
Nov 03 17:34:36.562711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.562635    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e21c192-qsd9q\" (UID: \"cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q"
Nov 03 17:34:36.562711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.562677    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjbx2\" (UniqueName: \"kubernetes.io/projected/cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd-kube-api-access-fjbx2\") pod \"maas-api-revoke-sub-9e21c192-qsd9q\" (UID: \"cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q"
Nov 03 17:34:36.562877 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.562730    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e21c192-qsd9q\" (UID: \"cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q"
Nov 03 17:34:36.563294 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.563269    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e21c192-qsd9q\" (UID: \"cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q"
Nov 03 17:34:36.564871 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.564845    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e21c192-qsd9q\" (UID: \"cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q"
Nov 03 17:34:36.625715 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.625685    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjbx2\" (UniqueName: \"kubernetes.io/projected/cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd-kube-api-access-fjbx2\") pod \"maas-api-revoke-sub-9e21c192-qsd9q\" (UID: \"cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q"
Nov 03 17:34:36.625869 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.625857    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q"
Nov 03 17:34:36.861174 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:34:36.861091    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:34:36.977896 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:36.977865    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:37.015516 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.015492    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q"]
Nov 03 17:34:37.468907 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.468864    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"]
Nov 03 17:34:37.475589 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.475561    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.539144 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.539060    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"]
Nov 03 17:34:37.557414 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.557351    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-8c9b0f-openshift-default-dockercfg-5s7kz\""
Nov 03 17:34:37.575906 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.575874    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-data\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.576063 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.575916    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-credential-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.576140 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.576090    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-token\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.576140 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.576132    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trlk6\" (UniqueName: \"kubernetes.io/projected/2861a97b-ab99-4cfc-8a5c-bd2408972572-kube-api-access-trlk6\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.576241 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.576195    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-podinfo\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.576282 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.576241    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-workload-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.576349 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.576331    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-workload-certs\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.576472 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.576435    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2861a97b-ab99-4cfc-8a5c-bd2408972572-istiod-ca-cert\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.576524 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.576491    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-envoy\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.591741 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.591706    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:37.678278 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.678233    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-data\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.678511 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.678295    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-credential-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.678511 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.678385    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-token\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.678511 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.678413    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trlk6\" (UniqueName: \"kubernetes.io/projected/2861a97b-ab99-4cfc-8a5c-bd2408972572-kube-api-access-trlk6\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.678511 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.678450    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-podinfo\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.678511 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.678498    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-workload-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.678799 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.678577    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-workload-certs\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.678799 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.678648    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2861a97b-ab99-4cfc-8a5c-bd2408972572-istiod-ca-cert\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.678799 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.678695    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-envoy\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.683846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.683810    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-data\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.684117 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.684094    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-credential-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.684376 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.684342    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-workload-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.684636 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.684604    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-workload-certs\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.685242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.685211    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2861a97b-ab99-4cfc-8a5c-bd2408972572-istiod-ca-cert\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.686544 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.686293    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-envoy\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.705540 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.705498    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-podinfo\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.751158 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.751124    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-token\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.751404 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.751377    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trlk6\" (UniqueName: \"kubernetes.io/projected/2861a97b-ab99-4cfc-8a5c-bd2408972572-kube-api-access-trlk6\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.792102 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.792010    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:37.808332 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.808300    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:37.840404 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.838987    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q" event={"ID":"cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd","Type":"ContainerStarted","Data":"9fda63b2860641b2711b05f545ee68ff7c66d9c47ebf747f0626c2dbc6aa737e"}
Nov 03 17:34:37.840404 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.839061    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q" event={"ID":"cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd","Type":"ContainerStarted","Data":"2305fbfd0e1517dffa8da3c465304eb9744233a29dd4eebb35f080d3a16bc50a"}
Nov 03 17:34:37.967232 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:37.967170    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q" podStartSLOduration=1.967153887 podStartE2EDuration="1.967153887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:37.963086101 +0000 UTC m=+1018.760798883" watchObservedRunningTime="2026-11-03 17:34:37.967153887 +0000 UTC m=+1018.764866661"
Nov 03 17:34:38.231441 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:34:38.231400    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2861a97b_ab99_4cfc_8a5c_bd2408972572.slice/crio-004e813e89b5d332c786f0eecf9fe1ce88a0c58ec083ca2145229acf2a64352e WatchSource:0}: Error finding container 004e813e89b5d332c786f0eecf9fe1ce88a0c58ec083ca2145229acf2a64352e: Status 404 returned error can't find the container with id 004e813e89b5d332c786f0eecf9fe1ce88a0c58ec083ca2145229acf2a64352e
Nov 03 17:34:38.233391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:38.233340    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"]
Nov 03 17:34:38.234085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:38.234054    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:34:38.234130 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:38.234118    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:34:38.234161 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:38.234153    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:34:38.365291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:38.365250    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:38.847419 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:38.847302    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l" event={"ID":"2861a97b-ab99-4cfc-8a5c-bd2408972572","Type":"ContainerStarted","Data":"503cfd967c23e1ec0f4e2fd0cb8b59adcee7f840f752e7aa337c5119f3856750"}
Nov 03 17:34:38.847419 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:38.847406    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l" event={"ID":"2861a97b-ab99-4cfc-8a5c-bd2408972572","Type":"ContainerStarted","Data":"004e813e89b5d332c786f0eecf9fe1ce88a0c58ec083ca2145229acf2a64352e"}
Nov 03 17:34:38.849164 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:38.849136    6059 generic.go:356] "Generic (PLEG): container finished" podID="cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd" containerID="9fda63b2860641b2711b05f545ee68ff7c66d9c47ebf747f0626c2dbc6aa737e" exitCode=0
Nov 03 17:34:38.849280 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:38.849173    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q" event={"ID":"cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd","Type":"ContainerDied","Data":"9fda63b2860641b2711b05f545ee68ff7c66d9c47ebf747f0626c2dbc6aa737e"}
Nov 03 17:34:39.025767 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:39.025711    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l" podStartSLOduration=2.02569216 podStartE2EDuration="2.02569216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:38.972165839 +0000 UTC m=+1019.769878621" watchObservedRunningTime="2026-11-03 17:34:39.02569216 +0000 UTC m=+1019.823404941"
Nov 03 17:34:39.511410 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:39.511350    6059 scope.go:122] "RemoveContainer" containerID="6c5c764eb473fbfa14d778d3a88446ee8cde0dc03a6b239680826399c41d9ce2"
Nov 03 17:34:39.793285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:39.793204    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:39.798710 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:39.798687    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:39.856809 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:39.856775    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:39.857821 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:39.857799    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:34:40.024468 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:40.024439    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q"
Nov 03 17:34:40.211064 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:40.211026    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd-maas-api-openshift-ca\") pod \"cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd\" (UID: \"cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd\") "
Nov 03 17:34:40.211533 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:40.211179    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd-maas-api-xks-ca\") pod \"cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd\" (UID: \"cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd\") "
Nov 03 17:34:40.211533 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:40.211302    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd-kube-api-access-fjbx2\" (UniqueName: \"kubernetes.io/projected/cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd-kube-api-access-fjbx2\") pod \"cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd\" (UID: \"cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd\") "
Nov 03 17:34:40.211533 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:40.211369    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd-maas-api-openshift-ca" pod "cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd" (UID: "cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:34:40.211808 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:40.211784    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:34:40.213483 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:40.213457    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd-kube-api-access-fjbx2" pod "cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd" (UID: "cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd"). InnerVolumeSpecName "kube-api-access-fjbx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:34:40.213626 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:40.213606    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd-maas-api-xks-ca" pod "cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd" (UID: "cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:34:40.312724 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:40.312688    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:34:40.312724 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:40.312720    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjbx2\" (UniqueName: \"kubernetes.io/projected/cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd-kube-api-access-fjbx2\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:34:40.863888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:40.863859    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q"
Nov 03 17:34:40.863888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:40.863874    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q" event={"ID":"cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd","Type":"ContainerDied","Data":"2305fbfd0e1517dffa8da3c465304eb9744233a29dd4eebb35f080d3a16bc50a"}
Nov 03 17:34:40.864095 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:40.863949    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2305fbfd0e1517dffa8da3c465304eb9744233a29dd4eebb35f080d3a16bc50a"
Nov 03 17:34:41.955919 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:41.955882    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:42.117239 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.117192    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-312bd564-tj8mt"]
Nov 03 17:34:42.117822 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.117805    6059 cpu_manager.go:414] "containerMap: removing container" podUID="cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd" containerName="revoke-keys"
Nov 03 17:34:42.117880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.117823    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd" containerName="revoke-keys"
Nov 03 17:34:42.117939 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.117929    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd" containerName="revoke-keys"
Nov 03 17:34:42.121622 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.121603    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-312bd564-tj8mt"
Nov 03 17:34:42.133425 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.133399    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d822d650-6b28-4159-a7e8-95cf97f75d46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-312bd564-tj8mt\" (UID: \"d822d650-6b28-4159-a7e8-95cf97f75d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-312bd564-tj8mt"
Nov 03 17:34:42.133552 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.133434    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d822d650-6b28-4159-a7e8-95cf97f75d46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-312bd564-tj8mt\" (UID: \"d822d650-6b28-4159-a7e8-95cf97f75d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-312bd564-tj8mt"
Nov 03 17:34:42.133622 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.133580    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-428cn\" (UniqueName: \"kubernetes.io/projected/d822d650-6b28-4159-a7e8-95cf97f75d46-kube-api-access-428cn\") pod \"maas-api-revoke-sub-312bd564-tj8mt\" (UID: \"d822d650-6b28-4159-a7e8-95cf97f75d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-312bd564-tj8mt"
Nov 03 17:34:42.182420 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.182390    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-312bd564-tj8mt"]
Nov 03 17:34:42.228725 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.228698    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:34:42.228940 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.228917    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:34:42.228940 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.228936    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:34:42.229015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.228971    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:34:42.234249 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.234217    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:42.234502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.234483    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d822d650-6b28-4159-a7e8-95cf97f75d46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-312bd564-tj8mt\" (UID: \"d822d650-6b28-4159-a7e8-95cf97f75d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-312bd564-tj8mt"
Nov 03 17:34:42.234600 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.234520    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d822d650-6b28-4159-a7e8-95cf97f75d46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-312bd564-tj8mt\" (UID: \"d822d650-6b28-4159-a7e8-95cf97f75d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-312bd564-tj8mt"
Nov 03 17:34:42.234659 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.234618    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-428cn\" (UniqueName: \"kubernetes.io/projected/d822d650-6b28-4159-a7e8-95cf97f75d46-kube-api-access-428cn\") pod \"maas-api-revoke-sub-312bd564-tj8mt\" (UID: \"d822d650-6b28-4159-a7e8-95cf97f75d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-312bd564-tj8mt"
Nov 03 17:34:42.235272 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.235251    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d822d650-6b28-4159-a7e8-95cf97f75d46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-312bd564-tj8mt\" (UID: \"d822d650-6b28-4159-a7e8-95cf97f75d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-312bd564-tj8mt"
Nov 03 17:34:42.236882 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.236858    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d822d650-6b28-4159-a7e8-95cf97f75d46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-312bd564-tj8mt\" (UID: \"d822d650-6b28-4159-a7e8-95cf97f75d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-312bd564-tj8mt"
Nov 03 17:34:42.300809 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.300782    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-428cn\" (UniqueName: \"kubernetes.io/projected/d822d650-6b28-4159-a7e8-95cf97f75d46-kube-api-access-428cn\") pod \"maas-api-revoke-sub-312bd564-tj8mt\" (UID: \"d822d650-6b28-4159-a7e8-95cf97f75d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-312bd564-tj8mt"
Nov 03 17:34:42.436938 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.436898    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-312bd564-tj8mt"
Nov 03 17:34:42.812117 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:34:42.812077    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd822d650_6b28_4159_a7e8_95cf97f75d46.slice/crio-2fd2053379b2f047f20e19f9c4df04b5e100027a02b728203d60fb3b69fecf10 WatchSource:0}: Error finding container 2fd2053379b2f047f20e19f9c4df04b5e100027a02b728203d60fb3b69fecf10: Status 404 returned error can't find the container with id 2fd2053379b2f047f20e19f9c4df04b5e100027a02b728203d60fb3b69fecf10
Nov 03 17:34:42.816172 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.816142    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-312bd564-tj8mt"]
Nov 03 17:34:42.880456 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:42.880421    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-312bd564-tj8mt" event={"ID":"d822d650-6b28-4159-a7e8-95cf97f75d46","Type":"ContainerStarted","Data":"2fd2053379b2f047f20e19f9c4df04b5e100027a02b728203d60fb3b69fecf10"}
Nov 03 17:34:43.159441 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:43.159327    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:43.842198 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:43.842164    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:43.888682 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:43.888644    6059 generic.go:356] "Generic (PLEG): container finished" podID="d822d650-6b28-4159-a7e8-95cf97f75d46" containerID="4f7bb50953c2fe3ee0585c54a4d24c7d2420c6b8b2f7416dead9d46826e02aa4" exitCode=0
Nov 03 17:34:43.888870 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:43.888727    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-312bd564-tj8mt" event={"ID":"d822d650-6b28-4159-a7e8-95cf97f75d46","Type":"ContainerDied","Data":"4f7bb50953c2fe3ee0585c54a4d24c7d2420c6b8b2f7416dead9d46826e02aa4"}
Nov 03 17:34:44.044673 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:44.044640    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:45.068440 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.068415    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-312bd564-tj8mt"
Nov 03 17:34:45.095270 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.095236    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:45.170021 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.169915    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d822d650-6b28-4159-a7e8-95cf97f75d46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d822d650-6b28-4159-a7e8-95cf97f75d46-maas-api-openshift-ca\") pod \"d822d650-6b28-4159-a7e8-95cf97f75d46\" (UID: \"d822d650-6b28-4159-a7e8-95cf97f75d46\") "
Nov 03 17:34:45.170021 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.169976    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d822d650-6b28-4159-a7e8-95cf97f75d46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d822d650-6b28-4159-a7e8-95cf97f75d46-maas-api-xks-ca\") pod \"d822d650-6b28-4159-a7e8-95cf97f75d46\" (UID: \"d822d650-6b28-4159-a7e8-95cf97f75d46\") "
Nov 03 17:34:45.170459 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.170306    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d822d650-6b28-4159-a7e8-95cf97f75d46-kube-api-access-428cn\" (UniqueName: \"kubernetes.io/projected/d822d650-6b28-4159-a7e8-95cf97f75d46-kube-api-access-428cn\") pod \"d822d650-6b28-4159-a7e8-95cf97f75d46\" (UID: \"d822d650-6b28-4159-a7e8-95cf97f75d46\") "
Nov 03 17:34:45.171421 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.171392    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d822d650-6b28-4159-a7e8-95cf97f75d46-maas-api-openshift-ca" pod "d822d650-6b28-4159-a7e8-95cf97f75d46" (UID: "d822d650-6b28-4159-a7e8-95cf97f75d46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:34:45.173524 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.173499    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d822d650-6b28-4159-a7e8-95cf97f75d46-maas-api-xks-ca" pod "d822d650-6b28-4159-a7e8-95cf97f75d46" (UID: "d822d650-6b28-4159-a7e8-95cf97f75d46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:34:45.173660 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.173632    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d822d650-6b28-4159-a7e8-95cf97f75d46-kube-api-access-428cn" pod "d822d650-6b28-4159-a7e8-95cf97f75d46" (UID: "d822d650-6b28-4159-a7e8-95cf97f75d46"). InnerVolumeSpecName "kube-api-access-428cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:34:45.272825 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.272788    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-428cn\" (UniqueName: \"kubernetes.io/projected/d822d650-6b28-4159-a7e8-95cf97f75d46-kube-api-access-428cn\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:34:45.272825 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.272821    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d822d650-6b28-4159-a7e8-95cf97f75d46-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:34:45.272825 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.272831    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d822d650-6b28-4159-a7e8-95cf97f75d46-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:34:45.275714 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.275686    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:45.445101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.445021    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5"]
Nov 03 17:34:45.445662 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.445646    6059 cpu_manager.go:414] "containerMap: removing container" podUID="d822d650-6b28-4159-a7e8-95cf97f75d46" containerName="revoke-keys"
Nov 03 17:34:45.445709 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.445664    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d822d650-6b28-4159-a7e8-95cf97f75d46" containerName="revoke-keys"
Nov 03 17:34:45.445779 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.445769    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d822d650-6b28-4159-a7e8-95cf97f75d46" containerName="revoke-keys"
Nov 03 17:34:45.449472 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.449452    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5"
Nov 03 17:34:45.475166 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.475135    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c4f4203-c72f-408c-9da3-f83a246aaeda-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5\" (UID: \"8c4f4203-c72f-408c-9da3-f83a246aaeda\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5"
Nov 03 17:34:45.475334 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.475309    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2gvg\" (UniqueName: \"kubernetes.io/projected/8c4f4203-c72f-408c-9da3-f83a246aaeda-kube-api-access-h2gvg\") pod \"payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5\" (UID: \"8c4f4203-c72f-408c-9da3-f83a246aaeda\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5"
Nov 03 17:34:45.508084 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.508048    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-8c9b0f-dockercfg-wl6h8\""
Nov 03 17:34:45.561713 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.561670    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-8c9b0f\""
Nov 03 17:34:45.588608 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.588568    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2gvg\" (UniqueName: \"kubernetes.io/projected/8c4f4203-c72f-408c-9da3-f83a246aaeda-kube-api-access-h2gvg\") pod \"payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5\" (UID: \"8c4f4203-c72f-408c-9da3-f83a246aaeda\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5"
Nov 03 17:34:45.590673 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.589232    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c4f4203-c72f-408c-9da3-f83a246aaeda-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5\" (UID: \"8c4f4203-c72f-408c-9da3-f83a246aaeda\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5"
Nov 03 17:34:45.590908 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.590218    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c4f4203-c72f-408c-9da3-f83a246aaeda-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5\" (UID: \"8c4f4203-c72f-408c-9da3-f83a246aaeda\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5"
Nov 03 17:34:45.594318 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.594285    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5"]
Nov 03 17:34:45.612856 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.612821    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25"]
Nov 03 17:34:45.627984 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.627957    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25"
Nov 03 17:34:45.651088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.651060    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2gvg\" (UniqueName: \"kubernetes.io/projected/8c4f4203-c72f-408c-9da3-f83a246aaeda-kube-api-access-h2gvg\") pod \"payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5\" (UID: \"8c4f4203-c72f-408c-9da3-f83a246aaeda\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5"
Nov 03 17:34:45.670051 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.670019    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25"]
Nov 03 17:34:45.692145 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.692107    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1341c5b9-3ac7-4b13-8026-cb22e3fea9c8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25\" (UID: \"1341c5b9-3ac7-4b13-8026-cb22e3fea9c8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25"
Nov 03 17:34:45.692326 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.692303    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx48x\" (UniqueName: \"kubernetes.io/projected/1341c5b9-3ac7-4b13-8026-cb22e3fea9c8-kube-api-access-mx48x\") pod \"payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25\" (UID: \"1341c5b9-3ac7-4b13-8026-cb22e3fea9c8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25"
Nov 03 17:34:45.764590 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.764555    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5"
Nov 03 17:34:45.793414 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.793374    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx48x\" (UniqueName: \"kubernetes.io/projected/1341c5b9-3ac7-4b13-8026-cb22e3fea9c8-kube-api-access-mx48x\") pod \"payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25\" (UID: \"1341c5b9-3ac7-4b13-8026-cb22e3fea9c8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25"
Nov 03 17:34:45.793598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.793489    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1341c5b9-3ac7-4b13-8026-cb22e3fea9c8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25\" (UID: \"1341c5b9-3ac7-4b13-8026-cb22e3fea9c8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25"
Nov 03 17:34:45.794163 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.794141    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1341c5b9-3ac7-4b13-8026-cb22e3fea9c8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25\" (UID: \"1341c5b9-3ac7-4b13-8026-cb22e3fea9c8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25"
Nov 03 17:34:45.858085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.858042    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx48x\" (UniqueName: \"kubernetes.io/projected/1341c5b9-3ac7-4b13-8026-cb22e3fea9c8-kube-api-access-mx48x\") pod \"payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25\" (UID: \"1341c5b9-3ac7-4b13-8026-cb22e3fea9c8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25"
Nov 03 17:34:45.904625 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.904588    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-312bd564-tj8mt"
Nov 03 17:34:45.904806 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.904596    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-312bd564-tj8mt" event={"ID":"d822d650-6b28-4159-a7e8-95cf97f75d46","Type":"ContainerDied","Data":"2fd2053379b2f047f20e19f9c4df04b5e100027a02b728203d60fb3b69fecf10"}
Nov 03 17:34:45.904806 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.904707    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fd2053379b2f047f20e19f9c4df04b5e100027a02b728203d60fb3b69fecf10"
Nov 03 17:34:45.941901 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:45.941873    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25"
Nov 03 17:34:46.139717 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:34:46.139675    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4f4203_c72f_408c_9da3_f83a246aaeda.slice/crio-92965ca1fe811e2b202d77835d2504339074603fd9b4533db99d901608cb6bea WatchSource:0}: Error finding container 92965ca1fe811e2b202d77835d2504339074603fd9b4533db99d901608cb6bea: Status 404 returned error can't find the container with id 92965ca1fe811e2b202d77835d2504339074603fd9b4533db99d901608cb6bea
Nov 03 17:34:46.142624 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:46.142491    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5"]
Nov 03 17:34:46.320351 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:34:46.320321    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1341c5b9_3ac7_4b13_8026_cb22e3fea9c8.slice/crio-a5ef8c3f691068c96cdd0461145a70c3d70c485cd0d649e1f8f7de2e4149e09f WatchSource:0}: Error finding container a5ef8c3f691068c96cdd0461145a70c3d70c485cd0d649e1f8f7de2e4149e09f: Status 404 returned error can't find the container with id a5ef8c3f691068c96cdd0461145a70c3d70c485cd0d649e1f8f7de2e4149e09f
Nov 03 17:34:46.323320 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:46.323298    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25"]
Nov 03 17:34:46.913496 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:46.913451    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25" event={"ID":"1341c5b9-3ac7-4b13-8026-cb22e3fea9c8","Type":"ContainerStarted","Data":"78e671491c9123a83d3a64432972863734f42fb24d56011f4c9c8d2abfbde142"}
Nov 03 17:34:46.913672 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:46.913519    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25" event={"ID":"1341c5b9-3ac7-4b13-8026-cb22e3fea9c8","Type":"ContainerStarted","Data":"a5ef8c3f691068c96cdd0461145a70c3d70c485cd0d649e1f8f7de2e4149e09f"}
Nov 03 17:34:46.913672 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:46.913550    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25"
Nov 03 17:34:46.915268 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:46.915245    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5" event={"ID":"8c4f4203-c72f-408c-9da3-f83a246aaeda","Type":"ContainerStarted","Data":"0f5ff96bc28bbdbb043a11dbc592635db7d096704e38ca8e38b7a40c05271e28"}
Nov 03 17:34:46.915400 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:46.915292    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5" event={"ID":"8c4f4203-c72f-408c-9da3-f83a246aaeda","Type":"ContainerStarted","Data":"92965ca1fe811e2b202d77835d2504339074603fd9b4533db99d901608cb6bea"}
Nov 03 17:34:46.915400 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:46.915321    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5"
Nov 03 17:34:47.036382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:47.036294    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25" podStartSLOduration=2.036281623 podStartE2EDuration="2.036281623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:47.03366192 +0000 UTC m=+1027.831374703" watchObservedRunningTime="2026-11-03 17:34:47.036281623 +0000 UTC m=+1027.833994423"
Nov 03 17:34:47.160233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:47.160177    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5" podStartSLOduration=2.160161766 podStartE2EDuration="2.160161766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:47.156571351 +0000 UTC m=+1027.954284132" watchObservedRunningTime="2026-11-03 17:34:47.160161766 +0000 UTC m=+1027.957874598"
Nov 03 17:34:48.148934 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:48.148891    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:50.319567 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:50.300805    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:34:50.319567 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:50.300867    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds" podUID="4f9ab6a0-a561-47fa-8cab-156ef9b8f331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:34:50.343406 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:50.343312    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:50.529380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:50.514541    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:34:50.529380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:50.514631    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925" podUID="49075270-6931-42bc-948c-d4cdfc72c5d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:34:50.555381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:50.549460    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:34:50.555381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:50.549530    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj" podUID="1aec66b4-f062-41dc-b9e7-f69143f51117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:34:53.051940 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:53.051906    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e38015c-hk58z"]
Nov 03 17:34:53.103490 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:53.103455    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e38015c-hk58z"]
Nov 03 17:34:53.232071 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:53.232026    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:53.336411 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:53.336331    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="320ccb69-f96d-47eb-81b0-e69a5758af8d" path="/var/lib/kubelet/pods/320ccb69-f96d-47eb-81b0-e69a5758af8d/volumes"
Nov 03 17:34:57.786230 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:57.786196    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:57.925162 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:57.925136    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5"
Nov 03 17:34:57.925351 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:57.925210    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25"
Nov 03 17:34:59.612905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:59.612862    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m"]
Nov 03 17:34:59.622074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:59.622044    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m"
Nov 03 17:34:59.677258 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:59.677226    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m"]
Nov 03 17:34:59.724761 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:59.724729    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:34:59.724761 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:59.724760    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:34:59.724978 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:59.724767    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:34:59.728268 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:59.728243    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:34:59.747291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:59.747264    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c41881f-5e83-4546-b412-8923b7a50af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec3560d3-hdc2m\" (UID: \"3c41881f-5e83-4546-b412-8923b7a50af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m"
Nov 03 17:34:59.747470 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:59.747309    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c41881f-5e83-4546-b412-8923b7a50af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec3560d3-hdc2m\" (UID: \"3c41881f-5e83-4546-b412-8923b7a50af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m"
Nov 03 17:34:59.747470 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:59.747325    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66rcn\" (UniqueName: \"kubernetes.io/projected/3c41881f-5e83-4546-b412-8923b7a50af5-kube-api-access-66rcn\") pod \"maas-api-revoke-sub-ec3560d3-hdc2m\" (UID: \"3c41881f-5e83-4546-b412-8923b7a50af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m"
Nov 03 17:34:59.849187 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:59.849150    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c41881f-5e83-4546-b412-8923b7a50af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec3560d3-hdc2m\" (UID: \"3c41881f-5e83-4546-b412-8923b7a50af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m"
Nov 03 17:34:59.849400 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:59.849215    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c41881f-5e83-4546-b412-8923b7a50af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec3560d3-hdc2m\" (UID: \"3c41881f-5e83-4546-b412-8923b7a50af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m"
Nov 03 17:34:59.849400 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:59.849244    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66rcn\" (UniqueName: \"kubernetes.io/projected/3c41881f-5e83-4546-b412-8923b7a50af5-kube-api-access-66rcn\") pod \"maas-api-revoke-sub-ec3560d3-hdc2m\" (UID: \"3c41881f-5e83-4546-b412-8923b7a50af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m"
Nov 03 17:34:59.849978 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:59.849952    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c41881f-5e83-4546-b412-8923b7a50af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec3560d3-hdc2m\" (UID: \"3c41881f-5e83-4546-b412-8923b7a50af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m"
Nov 03 17:34:59.851603 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:59.851580    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c41881f-5e83-4546-b412-8923b7a50af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec3560d3-hdc2m\" (UID: \"3c41881f-5e83-4546-b412-8923b7a50af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m"
Nov 03 17:34:59.914130 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:59.914053    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66rcn\" (UniqueName: \"kubernetes.io/projected/3c41881f-5e83-4546-b412-8923b7a50af5-kube-api-access-66rcn\") pod \"maas-api-revoke-sub-ec3560d3-hdc2m\" (UID: \"3c41881f-5e83-4546-b412-8923b7a50af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m"
Nov 03 17:34:59.935920 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:34:59.935885    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m"
Nov 03 17:35:00.315349 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:35:00.315316    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c41881f_5e83_4546_b412_8923b7a50af5.slice/crio-a1f5d57d44ca9b4465a5cac20c143271427b88020de98ed09c179084750a1b51 WatchSource:0}: Error finding container a1f5d57d44ca9b4465a5cac20c143271427b88020de98ed09c179084750a1b51: Status 404 returned error can't find the container with id a1f5d57d44ca9b4465a5cac20c143271427b88020de98ed09c179084750a1b51
Nov 03 17:35:00.320565 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:00.320540    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m"]
Nov 03 17:35:01.084372 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:01.084332    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m" event={"ID":"3c41881f-5e83-4546-b412-8923b7a50af5","Type":"ContainerStarted","Data":"fc04c4dedefee596304da30f6c528ff3eda2226f5cad6c9dd176516b63b284c7"}
Nov 03 17:35:01.084751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:01.084412    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m" event={"ID":"3c41881f-5e83-4546-b412-8923b7a50af5","Type":"ContainerStarted","Data":"a1f5d57d44ca9b4465a5cac20c143271427b88020de98ed09c179084750a1b51"}
Nov 03 17:35:01.207689 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:01.207623    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m" podStartSLOduration=2.207606097 podStartE2EDuration="2.207606097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:35:01.205281316 +0000 UTC m=+1042.002994112" watchObservedRunningTime="2026-11-03 17:35:01.207606097 +0000 UTC m=+1042.005318893"
Nov 03 17:35:02.094585 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:02.094550    6059 generic.go:356] "Generic (PLEG): container finished" podID="3c41881f-5e83-4546-b412-8923b7a50af5" containerID="fc04c4dedefee596304da30f6c528ff3eda2226f5cad6c9dd176516b63b284c7" exitCode=0
Nov 03 17:35:02.095001 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:02.094587    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m" event={"ID":"3c41881f-5e83-4546-b412-8923b7a50af5","Type":"ContainerDied","Data":"fc04c4dedefee596304da30f6c528ff3eda2226f5cad6c9dd176516b63b284c7"}
Nov 03 17:35:02.338414 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:02.338369    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:03.341281 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:03.341258    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m"
Nov 03 17:35:03.505379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:03.503913    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c41881f-5e83-4546-b412-8923b7a50af5-kube-api-access-66rcn\" (UniqueName: \"kubernetes.io/projected/3c41881f-5e83-4546-b412-8923b7a50af5-kube-api-access-66rcn\") pod \"3c41881f-5e83-4546-b412-8923b7a50af5\" (UID: \"3c41881f-5e83-4546-b412-8923b7a50af5\") "
Nov 03 17:35:03.505379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:03.504053    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c41881f-5e83-4546-b412-8923b7a50af5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c41881f-5e83-4546-b412-8923b7a50af5-maas-api-openshift-ca\") pod \"3c41881f-5e83-4546-b412-8923b7a50af5\" (UID: \"3c41881f-5e83-4546-b412-8923b7a50af5\") "
Nov 03 17:35:03.505379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:03.504170    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c41881f-5e83-4546-b412-8923b7a50af5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c41881f-5e83-4546-b412-8923b7a50af5-maas-api-xks-ca\") pod \"3c41881f-5e83-4546-b412-8923b7a50af5\" (UID: \"3c41881f-5e83-4546-b412-8923b7a50af5\") "
Nov 03 17:35:03.508073 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:03.508041    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c41881f-5e83-4546-b412-8923b7a50af5-maas-api-openshift-ca" pod "3c41881f-5e83-4546-b412-8923b7a50af5" (UID: "3c41881f-5e83-4546-b412-8923b7a50af5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:35:03.533580 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:03.533518    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c41881f-5e83-4546-b412-8923b7a50af5-kube-api-access-66rcn" pod "3c41881f-5e83-4546-b412-8923b7a50af5" (UID: "3c41881f-5e83-4546-b412-8923b7a50af5"). InnerVolumeSpecName "kube-api-access-66rcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:35:03.573946 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:03.573687    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c41881f-5e83-4546-b412-8923b7a50af5-maas-api-xks-ca" pod "3c41881f-5e83-4546-b412-8923b7a50af5" (UID: "3c41881f-5e83-4546-b412-8923b7a50af5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:35:03.608645 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:03.608590    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c41881f-5e83-4546-b412-8923b7a50af5-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:35:03.608924 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:03.608900    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c41881f-5e83-4546-b412-8923b7a50af5-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:35:03.609074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:03.609058    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66rcn\" (UniqueName: \"kubernetes.io/projected/3c41881f-5e83-4546-b412-8923b7a50af5-kube-api-access-66rcn\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:35:04.293184 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:04.293146    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m" event={"ID":"3c41881f-5e83-4546-b412-8923b7a50af5","Type":"ContainerDied","Data":"a1f5d57d44ca9b4465a5cac20c143271427b88020de98ed09c179084750a1b51"}
Nov 03 17:35:04.293601 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:04.293580    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1f5d57d44ca9b4465a5cac20c143271427b88020de98ed09c179084750a1b51"
Nov 03 17:35:04.293737 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:04.293464    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m"
Nov 03 17:35:05.937381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:05.934487    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v" podUID="2bbc9082-b2da-4198-8a8a-b6ab26b5011e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:35:06.093508 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:06.093454    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl" podUID="c194a4be-0499-419f-a4a1-910dbaafd303" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:35:06.258518 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:06.258473    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n" podUID="d6c1263b-e648-4f10-af7d-86397d6b207a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:35:07.085853 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:35:07.085788    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:35:07.585930 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:07.585867    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:07.784379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:07.783382    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:10.347888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:10.347852    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls"]
Nov 03 17:35:10.348450 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:10.348434    6059 cpu_manager.go:414] "containerMap: removing container" podUID="3c41881f-5e83-4546-b412-8923b7a50af5" containerName="revoke-keys"
Nov 03 17:35:10.348503 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:10.348451    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c41881f-5e83-4546-b412-8923b7a50af5" containerName="revoke-keys"
Nov 03 17:35:10.348676 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:10.348662    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c41881f-5e83-4546-b412-8923b7a50af5" containerName="revoke-keys"
Nov 03 17:35:10.353902 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:10.353881    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls"
Nov 03 17:35:10.393157 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:10.393119    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-597nh\" (UniqueName: \"kubernetes.io/projected/37b19cdf-69a5-4c58-84dc-7d08992c260a-kube-api-access-597nh\") pod \"maas-api-revoke-sub-e988894f-6qvls\" (UID: \"37b19cdf-69a5-4c58-84dc-7d08992c260a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls"
Nov 03 17:35:10.393341 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:10.393252    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37b19cdf-69a5-4c58-84dc-7d08992c260a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e988894f-6qvls\" (UID: \"37b19cdf-69a5-4c58-84dc-7d08992c260a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls"
Nov 03 17:35:10.393472 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:10.393454    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37b19cdf-69a5-4c58-84dc-7d08992c260a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e988894f-6qvls\" (UID: \"37b19cdf-69a5-4c58-84dc-7d08992c260a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls"
Nov 03 17:35:10.409575 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:10.409541    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:35:10.413109 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:10.413079    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls"]
Nov 03 17:35:10.460596 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:10.460567    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:35:10.461044 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:10.461029    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:35:10.461267 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:10.461255    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:35:10.494296 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:10.494265    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37b19cdf-69a5-4c58-84dc-7d08992c260a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e988894f-6qvls\" (UID: \"37b19cdf-69a5-4c58-84dc-7d08992c260a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls"
Nov 03 17:35:10.494521 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:10.494339    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-597nh\" (UniqueName: \"kubernetes.io/projected/37b19cdf-69a5-4c58-84dc-7d08992c260a-kube-api-access-597nh\") pod \"maas-api-revoke-sub-e988894f-6qvls\" (UID: \"37b19cdf-69a5-4c58-84dc-7d08992c260a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls"
Nov 03 17:35:10.494521 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:10.494427    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37b19cdf-69a5-4c58-84dc-7d08992c260a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e988894f-6qvls\" (UID: \"37b19cdf-69a5-4c58-84dc-7d08992c260a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls"
Nov 03 17:35:10.495104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:10.495079    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37b19cdf-69a5-4c58-84dc-7d08992c260a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e988894f-6qvls\" (UID: \"37b19cdf-69a5-4c58-84dc-7d08992c260a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls"
Nov 03 17:35:10.496669 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:10.496649    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37b19cdf-69a5-4c58-84dc-7d08992c260a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e988894f-6qvls\" (UID: \"37b19cdf-69a5-4c58-84dc-7d08992c260a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls"
Nov 03 17:35:10.558086 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:10.558048    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-597nh\" (UniqueName: \"kubernetes.io/projected/37b19cdf-69a5-4c58-84dc-7d08992c260a-kube-api-access-597nh\") pod \"maas-api-revoke-sub-e988894f-6qvls\" (UID: \"37b19cdf-69a5-4c58-84dc-7d08992c260a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls"
Nov 03 17:35:10.668182 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:10.668100    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls"
Nov 03 17:35:11.046975 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:35:11.046946    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b19cdf_69a5_4c58_84dc_7d08992c260a.slice/crio-e94333224ad8244f5eded779f23da7362d04f48ff4fb67416c620077d1528488 WatchSource:0}: Error finding container e94333224ad8244f5eded779f23da7362d04f48ff4fb67416c620077d1528488: Status 404 returned error can't find the container with id e94333224ad8244f5eded779f23da7362d04f48ff4fb67416c620077d1528488
Nov 03 17:35:11.051380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:11.051337    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls"]
Nov 03 17:35:11.411458 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:11.411416    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls" event={"ID":"37b19cdf-69a5-4c58-84dc-7d08992c260a","Type":"ContainerStarted","Data":"d62caa29bf94f9f6deb054a340b9d3e496a9f4a313909c511a93afb353765090"}
Nov 03 17:35:11.411838 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:11.411479    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls" event={"ID":"37b19cdf-69a5-4c58-84dc-7d08992c260a","Type":"ContainerStarted","Data":"e94333224ad8244f5eded779f23da7362d04f48ff4fb67416c620077d1528488"}
Nov 03 17:35:11.542208 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:11.542149    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls" podStartSLOduration=1.542132576 podStartE2EDuration="1.542132576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:35:11.541229971 +0000 UTC m=+1052.338942752" watchObservedRunningTime="2026-11-03 17:35:11.542132576 +0000 UTC m=+1052.339845358"
Nov 03 17:35:12.346919 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:12.346883    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:12.420878 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:12.420845    6059 generic.go:356] "Generic (PLEG): container finished" podID="37b19cdf-69a5-4c58-84dc-7d08992c260a" containerID="d62caa29bf94f9f6deb054a340b9d3e496a9f4a313909c511a93afb353765090" exitCode=0
Nov 03 17:35:12.421269 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:12.420935    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls" event={"ID":"37b19cdf-69a5-4c58-84dc-7d08992c260a","Type":"ContainerDied","Data":"d62caa29bf94f9f6deb054a340b9d3e496a9f4a313909c511a93afb353765090"}
Nov 03 17:35:13.589051 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:13.589025    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls"
Nov 03 17:35:13.631439 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:13.631406    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37b19cdf-69a5-4c58-84dc-7d08992c260a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37b19cdf-69a5-4c58-84dc-7d08992c260a-maas-api-openshift-ca\") pod \"37b19cdf-69a5-4c58-84dc-7d08992c260a\" (UID: \"37b19cdf-69a5-4c58-84dc-7d08992c260a\") "
Nov 03 17:35:13.631622 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:13.631605    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37b19cdf-69a5-4c58-84dc-7d08992c260a-kube-api-access-597nh\" (UniqueName: \"kubernetes.io/projected/37b19cdf-69a5-4c58-84dc-7d08992c260a-kube-api-access-597nh\") pod \"37b19cdf-69a5-4c58-84dc-7d08992c260a\" (UID: \"37b19cdf-69a5-4c58-84dc-7d08992c260a\") "
Nov 03 17:35:13.631706 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:13.631693    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37b19cdf-69a5-4c58-84dc-7d08992c260a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37b19cdf-69a5-4c58-84dc-7d08992c260a-maas-api-xks-ca\") pod \"37b19cdf-69a5-4c58-84dc-7d08992c260a\" (UID: \"37b19cdf-69a5-4c58-84dc-7d08992c260a\") "
Nov 03 17:35:13.631824 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:13.631793    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b19cdf-69a5-4c58-84dc-7d08992c260a-maas-api-openshift-ca" pod "37b19cdf-69a5-4c58-84dc-7d08992c260a" (UID: "37b19cdf-69a5-4c58-84dc-7d08992c260a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:35:13.632136 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:13.632113    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37b19cdf-69a5-4c58-84dc-7d08992c260a-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:35:13.633701 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:13.633676    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b19cdf-69a5-4c58-84dc-7d08992c260a-kube-api-access-597nh" pod "37b19cdf-69a5-4c58-84dc-7d08992c260a" (UID: "37b19cdf-69a5-4c58-84dc-7d08992c260a"). InnerVolumeSpecName "kube-api-access-597nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:35:13.633783 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:13.633752    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37b19cdf-69a5-4c58-84dc-7d08992c260a-maas-api-xks-ca" pod "37b19cdf-69a5-4c58-84dc-7d08992c260a" (UID: "37b19cdf-69a5-4c58-84dc-7d08992c260a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:35:13.733447 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:13.733412    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-597nh\" (UniqueName: \"kubernetes.io/projected/37b19cdf-69a5-4c58-84dc-7d08992c260a-kube-api-access-597nh\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:35:13.733447 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:13.733446    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37b19cdf-69a5-4c58-84dc-7d08992c260a-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:35:14.437151 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:14.437120    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls"
Nov 03 17:35:14.437310 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:14.437157    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls" event={"ID":"37b19cdf-69a5-4c58-84dc-7d08992c260a","Type":"ContainerDied","Data":"e94333224ad8244f5eded779f23da7362d04f48ff4fb67416c620077d1528488"}
Nov 03 17:35:14.437310 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:14.437209    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e94333224ad8244f5eded779f23da7362d04f48ff4fb67416c620077d1528488"
Nov 03 17:35:14.495803 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:35:14.495767    6059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b19cdf_69a5_4c58_84dc_7d08992c260a.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 17:35:16.935186 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:16.935149    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:21.014382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:20.999258    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7484-20f89c"]
Nov 03 17:35:21.014382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:21.000249    6059 cpu_manager.go:414] "containerMap: removing container" podUID="37b19cdf-69a5-4c58-84dc-7d08992c260a" containerName="revoke-keys"
Nov 03 17:35:21.014382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:21.000274    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37b19cdf-69a5-4c58-84dc-7d08992c260a" containerName="revoke-keys"
Nov 03 17:35:21.014382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:21.000455    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="37b19cdf-69a5-4c58-84dc-7d08992c260a" containerName="revoke-keys"
Nov 03 17:35:21.014382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:21.009568    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-20f89c"
Nov 03 17:35:21.070083 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:21.070053    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hjccf\""
Nov 03 17:35:21.151545 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:21.151501    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8qgt\" (UniqueName: \"kubernetes.io/projected/f83788c3-97ac-48ca-9a7b-6c22397f91c9-kube-api-access-q8qgt\") pod \"test-curl-7484-20f89c\" (UID: \"f83788c3-97ac-48ca-9a7b-6c22397f91c9\") " pod="opendatahub/test-curl-7484-20f89c"
Nov 03 17:35:21.181053 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:21.181008    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7484-20f89c"]
Nov 03 17:35:21.257313 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:21.257268    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8qgt\" (UniqueName: \"kubernetes.io/projected/f83788c3-97ac-48ca-9a7b-6c22397f91c9-kube-api-access-q8qgt\") pod \"test-curl-7484-20f89c\" (UID: \"f83788c3-97ac-48ca-9a7b-6c22397f91c9\") " pod="opendatahub/test-curl-7484-20f89c"
Nov 03 17:35:21.371428 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:21.370349    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8qgt\" (UniqueName: \"kubernetes.io/projected/f83788c3-97ac-48ca-9a7b-6c22397f91c9-kube-api-access-q8qgt\") pod \"test-curl-7484-20f89c\" (UID: \"f83788c3-97ac-48ca-9a7b-6c22397f91c9\") " pod="opendatahub/test-curl-7484-20f89c"
Nov 03 17:35:21.523381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:21.522666    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-20f89c"
Nov 03 17:35:22.475291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:22.475249    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:22.760389 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:22.757482    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g" podUID="83ca6484-7e38-4782-b3a9-57b58bdc3920" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:35:22.777904 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:22.776237    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7484-20f89c"]
Nov 03 17:35:22.777904 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:22.776862    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:22.908609 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:22.906534    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v" podUID="2bbc9082-b2da-4198-8a8a-b6ab26b5011e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 17:35:22.984415 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:22.983097    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:23.092481 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:23.091766    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl" podUID="c194a4be-0499-419f-a4a1-910dbaafd303" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:35:23.705153 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:23.705112    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7484-20f89c" event={"ID":"f83788c3-97ac-48ca-9a7b-6c22397f91c9","Type":"ContainerStarted","Data":"af4826a15c4ca122495aa526b8ad8bc171469e790de7805ceef8fddda1ada765"}
Nov 03 17:35:23.705596 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:23.705173    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7484-20f89c" event={"ID":"f83788c3-97ac-48ca-9a7b-6c22397f91c9","Type":"ContainerStarted","Data":"85f33ab2e875c6c2c66599073b0488be9c65c70088f1175238d66a84cdaf4f1d"}
Nov 03 17:35:23.844578 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:23.844518    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7484-20f89c" podStartSLOduration=3.844503725 podStartE2EDuration="3.844503725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:35:23.840350182 +0000 UTC m=+1064.638062964" watchObservedRunningTime="2026-11-03 17:35:23.844503725 +0000 UTC m=+1064.642216505"
Nov 03 17:35:24.158844 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:24.158793    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:25.202145 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:25.202103    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:25.371180 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:25.371144    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7484-20f89c"]
Nov 03 17:35:25.376203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:25.376170    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7484-20f89c"]
Nov 03 17:35:25.539536 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:25.539498    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7484-3e3a3d"]
Nov 03 17:35:25.540082 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:25.540066    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f83788c3-97ac-48ca-9a7b-6c22397f91c9" containerName="test-curl-7484-20f89c"
Nov 03 17:35:25.540147 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:25.540083    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f83788c3-97ac-48ca-9a7b-6c22397f91c9" containerName="test-curl-7484-20f89c"
Nov 03 17:35:25.540229 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:25.540219    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f83788c3-97ac-48ca-9a7b-6c22397f91c9" containerName="test-curl-7484-20f89c"
Nov 03 17:35:25.543935 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:25.543915    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-3e3a3d"
Nov 03 17:35:25.598915 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:25.598878    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tglqz\" (UniqueName: \"kubernetes.io/projected/a23a2e30-6e6d-437b-807d-85879a76f257-kube-api-access-tglqz\") pod \"test-curl-7484-3e3a3d\" (UID: \"a23a2e30-6e6d-437b-807d-85879a76f257\") " pod="opendatahub/test-curl-7484-3e3a3d"
Nov 03 17:35:25.599292 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:25.599263    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7484-3e3a3d"]
Nov 03 17:35:25.700688 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:25.700636    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tglqz\" (UniqueName: \"kubernetes.io/projected/a23a2e30-6e6d-437b-807d-85879a76f257-kube-api-access-tglqz\") pod \"test-curl-7484-3e3a3d\" (UID: \"a23a2e30-6e6d-437b-807d-85879a76f257\") " pod="opendatahub/test-curl-7484-3e3a3d"
Nov 03 17:35:25.719541 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:25.719500    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7484-20f89c" podUID="f83788c3-97ac-48ca-9a7b-6c22397f91c9" containerName="test-curl-7484-20f89c" containerID="cri-o://af4826a15c4ca122495aa526b8ad8bc171469e790de7805ceef8fddda1ada765" gracePeriod=2
Nov 03 17:35:25.773153 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:35:25.773110    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-20f89c\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="f83788c3-97ac-48ca-9a7b-6c22397f91c9" pod="opendatahub/test-curl-7484-20f89c"
Nov 03 17:35:25.776728 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:25.776702    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tglqz\" (UniqueName: \"kubernetes.io/projected/a23a2e30-6e6d-437b-807d-85879a76f257-kube-api-access-tglqz\") pod \"test-curl-7484-3e3a3d\" (UID: \"a23a2e30-6e6d-437b-807d-85879a76f257\") " pod="opendatahub/test-curl-7484-3e3a3d"
Nov 03 17:35:25.856971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:25.856877    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-3e3a3d"
Nov 03 17:35:26.231437 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:35:26.231406    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda23a2e30_6e6d_437b_807d_85879a76f257.slice/crio-b59fe4b625e63203f30bf4358afa419131387c3c67d7de4461ac0dd3110f1083 WatchSource:0}: Error finding container b59fe4b625e63203f30bf4358afa419131387c3c67d7de4461ac0dd3110f1083: Status 404 returned error can't find the container with id b59fe4b625e63203f30bf4358afa419131387c3c67d7de4461ac0dd3110f1083
Nov 03 17:35:26.235646 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:26.235622    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7484-3e3a3d"]
Nov 03 17:35:26.731801 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:26.731762    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7484-3e3a3d" event={"ID":"a23a2e30-6e6d-437b-807d-85879a76f257","Type":"ContainerStarted","Data":"d894f9edfec765844ee55af61f8dbd40ef793a795549fd7f8b6565077b89cbde"}
Nov 03 17:35:26.731801 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:26.731810    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7484-3e3a3d" event={"ID":"a23a2e30-6e6d-437b-807d-85879a76f257","Type":"ContainerStarted","Data":"b59fe4b625e63203f30bf4358afa419131387c3c67d7de4461ac0dd3110f1083"}
Nov 03 17:35:26.785051 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:35:26.785009    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-20f89c\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="f83788c3-97ac-48ca-9a7b-6c22397f91c9" pod="opendatahub/test-curl-7484-20f89c"
Nov 03 17:35:26.909964 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:26.909912    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7484-3e3a3d" podStartSLOduration=1.90989864 podStartE2EDuration="1.90989864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:35:26.907091191 +0000 UTC m=+1067.704803973" watchObservedRunningTime="2026-11-03 17:35:26.90989864 +0000 UTC m=+1067.707611438"
Nov 03 17:35:27.362839 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:27.362802    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:27.529894 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:27.529857    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff"]
Nov 03 17:35:27.534338 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:27.534312    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff"
Nov 03 17:35:27.589465 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:27.589438    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:35:27.627057 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:27.626952    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-121693e7-tdjff\" (UID: \"2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff"
Nov 03 17:35:27.627251 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:27.627167    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-121693e7-tdjff\" (UID: \"2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff"
Nov 03 17:35:27.627251 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:27.627205    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpdtw\" (UniqueName: \"kubernetes.io/projected/2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7-kube-api-access-lpdtw\") pod \"maas-api-revoke-sub-121693e7-tdjff\" (UID: \"2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff"
Nov 03 17:35:27.649248 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:27.649217    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:27.651141 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:27.651113    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff"]
Nov 03 17:35:27.651950 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:27.651925    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:35:27.652061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:27.652029    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:35:27.691950 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:27.691920    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:35:27.728339 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:27.728305    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-121693e7-tdjff\" (UID: \"2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff"
Nov 03 17:35:27.728466 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:27.728348    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpdtw\" (UniqueName: \"kubernetes.io/projected/2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7-kube-api-access-lpdtw\") pod \"maas-api-revoke-sub-121693e7-tdjff\" (UID: \"2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff"
Nov 03 17:35:27.728466 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:27.728460    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-121693e7-tdjff\" (UID: \"2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff"
Nov 03 17:35:27.729013 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:27.728989    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-121693e7-tdjff\" (UID: \"2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff"
Nov 03 17:35:27.730711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:27.730690    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-121693e7-tdjff\" (UID: \"2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff"
Nov 03 17:35:27.792073 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:27.792045    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpdtw\" (UniqueName: \"kubernetes.io/projected/2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7-kube-api-access-lpdtw\") pod \"maas-api-revoke-sub-121693e7-tdjff\" (UID: \"2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff"
Nov 03 17:35:27.849093 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:27.849055    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff"
Nov 03 17:35:28.020157 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.020130    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-20f89c"
Nov 03 17:35:28.133042 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.133003    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f83788c3-97ac-48ca-9a7b-6c22397f91c9-kube-api-access-q8qgt\" (UniqueName: \"kubernetes.io/projected/f83788c3-97ac-48ca-9a7b-6c22397f91c9-kube-api-access-q8qgt\") pod \"f83788c3-97ac-48ca-9a7b-6c22397f91c9\" (UID: \"f83788c3-97ac-48ca-9a7b-6c22397f91c9\") "
Nov 03 17:35:28.135108 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.135084    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f83788c3-97ac-48ca-9a7b-6c22397f91c9-kube-api-access-q8qgt" pod "f83788c3-97ac-48ca-9a7b-6c22397f91c9" (UID: "f83788c3-97ac-48ca-9a7b-6c22397f91c9"). InnerVolumeSpecName "kube-api-access-q8qgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:35:28.235091 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.235052    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8qgt\" (UniqueName: \"kubernetes.io/projected/f83788c3-97ac-48ca-9a7b-6c22397f91c9-kube-api-access-q8qgt\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:35:28.264473 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:35:28.264441    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd9ecd8_3fd2_44cc_8005_f30e6c0263d7.slice/crio-ebdb11cdd1d68a456eefe550fb429df079e1f1687e3bc1eb868ec17b469c238f WatchSource:0}: Error finding container ebdb11cdd1d68a456eefe550fb429df079e1f1687e3bc1eb868ec17b469c238f: Status 404 returned error can't find the container with id ebdb11cdd1d68a456eefe550fb429df079e1f1687e3bc1eb868ec17b469c238f
Nov 03 17:35:28.268639 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.268614    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff"]
Nov 03 17:35:28.515345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.515260    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7484-3e3a3d"]
Nov 03 17:35:28.522629 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.522602    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7484-3e3a3d"]
Nov 03 17:35:28.681479 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.681442    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7484-0dc99d"]
Nov 03 17:35:28.682060 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.682045    6059 cpu_manager.go:414] "containerMap: removing container" podUID="a23a2e30-6e6d-437b-807d-85879a76f257" containerName="test-curl-7484-3e3a3d"
Nov 03 17:35:28.682101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.682061    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a23a2e30-6e6d-437b-807d-85879a76f257" containerName="test-curl-7484-3e3a3d"
Nov 03 17:35:28.682158 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.682149    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a23a2e30-6e6d-437b-807d-85879a76f257" containerName="test-curl-7484-3e3a3d"
Nov 03 17:35:28.685719 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.685702    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-0dc99d"
Nov 03 17:35:28.742302 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.742274    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7484-0dc99d"]
Nov 03 17:35:28.753827 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.753793    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff" event={"ID":"2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7","Type":"ContainerStarted","Data":"a25b3ee0d142f2e267d68e42cbea5fc9050eec61a40709f089708435eef862bd"}
Nov 03 17:35:28.754011 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.753856    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff" event={"ID":"2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7","Type":"ContainerStarted","Data":"ebdb11cdd1d68a456eefe550fb429df079e1f1687e3bc1eb868ec17b469c238f"}
Nov 03 17:35:28.755676 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.755639    6059 generic.go:356] "Generic (PLEG): container finished" podID="f83788c3-97ac-48ca-9a7b-6c22397f91c9" containerID="af4826a15c4ca122495aa526b8ad8bc171469e790de7805ceef8fddda1ada765" exitCode=137
Nov 03 17:35:28.755874 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.755697    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-20f89c"
Nov 03 17:35:28.755874 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.755735    6059 scope.go:122] "RemoveContainer" containerID="af4826a15c4ca122495aa526b8ad8bc171469e790de7805ceef8fddda1ada765"
Nov 03 17:35:28.756123 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.756065    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7484-3e3a3d" podUID="a23a2e30-6e6d-437b-807d-85879a76f257" containerName="test-curl-7484-3e3a3d" containerID="cri-o://d894f9edfec765844ee55af61f8dbd40ef793a795549fd7f8b6565077b89cbde" gracePeriod=2
Nov 03 17:35:28.767176 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.767140    6059 scope.go:122] "RemoveContainer" containerID="af4826a15c4ca122495aa526b8ad8bc171469e790de7805ceef8fddda1ada765"
Nov 03 17:35:28.767504 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:35:28.767478    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af4826a15c4ca122495aa526b8ad8bc171469e790de7805ceef8fddda1ada765\": container with ID starting with af4826a15c4ca122495aa526b8ad8bc171469e790de7805ceef8fddda1ada765 not found: ID does not exist" containerID="af4826a15c4ca122495aa526b8ad8bc171469e790de7805ceef8fddda1ada765"
Nov 03 17:35:28.767573 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.767514    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af4826a15c4ca122495aa526b8ad8bc171469e790de7805ceef8fddda1ada765"} err="failed to get container status \"af4826a15c4ca122495aa526b8ad8bc171469e790de7805ceef8fddda1ada765\": rpc error: code = NotFound desc = could not find container \"af4826a15c4ca122495aa526b8ad8bc171469e790de7805ceef8fddda1ada765\": container with ID starting with af4826a15c4ca122495aa526b8ad8bc171469e790de7805ceef8fddda1ada765 not found: ID does not exist"
Nov 03 17:35:28.844891 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.844845    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m6pd\" (UniqueName: \"kubernetes.io/projected/9afbe75d-6028-4c46-b25d-f84350dc689e-kube-api-access-8m6pd\") pod \"test-curl-7484-0dc99d\" (UID: \"9afbe75d-6028-4c46-b25d-f84350dc689e\") " pod="opendatahub/test-curl-7484-0dc99d"
Nov 03 17:35:28.931173 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.931115    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff" podStartSLOduration=1.9311001330000002 podStartE2EDuration="1.931100133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:35:28.923986722 +0000 UTC m=+1069.721699504" watchObservedRunningTime="2026-11-03 17:35:28.931100133 +0000 UTC m=+1069.728812913"
Nov 03 17:35:28.946049 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:28.946012    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m6pd\" (UniqueName: \"kubernetes.io/projected/9afbe75d-6028-4c46-b25d-f84350dc689e-kube-api-access-8m6pd\") pod \"test-curl-7484-0dc99d\" (UID: \"9afbe75d-6028-4c46-b25d-f84350dc689e\") " pod="opendatahub/test-curl-7484-0dc99d"
Nov 03 17:35:28.977055 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:35:28.977009    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-3e3a3d\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="a23a2e30-6e6d-437b-807d-85879a76f257" pod="opendatahub/test-curl-7484-3e3a3d"
Nov 03 17:35:29.015236 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:29.015207    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m6pd\" (UniqueName: \"kubernetes.io/projected/9afbe75d-6028-4c46-b25d-f84350dc689e-kube-api-access-8m6pd\") pod \"test-curl-7484-0dc99d\" (UID: \"9afbe75d-6028-4c46-b25d-f84350dc689e\") " pod="opendatahub/test-curl-7484-0dc99d"
Nov 03 17:35:29.030283 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:35:29.030200    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-20f89c\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="f83788c3-97ac-48ca-9a7b-6c22397f91c9" pod="opendatahub/test-curl-7484-20f89c"
Nov 03 17:35:29.298665 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:29.298580    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-0dc99d"
Nov 03 17:35:29.339168 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:29.339126    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f83788c3-97ac-48ca-9a7b-6c22397f91c9" path="/var/lib/kubelet/pods/f83788c3-97ac-48ca-9a7b-6c22397f91c9/volumes"
Nov 03 17:35:29.388212 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:35:29.388168    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-3e3a3d\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="a23a2e30-6e6d-437b-807d-85879a76f257" pod="opendatahub/test-curl-7484-3e3a3d"
Nov 03 17:35:29.441460 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:35:29.441419    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-20f89c\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="f83788c3-97ac-48ca-9a7b-6c22397f91c9" pod="opendatahub/test-curl-7484-20f89c"
Nov 03 17:35:29.671488 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:35:29.671439    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9afbe75d_6028_4c46_b25d_f84350dc689e.slice/crio-5426ccab9b922113ff48c766b1b4f7d4cb90f213a2c01bf57c74c2e00bcddd18 WatchSource:0}: Error finding container 5426ccab9b922113ff48c766b1b4f7d4cb90f213a2c01bf57c74c2e00bcddd18: Status 404 returned error can't find the container with id 5426ccab9b922113ff48c766b1b4f7d4cb90f213a2c01bf57c74c2e00bcddd18
Nov 03 17:35:29.675348 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:29.675326    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7484-0dc99d"]
Nov 03 17:35:29.766421 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:29.766347    6059 generic.go:356] "Generic (PLEG): container finished" podID="2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7" containerID="a25b3ee0d142f2e267d68e42cbea5fc9050eec61a40709f089708435eef862bd" exitCode=0
Nov 03 17:35:29.766421 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:29.766387    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff" event={"ID":"2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7","Type":"ContainerDied","Data":"a25b3ee0d142f2e267d68e42cbea5fc9050eec61a40709f089708435eef862bd"}
Nov 03 17:35:29.767940 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:29.767917    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7484-0dc99d" event={"ID":"9afbe75d-6028-4c46-b25d-f84350dc689e","Type":"ContainerStarted","Data":"2dc99c3242b5ae5898eafdcfe67a5ffc25a4ec06dd50c3b0d282be0a4e88a272"}
Nov 03 17:35:29.768054 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:29.767952    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7484-0dc99d" event={"ID":"9afbe75d-6028-4c46-b25d-f84350dc689e","Type":"ContainerStarted","Data":"5426ccab9b922113ff48c766b1b4f7d4cb90f213a2c01bf57c74c2e00bcddd18"}
Nov 03 17:35:30.016729 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:30.016667    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7484-0dc99d" podStartSLOduration=2.016646322 podStartE2EDuration="2.016646322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:35:30.014630019 +0000 UTC m=+1070.812342800" watchObservedRunningTime="2026-11-03 17:35:30.016646322 +0000 UTC m=+1070.814359102"
Nov 03 17:35:30.934075 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:30.934048    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff"
Nov 03 17:35:31.010430 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.010407    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-3e3a3d"
Nov 03 17:35:31.072898 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.072808    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7-maas-api-xks-ca\") pod \"2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7\" (UID: \"2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7\") "
Nov 03 17:35:31.072898 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.072849    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7-maas-api-openshift-ca\") pod \"2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7\" (UID: \"2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7\") "
Nov 03 17:35:31.073122 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.073029    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7-kube-api-access-lpdtw\" (UniqueName: \"kubernetes.io/projected/2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7-kube-api-access-lpdtw\") pod \"2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7\" (UID: \"2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7\") "
Nov 03 17:35:31.073225 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.073195    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7-maas-api-openshift-ca" pod "2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7" (UID: "2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:35:31.073523 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.073492    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:35:31.075132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.075111    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7-kube-api-access-lpdtw" pod "2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7" (UID: "2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7"). InnerVolumeSpecName "kube-api-access-lpdtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:35:31.075254 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.075236    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7-maas-api-xks-ca" pod "2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7" (UID: "2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:35:31.173943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.173906    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23a2e30-6e6d-437b-807d-85879a76f257-kube-api-access-tglqz\" (UniqueName: \"kubernetes.io/projected/a23a2e30-6e6d-437b-807d-85879a76f257-kube-api-access-tglqz\") pod \"a23a2e30-6e6d-437b-807d-85879a76f257\" (UID: \"a23a2e30-6e6d-437b-807d-85879a76f257\") "
Nov 03 17:35:31.174345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.174331    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpdtw\" (UniqueName: \"kubernetes.io/projected/2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7-kube-api-access-lpdtw\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:35:31.174345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.174346    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:35:31.176046 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.176012    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23a2e30-6e6d-437b-807d-85879a76f257-kube-api-access-tglqz" pod "a23a2e30-6e6d-437b-807d-85879a76f257" (UID: "a23a2e30-6e6d-437b-807d-85879a76f257"). InnerVolumeSpecName "kube-api-access-tglqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:35:31.275384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.275324    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tglqz\" (UniqueName: \"kubernetes.io/projected/a23a2e30-6e6d-437b-807d-85879a76f257-kube-api-access-tglqz\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:35:31.336164 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.336087    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a23a2e30-6e6d-437b-807d-85879a76f257" path="/var/lib/kubelet/pods/a23a2e30-6e6d-437b-807d-85879a76f257/volumes"
Nov 03 17:35:31.450813 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.450776    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7484-0dc99d"]
Nov 03 17:35:31.456374 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.456328    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7484-0dc99d"]
Nov 03 17:35:31.615524 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.615440    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7484-0651f2"]
Nov 03 17:35:31.616003 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.615988    6059 cpu_manager.go:414] "containerMap: removing container" podUID="2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7" containerName="revoke-keys"
Nov 03 17:35:31.616043 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.616004    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7" containerName="revoke-keys"
Nov 03 17:35:31.616043 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.616024    6059 cpu_manager.go:414] "containerMap: removing container" podUID="9afbe75d-6028-4c46-b25d-f84350dc689e" containerName="test-curl-7484-0dc99d"
Nov 03 17:35:31.616043 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.616029    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9afbe75d-6028-4c46-b25d-f84350dc689e" containerName="test-curl-7484-0dc99d"
Nov 03 17:35:31.616153 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.616126    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9afbe75d-6028-4c46-b25d-f84350dc689e" containerName="test-curl-7484-0dc99d"
Nov 03 17:35:31.616153 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.616145    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7" containerName="revoke-keys"
Nov 03 17:35:31.619855 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.619836    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-0651f2"
Nov 03 17:35:31.677214 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.677181    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7484-0651f2"]
Nov 03 17:35:31.781334 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.781295    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7v42\" (UniqueName: \"kubernetes.io/projected/7f12033e-64f7-4297-bfed-85da58f409cb-kube-api-access-w7v42\") pod \"test-curl-7484-0651f2\" (UID: \"7f12033e-64f7-4297-bfed-85da58f409cb\") " pod="opendatahub/test-curl-7484-0651f2"
Nov 03 17:35:31.784666 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.784640    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff"
Nov 03 17:35:31.784784 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.784677    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff" event={"ID":"2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7","Type":"ContainerDied","Data":"ebdb11cdd1d68a456eefe550fb429df079e1f1687e3bc1eb868ec17b469c238f"}
Nov 03 17:35:31.784784 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.784727    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebdb11cdd1d68a456eefe550fb429df079e1f1687e3bc1eb868ec17b469c238f"
Nov 03 17:35:31.785958 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.785935    6059 generic.go:356] "Generic (PLEG): container finished" podID="a23a2e30-6e6d-437b-807d-85879a76f257" containerID="d894f9edfec765844ee55af61f8dbd40ef793a795549fd7f8b6565077b89cbde" exitCode=137
Nov 03 17:35:31.786069 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.785974    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-3e3a3d"
Nov 03 17:35:31.786069 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.786033    6059 scope.go:122] "RemoveContainer" containerID="d894f9edfec765844ee55af61f8dbd40ef793a795549fd7f8b6565077b89cbde"
Nov 03 17:35:31.786268 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.786244    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7484-0dc99d" podUID="9afbe75d-6028-4c46-b25d-f84350dc689e" containerName="test-curl-7484-0dc99d" containerID="cri-o://2dc99c3242b5ae5898eafdcfe67a5ffc25a4ec06dd50c3b0d282be0a4e88a272" gracePeriod=2
Nov 03 17:35:31.795709 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.795688    6059 scope.go:122] "RemoveContainer" containerID="d894f9edfec765844ee55af61f8dbd40ef793a795549fd7f8b6565077b89cbde"
Nov 03 17:35:31.795988 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:35:31.795963    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d894f9edfec765844ee55af61f8dbd40ef793a795549fd7f8b6565077b89cbde\": container with ID starting with d894f9edfec765844ee55af61f8dbd40ef793a795549fd7f8b6565077b89cbde not found: ID does not exist" containerID="d894f9edfec765844ee55af61f8dbd40ef793a795549fd7f8b6565077b89cbde"
Nov 03 17:35:31.796033 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.795989    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d894f9edfec765844ee55af61f8dbd40ef793a795549fd7f8b6565077b89cbde"} err="failed to get container status \"d894f9edfec765844ee55af61f8dbd40ef793a795549fd7f8b6565077b89cbde\": rpc error: code = NotFound desc = could not find container \"d894f9edfec765844ee55af61f8dbd40ef793a795549fd7f8b6565077b89cbde\": container with ID starting with d894f9edfec765844ee55af61f8dbd40ef793a795549fd7f8b6565077b89cbde not found: ID does not exist"
Nov 03 17:35:31.839435 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:35:31.839387    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-0dc99d\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="9afbe75d-6028-4c46-b25d-f84350dc689e" pod="opendatahub/test-curl-7484-0dc99d"
Nov 03 17:35:31.882241 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.882154    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7v42\" (UniqueName: \"kubernetes.io/projected/7f12033e-64f7-4297-bfed-85da58f409cb-kube-api-access-w7v42\") pod \"test-curl-7484-0651f2\" (UID: \"7f12033e-64f7-4297-bfed-85da58f409cb\") " pod="opendatahub/test-curl-7484-0651f2"
Nov 03 17:35:31.892448 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:35:31.892405    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-0dc99d\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="9afbe75d-6028-4c46-b25d-f84350dc689e" pod="opendatahub/test-curl-7484-0dc99d"
Nov 03 17:35:31.945762 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.945726    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7v42\" (UniqueName: \"kubernetes.io/projected/7f12033e-64f7-4297-bfed-85da58f409cb-kube-api-access-w7v42\") pod \"test-curl-7484-0651f2\" (UID: \"7f12033e-64f7-4297-bfed-85da58f409cb\") " pod="opendatahub/test-curl-7484-0651f2"
Nov 03 17:35:31.955202 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:31.955168    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:32.235941 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:32.234561    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-0651f2"
Nov 03 17:35:32.636711 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:35:32.636683    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f12033e_64f7_4297_bfed_85da58f409cb.slice/crio-09f9f68a714ea295d0b4d049ada4a7ad6818ef17e34db1d4e9124ea2e38f670c WatchSource:0}: Error finding container 09f9f68a714ea295d0b4d049ada4a7ad6818ef17e34db1d4e9124ea2e38f670c: Status 404 returned error can't find the container with id 09f9f68a714ea295d0b4d049ada4a7ad6818ef17e34db1d4e9124ea2e38f670c
Nov 03 17:35:32.640142 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:32.640120    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7484-0651f2"]
Nov 03 17:35:32.877403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:32.851171    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7484-0651f2" event={"ID":"7f12033e-64f7-4297-bfed-85da58f409cb","Type":"ContainerStarted","Data":"1171a48bd1c5871f8bad6446dfbadcee425416ce914e4d0a24415e6a270be271"}
Nov 03 17:35:32.877403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:32.851239    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7484-0651f2" event={"ID":"7f12033e-64f7-4297-bfed-85da58f409cb","Type":"ContainerStarted","Data":"09f9f68a714ea295d0b4d049ada4a7ad6818ef17e34db1d4e9124ea2e38f670c"}
Nov 03 17:35:33.113727 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:35:33.113665    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-0dc99d\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="9afbe75d-6028-4c46-b25d-f84350dc689e" pod="opendatahub/test-curl-7484-0dc99d"
Nov 03 17:35:34.127082 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:34.126947    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7484-0651f2" podStartSLOduration=3.126924298 podStartE2EDuration="3.126924298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:35:33.039958407 +0000 UTC m=+1073.837671213" watchObservedRunningTime="2026-11-03 17:35:34.126924298 +0000 UTC m=+1074.924637080"
Nov 03 17:35:34.128433 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:34.128402    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf"]
Nov 03 17:35:34.187611 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:34.187566    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7f136fb-sthsf"]
Nov 03 17:35:34.792384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:34.790837    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g" podUID="83ca6484-7e38-4782-b3a9-57b58bdc3920" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 03 17:35:34.950600 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:34.948308    6059 generic.go:356] "Generic (PLEG): container finished" podID="9afbe75d-6028-4c46-b25d-f84350dc689e" containerID="2dc99c3242b5ae5898eafdcfe67a5ffc25a4ec06dd50c3b0d282be0a4e88a272" exitCode=137
Nov 03 17:35:34.963288 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:34.962503    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v" podUID="2bbc9082-b2da-4198-8a8a-b6ab26b5011e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:35:35.019382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:35.016815    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-0dc99d"
Nov 03 17:35:35.043377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:35.026888    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:35:35.043377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:35.026987    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj" podUID="cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:35:35.102418 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:35.082461    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:35:35.102418 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:35.082525    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv" podUID="89c1509f-87be-4742-86ac-65c438f9523b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:35:35.160634 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:35.160590    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9afbe75d-6028-4c46-b25d-f84350dc689e-kube-api-access-8m6pd\" (UniqueName: \"kubernetes.io/projected/9afbe75d-6028-4c46-b25d-f84350dc689e-kube-api-access-8m6pd\") pod \"9afbe75d-6028-4c46-b25d-f84350dc689e\" (UID: \"9afbe75d-6028-4c46-b25d-f84350dc689e\") "
Nov 03 17:35:35.194443 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:35.193988    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afbe75d-6028-4c46-b25d-f84350dc689e-kube-api-access-8m6pd" pod "9afbe75d-6028-4c46-b25d-f84350dc689e" (UID: "9afbe75d-6028-4c46-b25d-f84350dc689e"). InnerVolumeSpecName "kube-api-access-8m6pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:35:35.270382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:35.269451    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n" podUID="d6c1263b-e648-4f10-af7d-86397d6b207a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:35:35.271337 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:35.271303    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m6pd\" (UniqueName: \"kubernetes.io/projected/9afbe75d-6028-4c46-b25d-f84350dc689e-kube-api-access-8m6pd\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:35:35.366313 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:35.365075    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2733a212-3155-401f-b9d5-e3800a07bc79" path="/var/lib/kubelet/pods/2733a212-3155-401f-b9d5-e3800a07bc79/volumes"
Nov 03 17:35:35.366313 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:35.365693    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9afbe75d-6028-4c46-b25d-f84350dc689e" path="/var/lib/kubelet/pods/9afbe75d-6028-4c46-b25d-f84350dc689e/volumes"
Nov 03 17:35:35.551381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:35.550468    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv" podUID="7942e9d1-597f-47a6-9c70-1d174fe4c8e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:35:35.793818 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:35.793738    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g" podUID="83ca6484-7e38-4782-b3a9-57b58bdc3920" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:35:35.794978 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:35.794955    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:35:35.964461 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:35.964412    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v" podUID="2bbc9082-b2da-4198-8a8a-b6ab26b5011e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:35:35.977385 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:35.965266    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:35:36.107382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:36.071720    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl" podUID="c194a4be-0499-419f-a4a1-910dbaafd303" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:35:36.275847 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:36.275800    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n" podUID="d6c1263b-e648-4f10-af7d-86397d6b207a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded"
Nov 03 17:35:36.280839 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:36.280808    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:35:36.411671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:36.411629    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:36.417576 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:36.417261    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-0dc99d"
Nov 03 17:35:36.436129 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:36.417382    6059 scope.go:122] "RemoveContainer" containerID="2dc99c3242b5ae5898eafdcfe67a5ffc25a4ec06dd50c3b0d282be0a4e88a272"
Nov 03 17:35:36.561116 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:36.558691    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv" podUID="7942e9d1-597f-47a6-9c70-1d174fe4c8e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:35:36.561116 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:36.559477    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:35:36.597862 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:36.597819    6059 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g" podUID="83ca6484-7e38-4782-b3a9-57b58bdc3920" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": context deadline exceeded"
Nov 03 17:35:36.771429 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:36.771395    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:36.834482 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:36.799474    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g" podUID="83ca6484-7e38-4782-b3a9-57b58bdc3920" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:35:36.966382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:36.964819    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v" podUID="2bbc9082-b2da-4198-8a8a-b6ab26b5011e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 17:35:36.984693 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:36.984657    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:35:37.098446 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:37.072573    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl" podUID="c194a4be-0499-419f-a4a1-910dbaafd303" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 17:35:37.098650 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:37.088265    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:35:37.099480 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:37.099452    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:35:37.161381 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:35:37.158874    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:35:37.215020 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:37.214991    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:35:37.267198 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:37.267159    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:37.394330 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:37.393931    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n" podUID="d6c1263b-e648-4f10-af7d-86397d6b207a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:35:37.524520 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:37.524464    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:35:37.524706 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:37.524540    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:35:37.621555 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:37.621513    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7484-0651f2"]
Nov 03 17:35:37.621784 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:37.621742    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7484-0651f2" podUID="7f12033e-64f7-4297-bfed-85da58f409cb" containerName="test-curl-7484-0651f2" containerID="cri-o://1171a48bd1c5871f8bad6446dfbadcee425416ce914e4d0a24415e6a270be271" gracePeriod=2
Nov 03 17:35:37.682691 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:37.682534    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7484-0651f2"]
Nov 03 17:35:37.854379 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:35:37.849933    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-0651f2\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="7f12033e-64f7-4297-bfed-85da58f409cb" pod="opendatahub/test-curl-7484-0651f2"
Nov 03 17:35:39.412145 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:35:39.412102    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-0651f2\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="7f12033e-64f7-4297-bfed-85da58f409cb" pod="opendatahub/test-curl-7484-0651f2"
Nov 03 17:35:39.893037 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:39.893009    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-0651f2"
Nov 03 17:35:39.923391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:39.923291    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f12033e-64f7-4297-bfed-85da58f409cb-kube-api-access-w7v42\" (UniqueName: \"kubernetes.io/projected/7f12033e-64f7-4297-bfed-85da58f409cb-kube-api-access-w7v42\") pod \"7f12033e-64f7-4297-bfed-85da58f409cb\" (UID: \"7f12033e-64f7-4297-bfed-85da58f409cb\") "
Nov 03 17:35:39.925488 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:39.925452    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f12033e-64f7-4297-bfed-85da58f409cb-kube-api-access-w7v42" pod "7f12033e-64f7-4297-bfed-85da58f409cb" (UID: "7f12033e-64f7-4297-bfed-85da58f409cb"). InnerVolumeSpecName "kube-api-access-w7v42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:35:40.025791 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:40.025745    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7v42\" (UniqueName: \"kubernetes.io/projected/7f12033e-64f7-4297-bfed-85da58f409cb-kube-api-access-w7v42\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:35:40.575153 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:40.575119    6059 generic.go:356] "Generic (PLEG): container finished" podID="7f12033e-64f7-4297-bfed-85da58f409cb" containerID="1171a48bd1c5871f8bad6446dfbadcee425416ce914e4d0a24415e6a270be271" exitCode=137
Nov 03 17:35:40.575584 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:40.575184    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-0651f2"
Nov 03 17:35:40.575584 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:40.575202    6059 scope.go:122] "RemoveContainer" containerID="1171a48bd1c5871f8bad6446dfbadcee425416ce914e4d0a24415e6a270be271"
Nov 03 17:35:40.590644 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:40.590616    6059 scope.go:122] "RemoveContainer" containerID="1171a48bd1c5871f8bad6446dfbadcee425416ce914e4d0a24415e6a270be271"
Nov 03 17:35:40.590940 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:35:40.590922    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1171a48bd1c5871f8bad6446dfbadcee425416ce914e4d0a24415e6a270be271\": container with ID starting with 1171a48bd1c5871f8bad6446dfbadcee425416ce914e4d0a24415e6a270be271 not found: ID does not exist" containerID="1171a48bd1c5871f8bad6446dfbadcee425416ce914e4d0a24415e6a270be271"
Nov 03 17:35:40.590995 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:40.590950    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1171a48bd1c5871f8bad6446dfbadcee425416ce914e4d0a24415e6a270be271"} err="failed to get container status \"1171a48bd1c5871f8bad6446dfbadcee425416ce914e4d0a24415e6a270be271\": rpc error: code = NotFound desc = could not find container \"1171a48bd1c5871f8bad6446dfbadcee425416ce914e4d0a24415e6a270be271\": container with ID starting with 1171a48bd1c5871f8bad6446dfbadcee425416ce914e4d0a24415e6a270be271 not found: ID does not exist"
Nov 03 17:35:40.637403 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:35:40.637295    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-0651f2\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="7f12033e-64f7-4297-bfed-85da58f409cb" pod="opendatahub/test-curl-7484-0651f2"
Nov 03 17:35:41.336042 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:41.336002    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f12033e-64f7-4297-bfed-85da58f409cb" path="/var/lib/kubelet/pods/7f12033e-64f7-4297-bfed-85da58f409cb/volumes"
Nov 03 17:35:41.885448 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:41.885403    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:43.602105 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:43.602061    6059 generic.go:356] "Generic (PLEG): container finished" podID="2861a97b-ab99-4cfc-8a5c-bd2408972572" containerID="503cfd967c23e1ec0f4e2fd0cb8b59adcee7f840f752e7aa337c5119f3856750" exitCode=0
Nov 03 17:35:43.602632 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:43.602132    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l" event={"ID":"2861a97b-ab99-4cfc-8a5c-bd2408972572","Type":"ContainerDied","Data":"503cfd967c23e1ec0f4e2fd0cb8b59adcee7f840f752e7aa337c5119f3856750"}
Nov 03 17:35:43.602632 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:43.602568    6059 scope.go:122] "RemoveContainer" containerID="503cfd967c23e1ec0f4e2fd0cb8b59adcee7f840f752e7aa337c5119f3856750"
Nov 03 17:35:43.604019 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:43.603991    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:35:43.604095 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:43.604057    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:35:43.604095 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:43.604088    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:35:44.611192 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:44.611156    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l" event={"ID":"2861a97b-ab99-4cfc-8a5c-bd2408972572","Type":"ContainerStarted","Data":"d2aabfa1ba75eb345ceed2e607fe8a7d1149b10cca7651538e77acf962c828b4"}
Nov 03 17:35:44.758386 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:44.757118    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc"]
Nov 03 17:35:44.758386 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:44.758085    6059 cpu_manager.go:414] "containerMap: removing container" podUID="7f12033e-64f7-4297-bfed-85da58f409cb" containerName="test-curl-7484-0651f2"
Nov 03 17:35:44.758386 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:44.758106    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f12033e-64f7-4297-bfed-85da58f409cb" containerName="test-curl-7484-0651f2"
Nov 03 17:35:44.758386 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:44.758243    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f12033e-64f7-4297-bfed-85da58f409cb" containerName="test-curl-7484-0651f2"
Nov 03 17:35:44.776694 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:44.774850    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc"]
Nov 03 17:35:44.776694 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:44.775024    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc"
Nov 03 17:35:44.792254 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:44.792230    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:35:44.792422 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:44.792402    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 17:35:44.792469 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:44.792448    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l" podUID="2861a97b-ab99-4cfc-8a5c-bd2408972572" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 17:35:44.853380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:44.852722    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:35:44.853380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:44.852831    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:35:44.853380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:44.852950    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:35:44.853380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:44.853066    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:35:44.887168 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:44.887081    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204564fa-b79b-4040-adfa-f732c266c065-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70ffcd57-lmcgc\" (UID: \"204564fa-b79b-4040-adfa-f732c266c065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc"
Nov 03 17:35:44.887168 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:44.887144    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtlnx\" (UniqueName: \"kubernetes.io/projected/204564fa-b79b-4040-adfa-f732c266c065-kube-api-access-dtlnx\") pod \"maas-api-revoke-sub-70ffcd57-lmcgc\" (UID: \"204564fa-b79b-4040-adfa-f732c266c065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc"
Nov 03 17:35:44.887342 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:44.887226    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204564fa-b79b-4040-adfa-f732c266c065-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70ffcd57-lmcgc\" (UID: \"204564fa-b79b-4040-adfa-f732c266c065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc"
Nov 03 17:35:44.989179 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:44.989140    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtlnx\" (UniqueName: \"kubernetes.io/projected/204564fa-b79b-4040-adfa-f732c266c065-kube-api-access-dtlnx\") pod \"maas-api-revoke-sub-70ffcd57-lmcgc\" (UID: \"204564fa-b79b-4040-adfa-f732c266c065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc"
Nov 03 17:35:44.989418 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:44.989200    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204564fa-b79b-4040-adfa-f732c266c065-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70ffcd57-lmcgc\" (UID: \"204564fa-b79b-4040-adfa-f732c266c065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc"
Nov 03 17:35:44.989505 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:44.989473    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204564fa-b79b-4040-adfa-f732c266c065-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70ffcd57-lmcgc\" (UID: \"204564fa-b79b-4040-adfa-f732c266c065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc"
Nov 03 17:35:44.990071 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:44.990048    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204564fa-b79b-4040-adfa-f732c266c065-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70ffcd57-lmcgc\" (UID: \"204564fa-b79b-4040-adfa-f732c266c065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc"
Nov 03 17:35:44.991673 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:44.991644    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204564fa-b79b-4040-adfa-f732c266c065-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70ffcd57-lmcgc\" (UID: \"204564fa-b79b-4040-adfa-f732c266c065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc"
Nov 03 17:35:45.057160 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:45.057131    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtlnx\" (UniqueName: \"kubernetes.io/projected/204564fa-b79b-4040-adfa-f732c266c065-kube-api-access-dtlnx\") pod \"maas-api-revoke-sub-70ffcd57-lmcgc\" (UID: \"204564fa-b79b-4040-adfa-f732c266c065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc"
Nov 03 17:35:45.090140 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:45.090101    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc"
Nov 03 17:35:45.490245 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:35:45.490194    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204564fa_b79b_4040_adfa_f732c266c065.slice/crio-1e29c0403f2226242bbeedab321d96208a9fdbe2868d882902ac4348300ad6e3 WatchSource:0}: Error finding container 1e29c0403f2226242bbeedab321d96208a9fdbe2868d882902ac4348300ad6e3: Status 404 returned error can't find the container with id 1e29c0403f2226242bbeedab321d96208a9fdbe2868d882902ac4348300ad6e3
Nov 03 17:35:45.493284 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:45.493260    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc"]
Nov 03 17:35:45.619517 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:45.619476    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc" event={"ID":"204564fa-b79b-4040-adfa-f732c266c065","Type":"ContainerStarted","Data":"1e29c0403f2226242bbeedab321d96208a9fdbe2868d882902ac4348300ad6e3"}
Nov 03 17:35:45.793109 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:45.793033    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 17:35:45.793109 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:45.793086    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l" podUID="2861a97b-ab99-4cfc-8a5c-bd2408972572" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 17:35:46.457377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:46.456881    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:46.628488 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:46.628445    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc" event={"ID":"204564fa-b79b-4040-adfa-f732c266c065","Type":"ContainerStarted","Data":"2043b288d95f82bb7c7dd4dcdaf2ba638f633b4a94a2f5aab0b33c550a9ced58"}
Nov 03 17:35:46.756681 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:46.756572    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc" podStartSLOduration=2.75655764 podStartE2EDuration="2.75655764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:35:46.754709689 +0000 UTC m=+1087.552422472" watchObservedRunningTime="2026-11-03 17:35:46.75655764 +0000 UTC m=+1087.554270418"
Nov 03 17:35:46.793262 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:46.793225    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 17:35:46.793467 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:46.793294    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l" podUID="2861a97b-ab99-4cfc-8a5c-bd2408972572" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 17:35:47.659326 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:47.659229    6059 generic.go:356] "Generic (PLEG): container finished" podID="204564fa-b79b-4040-adfa-f732c266c065" containerID="2043b288d95f82bb7c7dd4dcdaf2ba638f633b4a94a2f5aab0b33c550a9ced58" exitCode=0
Nov 03 17:35:47.659829 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:47.659445    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc" event={"ID":"204564fa-b79b-4040-adfa-f732c266c065","Type":"ContainerDied","Data":"2043b288d95f82bb7c7dd4dcdaf2ba638f633b4a94a2f5aab0b33c550a9ced58"}
Nov 03 17:35:47.793459 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:47.793416    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 17:35:47.793639 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:47.793481    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l" podUID="2861a97b-ab99-4cfc-8a5c-bd2408972572" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 17:35:48.793671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:48.793163    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 17:35:48.793671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:48.793251    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l" podUID="2861a97b-ab99-4cfc-8a5c-bd2408972572" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 17:35:48.868340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:48.868307    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc"
Nov 03 17:35:48.970061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:48.970032    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/204564fa-b79b-4040-adfa-f732c266c065-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204564fa-b79b-4040-adfa-f732c266c065-maas-api-xks-ca\") pod \"204564fa-b79b-4040-adfa-f732c266c065\" (UID: \"204564fa-b79b-4040-adfa-f732c266c065\") "
Nov 03 17:35:48.970259 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:48.970242    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204564fa-b79b-4040-adfa-f732c266c065-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204564fa-b79b-4040-adfa-f732c266c065-maas-api-openshift-ca\") pod \"204564fa-b79b-4040-adfa-f732c266c065\" (UID: \"204564fa-b79b-4040-adfa-f732c266c065\") "
Nov 03 17:35:48.970308 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:48.970268    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204564fa-b79b-4040-adfa-f732c266c065-kube-api-access-dtlnx\" (UniqueName: \"kubernetes.io/projected/204564fa-b79b-4040-adfa-f732c266c065-kube-api-access-dtlnx\") pod \"204564fa-b79b-4040-adfa-f732c266c065\" (UID: \"204564fa-b79b-4040-adfa-f732c266c065\") "
Nov 03 17:35:48.970734 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:48.970701    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204564fa-b79b-4040-adfa-f732c266c065-maas-api-openshift-ca" pod "204564fa-b79b-4040-adfa-f732c266c065" (UID: "204564fa-b79b-4040-adfa-f732c266c065"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:35:48.972243 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:48.972208    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/204564fa-b79b-4040-adfa-f732c266c065-maas-api-xks-ca" pod "204564fa-b79b-4040-adfa-f732c266c065" (UID: "204564fa-b79b-4040-adfa-f732c266c065"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:35:48.972388 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:48.972312    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204564fa-b79b-4040-adfa-f732c266c065-kube-api-access-dtlnx" pod "204564fa-b79b-4040-adfa-f732c266c065" (UID: "204564fa-b79b-4040-adfa-f732c266c065"). InnerVolumeSpecName "kube-api-access-dtlnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:35:49.071914 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:49.071872    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204564fa-b79b-4040-adfa-f732c266c065-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:35:49.071914 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:49.071904    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204564fa-b79b-4040-adfa-f732c266c065-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:35:49.071914 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:49.071916    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtlnx\" (UniqueName: \"kubernetes.io/projected/204564fa-b79b-4040-adfa-f732c266c065-kube-api-access-dtlnx\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:35:49.677209 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:49.677128    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc"
Nov 03 17:35:49.677209 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:49.677166    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc" event={"ID":"204564fa-b79b-4040-adfa-f732c266c065","Type":"ContainerDied","Data":"1e29c0403f2226242bbeedab321d96208a9fdbe2868d882902ac4348300ad6e3"}
Nov 03 17:35:49.677413 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:49.677216    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e29c0403f2226242bbeedab321d96208a9fdbe2868d882902ac4348300ad6e3"
Nov 03 17:35:49.796475 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:49.796443    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:35:49.796947 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:49.796850    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:35:49.797467 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:49.797446    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 17:35:51.152021 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:51.151979    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:35:55.749941 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:35:55.746192    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:00.157433 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:00.157398    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44878b2a-d6mqr"]
Nov 03 17:36:00.158863 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:00.158823    6059 cpu_manager.go:414] "containerMap: removing container" podUID="204564fa-b79b-4040-adfa-f732c266c065" containerName="revoke-keys"
Nov 03 17:36:00.158863 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:00.158855    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204564fa-b79b-4040-adfa-f732c266c065" containerName="revoke-keys"
Nov 03 17:36:00.159058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:00.159045    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="204564fa-b79b-4040-adfa-f732c266c065" containerName="revoke-keys"
Nov 03 17:36:00.169342 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:00.169278    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44878b2a-d6mqr"
Nov 03 17:36:00.245381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:00.244711    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44878b2a-d6mqr"]
Nov 03 17:36:00.350383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:00.273605    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:36:00.350383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:00.274288    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:36:00.350383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:00.274329    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:36:00.350383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:00.274424    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:36:00.350383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:00.304436    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9edabf3d-035f-42cd-8efd-c2150fd3921a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44878b2a-d6mqr\" (UID: \"9edabf3d-035f-42cd-8efd-c2150fd3921a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44878b2a-d6mqr"
Nov 03 17:36:00.350383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:00.304642    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9edabf3d-035f-42cd-8efd-c2150fd3921a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44878b2a-d6mqr\" (UID: \"9edabf3d-035f-42cd-8efd-c2150fd3921a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44878b2a-d6mqr"
Nov 03 17:36:00.350383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:00.304759    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gq7t\" (UniqueName: \"kubernetes.io/projected/9edabf3d-035f-42cd-8efd-c2150fd3921a-kube-api-access-4gq7t\") pod \"maas-api-revoke-sub-44878b2a-d6mqr\" (UID: \"9edabf3d-035f-42cd-8efd-c2150fd3921a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44878b2a-d6mqr"
Nov 03 17:36:00.434459 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:00.420419    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gq7t\" (UniqueName: \"kubernetes.io/projected/9edabf3d-035f-42cd-8efd-c2150fd3921a-kube-api-access-4gq7t\") pod \"maas-api-revoke-sub-44878b2a-d6mqr\" (UID: \"9edabf3d-035f-42cd-8efd-c2150fd3921a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44878b2a-d6mqr"
Nov 03 17:36:00.435627 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:00.435589    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9edabf3d-035f-42cd-8efd-c2150fd3921a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44878b2a-d6mqr\" (UID: \"9edabf3d-035f-42cd-8efd-c2150fd3921a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44878b2a-d6mqr"
Nov 03 17:36:00.436838 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:00.436812    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9edabf3d-035f-42cd-8efd-c2150fd3921a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44878b2a-d6mqr\" (UID: \"9edabf3d-035f-42cd-8efd-c2150fd3921a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44878b2a-d6mqr"
Nov 03 17:36:00.442516 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:00.436665    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9edabf3d-035f-42cd-8efd-c2150fd3921a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44878b2a-d6mqr\" (UID: \"9edabf3d-035f-42cd-8efd-c2150fd3921a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44878b2a-d6mqr"
Nov 03 17:36:00.442734 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:00.435301    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:00.511419 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:00.507846    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9edabf3d-035f-42cd-8efd-c2150fd3921a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44878b2a-d6mqr\" (UID: \"9edabf3d-035f-42cd-8efd-c2150fd3921a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44878b2a-d6mqr"
Nov 03 17:36:00.543113 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:00.543070    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gq7t\" (UniqueName: \"kubernetes.io/projected/9edabf3d-035f-42cd-8efd-c2150fd3921a-kube-api-access-4gq7t\") pod \"maas-api-revoke-sub-44878b2a-d6mqr\" (UID: \"9edabf3d-035f-42cd-8efd-c2150fd3921a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44878b2a-d6mqr"
Nov 03 17:36:00.790384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:00.789442    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44878b2a-d6mqr"
Nov 03 17:36:01.706268 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:01.706226    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44878b2a-d6mqr"]
Nov 03 17:36:01.853488 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:01.853443    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44878b2a-d6mqr" event={"ID":"9edabf3d-035f-42cd-8efd-c2150fd3921a","Type":"ContainerStarted","Data":"33606ec7a050ce09a6d0d1abb8733f171094def7cfd1c0b0e64611a58cf7b42a"}
Nov 03 17:36:02.862267 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:02.862171    6059 generic.go:356] "Generic (PLEG): container finished" podID="9edabf3d-035f-42cd-8efd-c2150fd3921a" containerID="f5a0c60adf16a7e62153b2819e30f21a465bc11fd3377651f9a0d35f89ddb95c" exitCode=0
Nov 03 17:36:02.862267 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:02.862221    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44878b2a-d6mqr" event={"ID":"9edabf3d-035f-42cd-8efd-c2150fd3921a","Type":"ContainerDied","Data":"f5a0c60adf16a7e62153b2819e30f21a465bc11fd3377651f9a0d35f89ddb95c"}
Nov 03 17:36:04.010583 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:04.010558    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44878b2a-d6mqr"
Nov 03 17:36:04.055579 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:04.055542    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9edabf3d-035f-42cd-8efd-c2150fd3921a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9edabf3d-035f-42cd-8efd-c2150fd3921a-maas-api-openshift-ca\") pod \"9edabf3d-035f-42cd-8efd-c2150fd3921a\" (UID: \"9edabf3d-035f-42cd-8efd-c2150fd3921a\") "
Nov 03 17:36:04.055756 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:04.055711    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9edabf3d-035f-42cd-8efd-c2150fd3921a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9edabf3d-035f-42cd-8efd-c2150fd3921a-maas-api-xks-ca\") pod \"9edabf3d-035f-42cd-8efd-c2150fd3921a\" (UID: \"9edabf3d-035f-42cd-8efd-c2150fd3921a\") "
Nov 03 17:36:04.055813 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:04.055803    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9edabf3d-035f-42cd-8efd-c2150fd3921a-kube-api-access-4gq7t\" (UniqueName: \"kubernetes.io/projected/9edabf3d-035f-42cd-8efd-c2150fd3921a-kube-api-access-4gq7t\") pod \"9edabf3d-035f-42cd-8efd-c2150fd3921a\" (UID: \"9edabf3d-035f-42cd-8efd-c2150fd3921a\") "
Nov 03 17:36:04.055919 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:04.055890    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9edabf3d-035f-42cd-8efd-c2150fd3921a-maas-api-openshift-ca" pod "9edabf3d-035f-42cd-8efd-c2150fd3921a" (UID: "9edabf3d-035f-42cd-8efd-c2150fd3921a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:36:04.056093 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:04.056079    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9edabf3d-035f-42cd-8efd-c2150fd3921a-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:36:04.057761 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:04.057738    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9edabf3d-035f-42cd-8efd-c2150fd3921a-maas-api-xks-ca" pod "9edabf3d-035f-42cd-8efd-c2150fd3921a" (UID: "9edabf3d-035f-42cd-8efd-c2150fd3921a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:36:04.058106 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:04.058078    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9edabf3d-035f-42cd-8efd-c2150fd3921a-kube-api-access-4gq7t" pod "9edabf3d-035f-42cd-8efd-c2150fd3921a" (UID: "9edabf3d-035f-42cd-8efd-c2150fd3921a"). InnerVolumeSpecName "kube-api-access-4gq7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:36:04.157116 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:04.157015    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9edabf3d-035f-42cd-8efd-c2150fd3921a-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:36:04.157116 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:04.157059    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gq7t\" (UniqueName: \"kubernetes.io/projected/9edabf3d-035f-42cd-8efd-c2150fd3921a-kube-api-access-4gq7t\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:36:04.878160 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:04.878126    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44878b2a-d6mqr"
Nov 03 17:36:04.878160 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:04.878150    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44878b2a-d6mqr" event={"ID":"9edabf3d-035f-42cd-8efd-c2150fd3921a","Type":"ContainerDied","Data":"33606ec7a050ce09a6d0d1abb8733f171094def7cfd1c0b0e64611a58cf7b42a"}
Nov 03 17:36:04.878399 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:04.878195    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33606ec7a050ce09a6d0d1abb8733f171094def7cfd1c0b0e64611a58cf7b42a"
Nov 03 17:36:05.078597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:05.078552    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:06.051894 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:06.051861    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a507cc28-l2n25"]
Nov 03 17:36:06.103202 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:06.103169    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a507cc28-l2n25"]
Nov 03 17:36:06.861618 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:36:06.861578    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:36:07.336289 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.336251    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61a3bf97-aa63-49f9-b2dd-75414055f4ab" path="/var/lib/kubelet/pods/61a3bf97-aa63-49f9-b2dd-75414055f4ab/volumes"
Nov 03 17:36:07.336684 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.336635    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"]
Nov 03 17:36:07.337172 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.337157    6059 cpu_manager.go:414] "containerMap: removing container" podUID="9edabf3d-035f-42cd-8efd-c2150fd3921a" containerName="revoke-keys"
Nov 03 17:36:07.337209 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.337174    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9edabf3d-035f-42cd-8efd-c2150fd3921a" containerName="revoke-keys"
Nov 03 17:36:07.337275 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.337266    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9edabf3d-035f-42cd-8efd-c2150fd3921a" containerName="revoke-keys"
Nov 03 17:36:07.341152 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.341136    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:07.393848 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.393816    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"]
Nov 03 17:36:07.400178 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.400149    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-home\") pod \"rate-test-model-af9e3c-kserve-68fd55f97-pjt6w\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:07.400299 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.400190    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6120f11d-14c8-4666-a201-469a5bca37de-tls-certs\") pod \"rate-test-model-af9e3c-kserve-68fd55f97-pjt6w\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:07.400381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.400346    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-model-cache\") pod \"rate-test-model-af9e3c-kserve-68fd55f97-pjt6w\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:07.400492 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.400465    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-tmp-dir\") pod \"rate-test-model-af9e3c-kserve-68fd55f97-pjt6w\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:07.400553 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.400539    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs2ts\" (UniqueName: \"kubernetes.io/projected/6120f11d-14c8-4666-a201-469a5bca37de-kube-api-access-bs2ts\") pod \"rate-test-model-af9e3c-kserve-68fd55f97-pjt6w\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:07.400615 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.400596    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-dshm\") pod \"rate-test-model-af9e3c-kserve-68fd55f97-pjt6w\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:07.441552 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.441527    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-af9e3c\"/\"kube-root-ca.crt\""
Nov 03 17:36:07.441715 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.441702    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-af9e3c\"/\"openshift-service-ca.crt\""
Nov 03 17:36:07.441770 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.441750    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-af9e3c\"/\"rate-test-model-af9e3c-kserve-self-signed-certs\""
Nov 03 17:36:07.447500 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.447483    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-af9e3c\"/\"default-dockercfg-bl29d\""
Nov 03 17:36:07.458884 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.458857    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:07.501901 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.501843    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-dshm\") pod \"rate-test-model-af9e3c-kserve-68fd55f97-pjt6w\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:07.502126 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.501982    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-home\") pod \"rate-test-model-af9e3c-kserve-68fd55f97-pjt6w\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:07.502126 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.502009    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6120f11d-14c8-4666-a201-469a5bca37de-tls-certs\") pod \"rate-test-model-af9e3c-kserve-68fd55f97-pjt6w\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:07.502243 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.502152    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-model-cache\") pod \"rate-test-model-af9e3c-kserve-68fd55f97-pjt6w\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:07.502243 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.502188    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-tmp-dir\") pod \"rate-test-model-af9e3c-kserve-68fd55f97-pjt6w\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:07.502349 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.502240    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs2ts\" (UniqueName: \"kubernetes.io/projected/6120f11d-14c8-4666-a201-469a5bca37de-kube-api-access-bs2ts\") pod \"rate-test-model-af9e3c-kserve-68fd55f97-pjt6w\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:07.502465 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.502447    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-home\") pod \"rate-test-model-af9e3c-kserve-68fd55f97-pjt6w\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:07.502552 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.502526    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-model-cache\") pod \"rate-test-model-af9e3c-kserve-68fd55f97-pjt6w\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:07.502611 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.502593    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-tmp-dir\") pod \"rate-test-model-af9e3c-kserve-68fd55f97-pjt6w\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:07.504860 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.504811    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6120f11d-14c8-4666-a201-469a5bca37de-tls-certs\") pod \"rate-test-model-af9e3c-kserve-68fd55f97-pjt6w\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:07.505062 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.504887    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-dshm\") pod \"rate-test-model-af9e3c-kserve-68fd55f97-pjt6w\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:07.566077 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.566050    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs2ts\" (UniqueName: \"kubernetes.io/projected/6120f11d-14c8-4666-a201-469a5bca37de-kube-api-access-bs2ts\") pod \"rate-test-model-af9e3c-kserve-68fd55f97-pjt6w\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:07.653475 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:07.653349    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:08.029554 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:36:08.029518    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6120f11d_14c8_4666_a201_469a5bca37de.slice/crio-fdc34b98d99ddabbcd972d82ce372f36e900d7d65646484042a65199fb85e114 WatchSource:0}: Error finding container fdc34b98d99ddabbcd972d82ce372f36e900d7d65646484042a65199fb85e114: Status 404 returned error can't find the container with id fdc34b98d99ddabbcd972d82ce372f36e900d7d65646484042a65199fb85e114
Nov 03 17:36:08.032339 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:08.032313    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"]
Nov 03 17:36:08.908694 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:08.908651    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w" event={"ID":"6120f11d-14c8-4666-a201-469a5bca37de","Type":"ContainerStarted","Data":"a380062a162c02aece834579a21a76edab1d371f891cce90f4d0f1657a7aa76d"}
Nov 03 17:36:08.909073 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:08.908707    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w" event={"ID":"6120f11d-14c8-4666-a201-469a5bca37de","Type":"ContainerStarted","Data":"fdc34b98d99ddabbcd972d82ce372f36e900d7d65646484042a65199fb85e114"}
Nov 03 17:36:09.031428 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:09.031368    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w" podStartSLOduration=2.031339 podStartE2EDuration="2.031339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:36:09.029470131 +0000 UTC m=+1109.827182912" watchObservedRunningTime="2026-11-03 17:36:09.031339 +0000 UTC m=+1109.829051780"
Nov 03 17:36:09.653625 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:09.653591    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:09.873039 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:09.872936    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:13.918981 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:13.918949    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:14.834526 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:14.834492    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:15.169005 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:15.168928    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:15.347234 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:15.347197    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:15.544506 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:15.544472    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:15.832841 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:15.832747    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:16.033606 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:16.033572    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:16.911523 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:16.911489    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3222aef-twttc"]
Nov 03 17:36:16.915588 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:16.915567    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3222aef-twttc"
Nov 03 17:36:16.977423 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:16.977373    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3222aef-twttc"]
Nov 03 17:36:17.023170 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:17.023135    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:36:17.023170 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:17.023150    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:36:17.023170 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:17.023166    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:36:17.023508 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:17.023191    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:36:17.061750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:17.061719    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe504ece-3403-4332-9029-12f5b7ca4671-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3222aef-twttc\" (UID: \"fe504ece-3403-4332-9029-12f5b7ca4671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3222aef-twttc"
Nov 03 17:36:17.061917 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:17.061766    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe504ece-3403-4332-9029-12f5b7ca4671-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3222aef-twttc\" (UID: \"fe504ece-3403-4332-9029-12f5b7ca4671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3222aef-twttc"
Nov 03 17:36:17.061917 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:17.061857    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk8wt\" (UniqueName: \"kubernetes.io/projected/fe504ece-3403-4332-9029-12f5b7ca4671-kube-api-access-dk8wt\") pod \"maas-api-revoke-sub-f3222aef-twttc\" (UID: \"fe504ece-3403-4332-9029-12f5b7ca4671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3222aef-twttc"
Nov 03 17:36:17.107298 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:17.107262    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:17.163305 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:17.163199    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk8wt\" (UniqueName: \"kubernetes.io/projected/fe504ece-3403-4332-9029-12f5b7ca4671-kube-api-access-dk8wt\") pod \"maas-api-revoke-sub-f3222aef-twttc\" (UID: \"fe504ece-3403-4332-9029-12f5b7ca4671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3222aef-twttc"
Nov 03 17:36:17.163527 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:17.163505    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe504ece-3403-4332-9029-12f5b7ca4671-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3222aef-twttc\" (UID: \"fe504ece-3403-4332-9029-12f5b7ca4671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3222aef-twttc"
Nov 03 17:36:17.163637 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:17.163598    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe504ece-3403-4332-9029-12f5b7ca4671-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3222aef-twttc\" (UID: \"fe504ece-3403-4332-9029-12f5b7ca4671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3222aef-twttc"
Nov 03 17:36:17.164320 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:17.164298    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe504ece-3403-4332-9029-12f5b7ca4671-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3222aef-twttc\" (UID: \"fe504ece-3403-4332-9029-12f5b7ca4671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3222aef-twttc"
Nov 03 17:36:17.165901 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:17.165873    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe504ece-3403-4332-9029-12f5b7ca4671-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3222aef-twttc\" (UID: \"fe504ece-3403-4332-9029-12f5b7ca4671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3222aef-twttc"
Nov 03 17:36:17.229558 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:17.229529    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk8wt\" (UniqueName: \"kubernetes.io/projected/fe504ece-3403-4332-9029-12f5b7ca4671-kube-api-access-dk8wt\") pod \"maas-api-revoke-sub-f3222aef-twttc\" (UID: \"fe504ece-3403-4332-9029-12f5b7ca4671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3222aef-twttc"
Nov 03 17:36:17.230287 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:17.230263    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3222aef-twttc"
Nov 03 17:36:17.311083 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:17.310053    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:17.542432 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:17.542268    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:17.639909 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:36:17.639881    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe504ece_3403_4332_9029_12f5b7ca4671.slice/crio-ded9c66a63fdd0f5ee7a67472b486d0c2e88a21a0b8d91a9be4d16779fdb2511 WatchSource:0}: Error finding container ded9c66a63fdd0f5ee7a67472b486d0c2e88a21a0b8d91a9be4d16779fdb2511: Status 404 returned error can't find the container with id ded9c66a63fdd0f5ee7a67472b486d0c2e88a21a0b8d91a9be4d16779fdb2511
Nov 03 17:36:17.643735 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:17.643711    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3222aef-twttc"]
Nov 03 17:36:17.653892 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:17.653870    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:17.671691 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:17.671661    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:18.032598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:18.032557    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3222aef-twttc" event={"ID":"fe504ece-3403-4332-9029-12f5b7ca4671","Type":"ContainerStarted","Data":"ded9c66a63fdd0f5ee7a67472b486d0c2e88a21a0b8d91a9be4d16779fdb2511"}
Nov 03 17:36:18.045025 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:18.045002    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:36:19.041700 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:19.041660    6059 generic.go:356] "Generic (PLEG): container finished" podID="fe504ece-3403-4332-9029-12f5b7ca4671" containerID="13551372c0c7bfa6c25d13166e6b266fcafe4b9fb60b9097cffa346d6c4f7548" exitCode=0
Nov 03 17:36:19.042159 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:19.041740    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3222aef-twttc" event={"ID":"fe504ece-3403-4332-9029-12f5b7ca4671","Type":"ContainerDied","Data":"13551372c0c7bfa6c25d13166e6b266fcafe4b9fb60b9097cffa346d6c4f7548"}
Nov 03 17:36:19.719858 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:19.719823    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:20.206649 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:20.206624    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3222aef-twttc"
Nov 03 17:36:20.312593 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:20.312507    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe504ece-3403-4332-9029-12f5b7ca4671-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe504ece-3403-4332-9029-12f5b7ca4671-maas-api-openshift-ca\") pod \"fe504ece-3403-4332-9029-12f5b7ca4671\" (UID: \"fe504ece-3403-4332-9029-12f5b7ca4671\") "
Nov 03 17:36:20.312593 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:20.312558    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe504ece-3403-4332-9029-12f5b7ca4671-kube-api-access-dk8wt\" (UniqueName: \"kubernetes.io/projected/fe504ece-3403-4332-9029-12f5b7ca4671-kube-api-access-dk8wt\") pod \"fe504ece-3403-4332-9029-12f5b7ca4671\" (UID: \"fe504ece-3403-4332-9029-12f5b7ca4671\") "
Nov 03 17:36:20.312841 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:20.312607    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe504ece-3403-4332-9029-12f5b7ca4671-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe504ece-3403-4332-9029-12f5b7ca4671-maas-api-xks-ca\") pod \"fe504ece-3403-4332-9029-12f5b7ca4671\" (UID: \"fe504ece-3403-4332-9029-12f5b7ca4671\") "
Nov 03 17:36:20.312933 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:20.312901    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe504ece-3403-4332-9029-12f5b7ca4671-maas-api-openshift-ca" pod "fe504ece-3403-4332-9029-12f5b7ca4671" (UID: "fe504ece-3403-4332-9029-12f5b7ca4671"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:36:20.313096 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:20.313080    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe504ece-3403-4332-9029-12f5b7ca4671-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:36:20.314662 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:20.314629    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe504ece-3403-4332-9029-12f5b7ca4671-maas-api-xks-ca" pod "fe504ece-3403-4332-9029-12f5b7ca4671" (UID: "fe504ece-3403-4332-9029-12f5b7ca4671"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:36:20.314771 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:20.314711    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe504ece-3403-4332-9029-12f5b7ca4671-kube-api-access-dk8wt" pod "fe504ece-3403-4332-9029-12f5b7ca4671" (UID: "fe504ece-3403-4332-9029-12f5b7ca4671"). InnerVolumeSpecName "kube-api-access-dk8wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:36:20.414433 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:20.414392    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk8wt\" (UniqueName: \"kubernetes.io/projected/fe504ece-3403-4332-9029-12f5b7ca4671-kube-api-access-dk8wt\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:36:20.414433 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:20.414433    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe504ece-3403-4332-9029-12f5b7ca4671-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:36:20.924200 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:20.924154    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:21.073209 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.073146    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3222aef-twttc" event={"ID":"fe504ece-3403-4332-9029-12f5b7ca4671","Type":"ContainerDied","Data":"ded9c66a63fdd0f5ee7a67472b486d0c2e88a21a0b8d91a9be4d16779fdb2511"}
Nov 03 17:36:21.073209 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.073211    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ded9c66a63fdd0f5ee7a67472b486d0c2e88a21a0b8d91a9be4d16779fdb2511"
Nov 03 17:36:21.073649 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.073286    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3222aef-twttc"
Nov 03 17:36:21.420678 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.420643    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"]
Nov 03 17:36:21.421266 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.421251    6059 cpu_manager.go:414] "containerMap: removing container" podUID="fe504ece-3403-4332-9029-12f5b7ca4671" containerName="revoke-keys"
Nov 03 17:36:21.421266 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.421266    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe504ece-3403-4332-9029-12f5b7ca4671" containerName="revoke-keys"
Nov 03 17:36:21.421381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.421369    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe504ece-3403-4332-9029-12f5b7ca4671" containerName="revoke-keys"
Nov 03 17:36:21.424986 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.424969    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"
Nov 03 17:36:21.480373 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.480325    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-bf3839ce3fb2\""
Nov 03 17:36:21.480730 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.480702    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"payload-processing-external-model-e2e--b603cb9e-dockercfg-78csf\""
Nov 03 17:36:21.527340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.527301    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bfc7b061-6dbc-4368-91e9-911e982ae50e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq\" (UID: \"bfc7b061-6dbc-4368-91e9-911e982ae50e\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"
Nov 03 17:36:21.527573 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.527439    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bfc7b061-6dbc-4368-91e9-911e982ae50e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq\" (UID: \"bfc7b061-6dbc-4368-91e9-911e982ae50e\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"
Nov 03 17:36:21.527573 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.527462    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc7b061-6dbc-4368-91e9-911e982ae50e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq\" (UID: \"bfc7b061-6dbc-4368-91e9-911e982ae50e\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"
Nov 03 17:36:21.527573 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.527478    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86dtt\" (UniqueName: \"kubernetes.io/projected/bfc7b061-6dbc-4368-91e9-911e982ae50e-kube-api-access-86dtt\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq\" (UID: \"bfc7b061-6dbc-4368-91e9-911e982ae50e\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"
Nov 03 17:36:21.532223 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.532196    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"routing-overlay\""
Nov 03 17:36:21.532381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.532329    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"openshift-service-ca.crt\""
Nov 03 17:36:21.532431 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.532352    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"kube-root-ca.crt\""
Nov 03 17:36:21.532431 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.532386    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-8c9b0f\""
Nov 03 17:36:21.536005 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.535984    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"]
Nov 03 17:36:21.628534 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.628495    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bfc7b061-6dbc-4368-91e9-911e982ae50e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq\" (UID: \"bfc7b061-6dbc-4368-91e9-911e982ae50e\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"
Nov 03 17:36:21.628733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.628571    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bfc7b061-6dbc-4368-91e9-911e982ae50e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq\" (UID: \"bfc7b061-6dbc-4368-91e9-911e982ae50e\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"
Nov 03 17:36:21.628733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.628592    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc7b061-6dbc-4368-91e9-911e982ae50e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq\" (UID: \"bfc7b061-6dbc-4368-91e9-911e982ae50e\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"
Nov 03 17:36:21.628733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.628608    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86dtt\" (UniqueName: \"kubernetes.io/projected/bfc7b061-6dbc-4368-91e9-911e982ae50e-kube-api-access-86dtt\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq\" (UID: \"bfc7b061-6dbc-4368-91e9-911e982ae50e\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"
Nov 03 17:36:21.629308 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.629279    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bfc7b061-6dbc-4368-91e9-911e982ae50e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq\" (UID: \"bfc7b061-6dbc-4368-91e9-911e982ae50e\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"
Nov 03 17:36:21.629449 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.629325    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc7b061-6dbc-4368-91e9-911e982ae50e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq\" (UID: \"bfc7b061-6dbc-4368-91e9-911e982ae50e\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"
Nov 03 17:36:21.630972 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.630951    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bfc7b061-6dbc-4368-91e9-911e982ae50e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq\" (UID: \"bfc7b061-6dbc-4368-91e9-911e982ae50e\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"
Nov 03 17:36:21.692142 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.692062    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86dtt\" (UniqueName: \"kubernetes.io/projected/bfc7b061-6dbc-4368-91e9-911e982ae50e-kube-api-access-86dtt\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq\" (UID: \"bfc7b061-6dbc-4368-91e9-911e982ae50e\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"
Nov 03 17:36:21.739245 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.739210    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"
Nov 03 17:36:21.950770 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:21.950700    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:22.061242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:22.061208    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc"]
Nov 03 17:36:22.064860 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:22.064832    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e57e242-4xlgc"]
Nov 03 17:36:22.224191 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:22.224156    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"]
Nov 03 17:36:23.089891 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:23.089845    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq" event={"ID":"bfc7b061-6dbc-4368-91e9-911e982ae50e","Type":"ContainerStarted","Data":"35a76971ef094ed2f54a9e1d21c97b04ad2605a105b48388e69ef5fd3c384ad5"}
Nov 03 17:36:23.090282 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:23.089940    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq" event={"ID":"bfc7b061-6dbc-4368-91e9-911e982ae50e","Type":"ContainerStarted","Data":"db228137b57be0141634614e56a22517c1bebdf2cc1ba8c850eb7a48b8522afd"}
Nov 03 17:36:23.090282 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:23.089965    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"
Nov 03 17:36:23.214410 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:23.214346    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq" podStartSLOduration=2.214329558 podStartE2EDuration="2.214329558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:36:23.213966969 +0000 UTC m=+1124.011679750" watchObservedRunningTime="2026-11-03 17:36:23.214329558 +0000 UTC m=+1124.012042339"
Nov 03 17:36:23.336643 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:23.336605    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96b27261-8577-4b18-af00-8aebc33b13a7" path="/var/lib/kubelet/pods/96b27261-8577-4b18-af00-8aebc33b13a7/volumes"
Nov 03 17:36:24.458151 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:24.458113    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:29.151172 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:29.151141    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:33.563030 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:33.562985    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:33.760819 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:33.760777    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:34.101421 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:34.101345    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"
Nov 03 17:36:36.066451 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:36.066413    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6"]
Nov 03 17:36:36.118326 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:36.118290    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7538718-nmxc6"]
Nov 03 17:36:36.642572 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:36.642516    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:36.862282 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:36:36.862245    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:36:37.335998 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:37.335964    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fad6af55-71a2-4be1-aa27-07a1b0a4daff" path="/var/lib/kubelet/pods/fad6af55-71a2-4be1-aa27-07a1b0a4daff/volumes"
Nov 03 17:36:38.338663 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:38.338628    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:39.591648 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:39.591617    6059 scope.go:122] "RemoveContainer" containerID="7b8102c03567df33f4dd331e98bc13be1e24ce5fd5b081f5617e68d600fbe342"
Nov 03 17:36:39.601740 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:39.601720    6059 scope.go:122] "RemoveContainer" containerID="8070cc0a1429b5db0bf100c44c450119493315f67e2d853e7f89a2b8c6bb129a"
Nov 03 17:36:42.123769 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.123736    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"]
Nov 03 17:36:42.135696 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.129950    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.189393 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.189347    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-af9e3c-openshift-default-dockercfg-pw9pq\""
Nov 03 17:36:42.228691 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.228658    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-podinfo\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.228863 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.228770    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-workload-certs\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.228863 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.228816    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istiod-ca-cert\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.228976 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.228867    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zkjw\" (UniqueName: \"kubernetes.io/projected/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-kube-api-access-2zkjw\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.228976 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.228901    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-token\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.229084 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.229004    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-credential-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.229146 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.229109    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-data\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.229203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.229143    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-envoy\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.229203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.229175    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-workload-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.308706 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.295769    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"]
Nov 03 17:36:42.382085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.334209    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zkjw\" (UniqueName: \"kubernetes.io/projected/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-kube-api-access-2zkjw\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.382085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.334267    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-token\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.382085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.334303    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-credential-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.382085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.334404    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-data\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.382085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.334438    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-envoy\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.382085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.334474    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-workload-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.382085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.334598    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-podinfo\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.382085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.334696    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-workload-certs\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.382085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.334750    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istiod-ca-cert\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.382085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.335692    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istiod-ca-cert\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.382085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.381031    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-workload-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.382085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.381326    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-credential-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.382085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.381617    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-data\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.392046 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.392007    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-workload-certs\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.406418 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.406316    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-envoy\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.417124 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.417066    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-podinfo\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.418522 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.418495    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zkjw\" (UniqueName: \"kubernetes.io/projected/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-kube-api-access-2zkjw\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.490394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.479933    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-token\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:42.508457 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:42.506614    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:43.071435 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:43.071404    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"]
Nov 03 17:36:43.102625 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:36:43.102585    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb73f8e5_cce4_4bca_9e84_717537d8a3e0.slice/crio-4c46635e69e05839a6fa235ebc15b32689701f32062fbf53fea86276bafdacf7 WatchSource:0}: Error finding container 4c46635e69e05839a6fa235ebc15b32689701f32062fbf53fea86276bafdacf7: Status 404 returned error can't find the container with id 4c46635e69e05839a6fa235ebc15b32689701f32062fbf53fea86276bafdacf7
Nov 03 17:36:43.105060 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:43.105028    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:36:43.105146 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:43.105110    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:36:43.105146 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:43.105139    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:36:43.241598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:43.241554    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s" event={"ID":"fb73f8e5-cce4-4bca-9e84-717537d8a3e0","Type":"ContainerStarted","Data":"ec1ea4077664f643b0d540270b38a15ca2ceebed8a0ee6a8f2a509db8f1eccbc"}
Nov 03 17:36:43.242087 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:43.241664    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s" event={"ID":"fb73f8e5-cce4-4bca-9e84-717537d8a3e0","Type":"ContainerStarted","Data":"4c46635e69e05839a6fa235ebc15b32689701f32062fbf53fea86276bafdacf7"}
Nov 03 17:36:43.374969 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:43.374863    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s" podStartSLOduration=1.374850328 podStartE2EDuration="1.374850328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:36:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:36:43.373016163 +0000 UTC m=+1144.170728944" watchObservedRunningTime="2026-11-03 17:36:43.374850328 +0000 UTC m=+1144.172563108"
Nov 03 17:36:43.508288 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:43.508251    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:43.717981 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:43.713729    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:43.778682 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:43.778625    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q"]
Nov 03 17:36:43.783112 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:43.783092    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q"
Nov 03 17:36:43.825826 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:43.825791    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q"]
Nov 03 17:36:43.839693 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:43.839651    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:36:43.840017 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:43.839998    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:36:43.843892 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:43.843860    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:36:43.844945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:43.844195    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:36:43.921752 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:43.921721    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596510a6-5e78-40ea-8d68-11104a26aad0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8df0ea99-kv54q\" (UID: \"596510a6-5e78-40ea-8d68-11104a26aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q"
Nov 03 17:36:43.921939 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:43.921794    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sqp9\" (UniqueName: \"kubernetes.io/projected/596510a6-5e78-40ea-8d68-11104a26aad0-kube-api-access-7sqp9\") pod \"maas-api-revoke-sub-8df0ea99-kv54q\" (UID: \"596510a6-5e78-40ea-8d68-11104a26aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q"
Nov 03 17:36:43.921939 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:43.921903    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596510a6-5e78-40ea-8d68-11104a26aad0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8df0ea99-kv54q\" (UID: \"596510a6-5e78-40ea-8d68-11104a26aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q"
Nov 03 17:36:44.023777 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:44.023681    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596510a6-5e78-40ea-8d68-11104a26aad0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8df0ea99-kv54q\" (UID: \"596510a6-5e78-40ea-8d68-11104a26aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q"
Nov 03 17:36:44.023960 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:44.023791    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sqp9\" (UniqueName: \"kubernetes.io/projected/596510a6-5e78-40ea-8d68-11104a26aad0-kube-api-access-7sqp9\") pod \"maas-api-revoke-sub-8df0ea99-kv54q\" (UID: \"596510a6-5e78-40ea-8d68-11104a26aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q"
Nov 03 17:36:44.023960 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:44.023832    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596510a6-5e78-40ea-8d68-11104a26aad0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8df0ea99-kv54q\" (UID: \"596510a6-5e78-40ea-8d68-11104a26aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q"
Nov 03 17:36:44.024591 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:44.024565    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596510a6-5e78-40ea-8d68-11104a26aad0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8df0ea99-kv54q\" (UID: \"596510a6-5e78-40ea-8d68-11104a26aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q"
Nov 03 17:36:44.026008 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:44.025987    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596510a6-5e78-40ea-8d68-11104a26aad0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8df0ea99-kv54q\" (UID: \"596510a6-5e78-40ea-8d68-11104a26aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q"
Nov 03 17:36:44.087175 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:44.087144    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sqp9\" (UniqueName: \"kubernetes.io/projected/596510a6-5e78-40ea-8d68-11104a26aad0-kube-api-access-7sqp9\") pod \"maas-api-revoke-sub-8df0ea99-kv54q\" (UID: \"596510a6-5e78-40ea-8d68-11104a26aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q"
Nov 03 17:36:44.098032 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:44.098004    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q"
Nov 03 17:36:44.499644 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:36:44.499605    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596510a6_5e78_40ea_8d68_11104a26aad0.slice/crio-397ef38ae81c62957dcd8fa62917fec882be30dff0c37cf97830b0348e2f6a5e WatchSource:0}: Error finding container 397ef38ae81c62957dcd8fa62917fec882be30dff0c37cf97830b0348e2f6a5e: Status 404 returned error can't find the container with id 397ef38ae81c62957dcd8fa62917fec882be30dff0c37cf97830b0348e2f6a5e
Nov 03 17:36:44.503010 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:44.502986    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q"]
Nov 03 17:36:44.510314 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:44.510289    6059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 17:36:44.510439 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:44.510344    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s" podUID="fb73f8e5-cce4-4bca-9e84-717537d8a3e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 17:36:45.274574 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:45.274539    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q" event={"ID":"596510a6-5e78-40ea-8d68-11104a26aad0","Type":"ContainerStarted","Data":"c7a8ff48df11ea25cec701febf092047dc8d21740b12801089c52f071a52468e"}
Nov 03 17:36:45.274740 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:45.274586    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q" event={"ID":"596510a6-5e78-40ea-8d68-11104a26aad0","Type":"ContainerStarted","Data":"397ef38ae81c62957dcd8fa62917fec882be30dff0c37cf97830b0348e2f6a5e"}
Nov 03 17:36:45.396640 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:45.396588    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q" podStartSLOduration=2.396575133 podStartE2EDuration="2.396575133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:36:45.39512213 +0000 UTC m=+1146.192834911" watchObservedRunningTime="2026-11-03 17:36:45.396575133 +0000 UTC m=+1146.194287914"
Nov 03 17:36:45.509215 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:45.509188    6059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 17:36:45.509575 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:45.509235    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s" podUID="fb73f8e5-cce4-4bca-9e84-717537d8a3e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 17:36:46.283694 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:46.283650    6059 generic.go:356] "Generic (PLEG): container finished" podID="596510a6-5e78-40ea-8d68-11104a26aad0" containerID="c7a8ff48df11ea25cec701febf092047dc8d21740b12801089c52f071a52468e" exitCode=0
Nov 03 17:36:46.283961 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:46.283705    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q" event={"ID":"596510a6-5e78-40ea-8d68-11104a26aad0","Type":"ContainerDied","Data":"c7a8ff48df11ea25cec701febf092047dc8d21740b12801089c52f071a52468e"}
Nov 03 17:36:46.508536 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:46.508502    6059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 17:36:46.508721 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:46.508563    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s" podUID="fb73f8e5-cce4-4bca-9e84-717537d8a3e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 17:36:47.025084 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:47.025050    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"]
Nov 03 17:36:47.103734 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:47.103700    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:47.486736 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:47.486710    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q"
Nov 03 17:36:47.509152 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:47.509120    6059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 17:36:47.509313 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:47.509180    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s" podUID="fb73f8e5-cce4-4bca-9e84-717537d8a3e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 17:36:47.592905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:47.592813    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/596510a6-5e78-40ea-8d68-11104a26aad0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596510a6-5e78-40ea-8d68-11104a26aad0-maas-api-openshift-ca\") pod \"596510a6-5e78-40ea-8d68-11104a26aad0\" (UID: \"596510a6-5e78-40ea-8d68-11104a26aad0\") "
Nov 03 17:36:47.593077 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:47.592953    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/596510a6-5e78-40ea-8d68-11104a26aad0-kube-api-access-7sqp9\" (UniqueName: \"kubernetes.io/projected/596510a6-5e78-40ea-8d68-11104a26aad0-kube-api-access-7sqp9\") pod \"596510a6-5e78-40ea-8d68-11104a26aad0\" (UID: \"596510a6-5e78-40ea-8d68-11104a26aad0\") "
Nov 03 17:36:47.593144 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:47.593102    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/596510a6-5e78-40ea-8d68-11104a26aad0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596510a6-5e78-40ea-8d68-11104a26aad0-maas-api-xks-ca\") pod \"596510a6-5e78-40ea-8d68-11104a26aad0\" (UID: \"596510a6-5e78-40ea-8d68-11104a26aad0\") "
Nov 03 17:36:47.593213 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:47.593158    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/596510a6-5e78-40ea-8d68-11104a26aad0-maas-api-openshift-ca" pod "596510a6-5e78-40ea-8d68-11104a26aad0" (UID: "596510a6-5e78-40ea-8d68-11104a26aad0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:36:47.593557 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:47.593536    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596510a6-5e78-40ea-8d68-11104a26aad0-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:36:47.595112 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:47.595090    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/596510a6-5e78-40ea-8d68-11104a26aad0-maas-api-xks-ca" pod "596510a6-5e78-40ea-8d68-11104a26aad0" (UID: "596510a6-5e78-40ea-8d68-11104a26aad0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:36:47.595169 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:47.595111    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596510a6-5e78-40ea-8d68-11104a26aad0-kube-api-access-7sqp9" pod "596510a6-5e78-40ea-8d68-11104a26aad0" (UID: "596510a6-5e78-40ea-8d68-11104a26aad0"). InnerVolumeSpecName "kube-api-access-7sqp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:36:47.695030 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:47.694994    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596510a6-5e78-40ea-8d68-11104a26aad0-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:36:47.695030 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:47.695024    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sqp9\" (UniqueName: \"kubernetes.io/projected/596510a6-5e78-40ea-8d68-11104a26aad0-kube-api-access-7sqp9\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:36:47.989706 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:47.989668    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf"]
Nov 03 17:36:47.990381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:47.990346    6059 cpu_manager.go:414] "containerMap: removing container" podUID="596510a6-5e78-40ea-8d68-11104a26aad0" containerName="revoke-keys"
Nov 03 17:36:47.990381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:47.990380    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596510a6-5e78-40ea-8d68-11104a26aad0" containerName="revoke-keys"
Nov 03 17:36:47.990531 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:47.990519    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="596510a6-5e78-40ea-8d68-11104a26aad0" containerName="revoke-keys"
Nov 03 17:36:47.993959 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:47.993943    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf"
Nov 03 17:36:48.065341 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:48.065310    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf"]
Nov 03 17:36:48.099325 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:48.099288    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8319bee0-f2bf-488e-bd77-7caaf56be17a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7663366-8njmf\" (UID: \"8319bee0-f2bf-488e-bd77-7caaf56be17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf"
Nov 03 17:36:48.099325 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:48.099324    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw6jb\" (UniqueName: \"kubernetes.io/projected/8319bee0-f2bf-488e-bd77-7caaf56be17a-kube-api-access-zw6jb\") pod \"maas-api-revoke-sub-e7663366-8njmf\" (UID: \"8319bee0-f2bf-488e-bd77-7caaf56be17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf"
Nov 03 17:36:48.099652 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:48.099460    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8319bee0-f2bf-488e-bd77-7caaf56be17a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7663366-8njmf\" (UID: \"8319bee0-f2bf-488e-bd77-7caaf56be17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf"
Nov 03 17:36:48.201198 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:48.201155    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8319bee0-f2bf-488e-bd77-7caaf56be17a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7663366-8njmf\" (UID: \"8319bee0-f2bf-488e-bd77-7caaf56be17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf"
Nov 03 17:36:48.201198 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:48.201202    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw6jb\" (UniqueName: \"kubernetes.io/projected/8319bee0-f2bf-488e-bd77-7caaf56be17a-kube-api-access-zw6jb\") pod \"maas-api-revoke-sub-e7663366-8njmf\" (UID: \"8319bee0-f2bf-488e-bd77-7caaf56be17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf"
Nov 03 17:36:48.201494 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:48.201292    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8319bee0-f2bf-488e-bd77-7caaf56be17a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7663366-8njmf\" (UID: \"8319bee0-f2bf-488e-bd77-7caaf56be17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf"
Nov 03 17:36:48.201937 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:48.201915    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8319bee0-f2bf-488e-bd77-7caaf56be17a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7663366-8njmf\" (UID: \"8319bee0-f2bf-488e-bd77-7caaf56be17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf"
Nov 03 17:36:48.203725 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:48.203694    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8319bee0-f2bf-488e-bd77-7caaf56be17a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7663366-8njmf\" (UID: \"8319bee0-f2bf-488e-bd77-7caaf56be17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf"
Nov 03 17:36:48.264473 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:48.264403    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw6jb\" (UniqueName: \"kubernetes.io/projected/8319bee0-f2bf-488e-bd77-7caaf56be17a-kube-api-access-zw6jb\") pod \"maas-api-revoke-sub-e7663366-8njmf\" (UID: \"8319bee0-f2bf-488e-bd77-7caaf56be17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf"
Nov 03 17:36:48.300915 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:48.300885    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q"
Nov 03 17:36:48.301101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:48.300931    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q" event={"ID":"596510a6-5e78-40ea-8d68-11104a26aad0","Type":"ContainerDied","Data":"397ef38ae81c62957dcd8fa62917fec882be30dff0c37cf97830b0348e2f6a5e"}
Nov 03 17:36:48.301101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:48.300992    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="397ef38ae81c62957dcd8fa62917fec882be30dff0c37cf97830b0348e2f6a5e"
Nov 03 17:36:48.310517 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:48.310494    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf"
Nov 03 17:36:48.500971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:48.500935    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:48.510053 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:48.509004    6059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 17:36:48.510053 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:48.509047    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s" podUID="fb73f8e5-cce4-4bca-9e84-717537d8a3e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 17:36:48.731381 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:36:48.726094    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8319bee0_f2bf_488e_bd77_7caaf56be17a.slice/crio-6a26f15c39f4cecaa200957fa3b56aeb33d899e0e63c2cffaf078b6d641ea4d3 WatchSource:0}: Error finding container 6a26f15c39f4cecaa200957fa3b56aeb33d899e0e63c2cffaf078b6d641ea4d3: Status 404 returned error can't find the container with id 6a26f15c39f4cecaa200957fa3b56aeb33d899e0e63c2cffaf078b6d641ea4d3
Nov 03 17:36:48.731381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:48.726346    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf"]
Nov 03 17:36:49.321271 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:49.321226    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf" event={"ID":"8319bee0-f2bf-488e-bd77-7caaf56be17a","Type":"ContainerStarted","Data":"89237ad86fc9faa9c3e4f99ad64b052e14caf0fa44f2de3018fa0227f73be61d"}
Nov 03 17:36:49.321729 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:49.321316    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf" event={"ID":"8319bee0-f2bf-488e-bd77-7caaf56be17a","Type":"ContainerStarted","Data":"6a26f15c39f4cecaa200957fa3b56aeb33d899e0e63c2cffaf078b6d641ea4d3"}
Nov 03 17:36:49.444177 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:49.444057    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf" podStartSLOduration=2.444043418 podStartE2EDuration="2.444043418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:36:49.440488843 +0000 UTC m=+1150.238201625" watchObservedRunningTime="2026-11-03 17:36:49.444043418 +0000 UTC m=+1150.241756199"
Nov 03 17:36:49.508479 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:49.508446    6059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 17:36:49.508663 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:49.508520    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s" podUID="fb73f8e5-cce4-4bca-9e84-717537d8a3e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 17:36:50.330718 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:50.330678    6059 generic.go:356] "Generic (PLEG): container finished" podID="8319bee0-f2bf-488e-bd77-7caaf56be17a" containerID="89237ad86fc9faa9c3e4f99ad64b052e14caf0fa44f2de3018fa0227f73be61d" exitCode=0
Nov 03 17:36:50.331121 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:50.330768    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf" event={"ID":"8319bee0-f2bf-488e-bd77-7caaf56be17a","Type":"ContainerDied","Data":"89237ad86fc9faa9c3e4f99ad64b052e14caf0fa44f2de3018fa0227f73be61d"}
Nov 03 17:36:50.513312 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:50.513279    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:50.513849 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:50.513823    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:50.514606 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:50.514582    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:36:51.496427 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:51.496404    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf"
Nov 03 17:36:51.551727 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:51.551697    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8319bee0-f2bf-488e-bd77-7caaf56be17a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8319bee0-f2bf-488e-bd77-7caaf56be17a-maas-api-xks-ca\") pod \"8319bee0-f2bf-488e-bd77-7caaf56be17a\" (UID: \"8319bee0-f2bf-488e-bd77-7caaf56be17a\") "
Nov 03 17:36:51.551894 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:51.551839    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8319bee0-f2bf-488e-bd77-7caaf56be17a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8319bee0-f2bf-488e-bd77-7caaf56be17a-maas-api-openshift-ca\") pod \"8319bee0-f2bf-488e-bd77-7caaf56be17a\" (UID: \"8319bee0-f2bf-488e-bd77-7caaf56be17a\") "
Nov 03 17:36:51.551971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:51.551953    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8319bee0-f2bf-488e-bd77-7caaf56be17a-kube-api-access-zw6jb\" (UniqueName: \"kubernetes.io/projected/8319bee0-f2bf-488e-bd77-7caaf56be17a-kube-api-access-zw6jb\") pod \"8319bee0-f2bf-488e-bd77-7caaf56be17a\" (UID: \"8319bee0-f2bf-488e-bd77-7caaf56be17a\") "
Nov 03 17:36:51.552242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:51.552215    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8319bee0-f2bf-488e-bd77-7caaf56be17a-maas-api-openshift-ca" pod "8319bee0-f2bf-488e-bd77-7caaf56be17a" (UID: "8319bee0-f2bf-488e-bd77-7caaf56be17a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:36:51.552480 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:51.552452    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8319bee0-f2bf-488e-bd77-7caaf56be17a-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:36:51.554175 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:51.554142    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8319bee0-f2bf-488e-bd77-7caaf56be17a-maas-api-xks-ca" pod "8319bee0-f2bf-488e-bd77-7caaf56be17a" (UID: "8319bee0-f2bf-488e-bd77-7caaf56be17a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:36:51.554282 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:51.554184    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8319bee0-f2bf-488e-bd77-7caaf56be17a-kube-api-access-zw6jb" pod "8319bee0-f2bf-488e-bd77-7caaf56be17a" (UID: "8319bee0-f2bf-488e-bd77-7caaf56be17a"). InnerVolumeSpecName "kube-api-access-zw6jb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:36:51.654010 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:51.653920    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw6jb\" (UniqueName: \"kubernetes.io/projected/8319bee0-f2bf-488e-bd77-7caaf56be17a-kube-api-access-zw6jb\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:36:51.654010 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:51.653950    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8319bee0-f2bf-488e-bd77-7caaf56be17a-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:36:52.347810 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:52.347776    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf"
Nov 03 17:36:52.347989 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:52.347784    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf" event={"ID":"8319bee0-f2bf-488e-bd77-7caaf56be17a","Type":"ContainerDied","Data":"6a26f15c39f4cecaa200957fa3b56aeb33d899e0e63c2cffaf078b6d641ea4d3"}
Nov 03 17:36:52.347989 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:52.347908    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a26f15c39f4cecaa200957fa3b56aeb33d899e0e63c2cffaf078b6d641ea4d3"
Nov 03 17:36:53.142621 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:53.142443    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:55.583544 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:55.583491    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:36:55.584218 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:55.584184    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5" podUID="63c795d8-dfff-43d8-88a2-de65cc01a51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:36:58.158255 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:58.158203    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:36:59.059436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:59.059399    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e57883-75k8d"]
Nov 03 17:36:59.109648 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:59.109614    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e57883-75k8d"]
Nov 03 17:36:59.337325 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:59.337250    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="683ae3d8-d754-4544-b8cc-342cff61f5f0" path="/var/lib/kubelet/pods/683ae3d8-d754-4544-b8cc-342cff61f5f0/volumes"
Nov 03 17:36:59.797977 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:59.797942    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf"]
Nov 03 17:36:59.798548 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:59.798534    6059 cpu_manager.go:414] "containerMap: removing container" podUID="8319bee0-f2bf-488e-bd77-7caaf56be17a" containerName="revoke-keys"
Nov 03 17:36:59.798598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:59.798550    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8319bee0-f2bf-488e-bd77-7caaf56be17a" containerName="revoke-keys"
Nov 03 17:36:59.798675 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:59.798666    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8319bee0-f2bf-488e-bd77-7caaf56be17a" containerName="revoke-keys"
Nov 03 17:36:59.802506 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:59.802474    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf"
Nov 03 17:36:59.859747 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:59.859715    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:36:59.859747 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:59.859731    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:36:59.860071 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:59.860053    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:36:59.899058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:59.899018    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d813936c-cm7mf\" (UID: \"76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf"
Nov 03 17:36:59.899238 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:59.899067    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d813936c-cm7mf\" (UID: \"76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf"
Nov 03 17:36:59.899238 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:59.899187    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qnlm\" (UniqueName: \"kubernetes.io/projected/76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95-kube-api-access-8qnlm\") pod \"maas-api-revoke-sub-d813936c-cm7mf\" (UID: \"76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf"
Nov 03 17:36:59.911652 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:59.911615    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:36:59.915574 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:36:59.915550    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf"]
Nov 03 17:37:00.000861 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:00.000827    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d813936c-cm7mf\" (UID: \"76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf"
Nov 03 17:37:00.001009 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:00.000874    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d813936c-cm7mf\" (UID: \"76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf"
Nov 03 17:37:00.001009 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:00.000922    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qnlm\" (UniqueName: \"kubernetes.io/projected/76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95-kube-api-access-8qnlm\") pod \"maas-api-revoke-sub-d813936c-cm7mf\" (UID: \"76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf"
Nov 03 17:37:00.001518 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:00.001484    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d813936c-cm7mf\" (UID: \"76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf"
Nov 03 17:37:00.003269 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:00.003243    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d813936c-cm7mf\" (UID: \"76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf"
Nov 03 17:37:00.066065 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:00.065993    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qnlm\" (UniqueName: \"kubernetes.io/projected/76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95-kube-api-access-8qnlm\") pod \"maas-api-revoke-sub-d813936c-cm7mf\" (UID: \"76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf"
Nov 03 17:37:00.117403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:00.117347    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf"
Nov 03 17:37:00.496433 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:37:00.496087    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76db6a7a_3db3_4c2f_9a0d_16e2ab3e5d95.slice/crio-143a2131d5cadc6b2be0941a30c40209443451834fc8b0f338618a5dbbc2c6ec WatchSource:0}: Error finding container 143a2131d5cadc6b2be0941a30c40209443451834fc8b0f338618a5dbbc2c6ec: Status 404 returned error can't find the container with id 143a2131d5cadc6b2be0941a30c40209443451834fc8b0f338618a5dbbc2c6ec
Nov 03 17:37:00.500510 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:00.500482    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf"]
Nov 03 17:37:00.889888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:00.889797    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:01.464461 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:01.464415    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf" event={"ID":"76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95","Type":"ContainerStarted","Data":"9eaa35bfda27baba1cc2c713469405a3d9a81c5e8591daeb8c63d257e0f5e852"}
Nov 03 17:37:01.464653 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:01.464490    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf" event={"ID":"76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95","Type":"ContainerStarted","Data":"143a2131d5cadc6b2be0941a30c40209443451834fc8b0f338618a5dbbc2c6ec"}
Nov 03 17:37:01.593911 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:01.593854    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf" podStartSLOduration=2.593834854 podStartE2EDuration="2.593834854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:37:01.592044643 +0000 UTC m=+1162.389757425" watchObservedRunningTime="2026-11-03 17:37:01.593834854 +0000 UTC m=+1162.391547634"
Nov 03 17:37:02.072661 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.072622    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g" podUID="83ca6484-7e38-4782-b3a9-57b58bdc3920" containerName="main" containerID="cri-o://741eb234c9b2f81453242547032304249bf0fffc82cbaa6e8528343b15e55747" gracePeriod=45
Nov 03 17:37:02.353592 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.353562    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:37:02.435866 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.435831    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ca6484-7e38-4782-b3a9-57b58bdc3920-kube-api-access-zlmvq\" (UniqueName: \"kubernetes.io/projected/83ca6484-7e38-4782-b3a9-57b58bdc3920-kube-api-access-zlmvq\") pod \"83ca6484-7e38-4782-b3a9-57b58bdc3920\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") "
Nov 03 17:37:02.436041 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.435909    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-model-cache\") pod \"83ca6484-7e38-4782-b3a9-57b58bdc3920\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") "
Nov 03 17:37:02.436041 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.436033    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83ca6484-7e38-4782-b3a9-57b58bdc3920-tls-certs\" (UniqueName: \"kubernetes.io/secret/83ca6484-7e38-4782-b3a9-57b58bdc3920-tls-certs\") pod \"83ca6484-7e38-4782-b3a9-57b58bdc3920\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") "
Nov 03 17:37:02.436137 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.436108    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-tmp-dir\") pod \"83ca6484-7e38-4782-b3a9-57b58bdc3920\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") "
Nov 03 17:37:02.436225 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.436200    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-model-cache" pod "83ca6484-7e38-4782-b3a9-57b58bdc3920" (UID: "83ca6484-7e38-4782-b3a9-57b58bdc3920"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:37:02.436225 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.436218    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-home\" (UniqueName: \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-home\") pod \"83ca6484-7e38-4782-b3a9-57b58bdc3920\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") "
Nov 03 17:37:02.436337 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.436256    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-dshm\" (UniqueName: \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-dshm\") pod \"83ca6484-7e38-4782-b3a9-57b58bdc3920\" (UID: \"83ca6484-7e38-4782-b3a9-57b58bdc3920\") "
Nov 03 17:37:02.436427 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.436396    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-tmp-dir" pod "83ca6484-7e38-4782-b3a9-57b58bdc3920" (UID: "83ca6484-7e38-4782-b3a9-57b58bdc3920"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:37:02.436484 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.436430    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-home" pod "83ca6484-7e38-4782-b3a9-57b58bdc3920" (UID: "83ca6484-7e38-4782-b3a9-57b58bdc3920"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:37:02.436876 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.436852    6059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-home\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:02.436984 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.436881    6059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-model-cache\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:02.436984 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.436897    6059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-tmp-dir\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:02.438279 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.438258    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-dshm" pod "83ca6484-7e38-4782-b3a9-57b58bdc3920" (UID: "83ca6484-7e38-4782-b3a9-57b58bdc3920"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:37:02.438385 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.438331    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83ca6484-7e38-4782-b3a9-57b58bdc3920-tls-certs" pod "83ca6484-7e38-4782-b3a9-57b58bdc3920" (UID: "83ca6484-7e38-4782-b3a9-57b58bdc3920"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:37:02.438456 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.438431    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ca6484-7e38-4782-b3a9-57b58bdc3920-kube-api-access-zlmvq" pod "83ca6484-7e38-4782-b3a9-57b58bdc3920" (UID: "83ca6484-7e38-4782-b3a9-57b58bdc3920"). InnerVolumeSpecName "kube-api-access-zlmvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:37:02.472842 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.472811    6059 generic.go:356] "Generic (PLEG): container finished" podID="83ca6484-7e38-4782-b3a9-57b58bdc3920" containerID="741eb234c9b2f81453242547032304249bf0fffc82cbaa6e8528343b15e55747" exitCode=0
Nov 03 17:37:02.472971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.472865    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"
Nov 03 17:37:02.472971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.472892    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g" event={"ID":"83ca6484-7e38-4782-b3a9-57b58bdc3920","Type":"ContainerDied","Data":"741eb234c9b2f81453242547032304249bf0fffc82cbaa6e8528343b15e55747"}
Nov 03 17:37:02.472971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.472941    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g" event={"ID":"83ca6484-7e38-4782-b3a9-57b58bdc3920","Type":"ContainerDied","Data":"aabd547ffc81af5fa5089d91bcb54eadd8462ae6476e383a62f9b1bdcbb95749"}
Nov 03 17:37:02.472971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.472963    6059 scope.go:122] "RemoveContainer" containerID="741eb234c9b2f81453242547032304249bf0fffc82cbaa6e8528343b15e55747"
Nov 03 17:37:02.475554 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.475352    6059 generic.go:356] "Generic (PLEG): container finished" podID="76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95" containerID="9eaa35bfda27baba1cc2c713469405a3d9a81c5e8591daeb8c63d257e0f5e852" exitCode=0
Nov 03 17:37:02.475554 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.475407    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf" event={"ID":"76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95","Type":"ContainerDied","Data":"9eaa35bfda27baba1cc2c713469405a3d9a81c5e8591daeb8c63d257e0f5e852"}
Nov 03 17:37:02.485445 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.485422    6059 scope.go:122] "RemoveContainer" containerID="741eb234c9b2f81453242547032304249bf0fffc82cbaa6e8528343b15e55747"
Nov 03 17:37:02.485782 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:37:02.485750    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"741eb234c9b2f81453242547032304249bf0fffc82cbaa6e8528343b15e55747\": container with ID starting with 741eb234c9b2f81453242547032304249bf0fffc82cbaa6e8528343b15e55747 not found: ID does not exist" containerID="741eb234c9b2f81453242547032304249bf0fffc82cbaa6e8528343b15e55747"
Nov 03 17:37:02.485875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.485795    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741eb234c9b2f81453242547032304249bf0fffc82cbaa6e8528343b15e55747"} err="failed to get container status \"741eb234c9b2f81453242547032304249bf0fffc82cbaa6e8528343b15e55747\": rpc error: code = NotFound desc = could not find container \"741eb234c9b2f81453242547032304249bf0fffc82cbaa6e8528343b15e55747\": container with ID starting with 741eb234c9b2f81453242547032304249bf0fffc82cbaa6e8528343b15e55747 not found: ID does not exist"
Nov 03 17:37:02.538009 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.537973    6059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83ca6484-7e38-4782-b3a9-57b58bdc3920-tls-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:02.538009 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.538006    6059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83ca6484-7e38-4782-b3a9-57b58bdc3920-dshm\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:02.538210 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.538020    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlmvq\" (UniqueName: \"kubernetes.io/projected/83ca6484-7e38-4782-b3a9-57b58bdc3920-kube-api-access-zlmvq\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:02.768018 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.767985    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"]
Nov 03 17:37:02.771692 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.771658    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-3f1cf1/test-model-3f1cf1-kserve-fbdcd7bb7-rgn6g"]
Nov 03 17:37:02.933822 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:02.933785    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:03.285234 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:03.285199    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:03.349812 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:03.347958    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83ca6484-7e38-4782-b3a9-57b58bdc3920" path="/var/lib/kubelet/pods/83ca6484-7e38-4782-b3a9-57b58bdc3920/volumes"
Nov 03 17:37:03.631600 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:03.631571    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf"
Nov 03 17:37:03.755485 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:03.755456    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95-kube-api-access-8qnlm\" (UniqueName: \"kubernetes.io/projected/76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95-kube-api-access-8qnlm\") pod \"76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95\" (UID: \"76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95\") "
Nov 03 17:37:03.755686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:03.755521    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95-maas-api-openshift-ca\") pod \"76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95\" (UID: \"76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95\") "
Nov 03 17:37:03.755746 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:03.755694    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95-maas-api-xks-ca\") pod \"76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95\" (UID: \"76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95\") "
Nov 03 17:37:03.755918 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:03.755889    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95-maas-api-openshift-ca" pod "76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95" (UID: "76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:37:03.756059 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:03.756043    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:03.757662 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:03.757624    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95-maas-api-xks-ca" pod "76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95" (UID: "76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:37:03.757847 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:03.757819    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95-kube-api-access-8qnlm" pod "76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95" (UID: "76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95"). InnerVolumeSpecName "kube-api-access-8qnlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:37:03.804415 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:03.804289    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:03.857826 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:03.857782    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:03.857826 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:03.857822    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qnlm\" (UniqueName: \"kubernetes.io/projected/76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95-kube-api-access-8qnlm\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:04.492414 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:04.492380    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf" event={"ID":"76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95","Type":"ContainerDied","Data":"143a2131d5cadc6b2be0941a30c40209443451834fc8b0f338618a5dbbc2c6ec"}
Nov 03 17:37:04.492598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:04.492431    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="143a2131d5cadc6b2be0941a30c40209443451834fc8b0f338618a5dbbc2c6ec"
Nov 03 17:37:04.492598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:04.492430    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf"
Nov 03 17:37:06.861031 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:37:06.860988    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:37:07.638518 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:07.638445    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:09.053218 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:09.053189    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6e0d5b2-wzz6r"]
Nov 03 17:37:09.104970 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:09.104935    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6e0d5b2-wzz6r"]
Nov 03 17:37:09.335978 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:09.335897    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27d41f28-9e55-4c45-9fd7-275c2c00555a" path="/var/lib/kubelet/pods/27d41f28-9e55-4c45-9fd7-275c2c00555a/volumes"
Nov 03 17:37:09.960999 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:09.960959    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:10.148458 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:10.148404    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:10.326571 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:10.326473    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:13.437383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:13.430751    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl" podUID="c194a4be-0499-419f-a4a1-910dbaafd303" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:37:13.530944 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:13.530895    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:13.615550 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:13.615506    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n" podUID="d6c1263b-e648-4f10-af7d-86397d6b207a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:37:13.969530 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:13.969475    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v" podUID="2bbc9082-b2da-4198-8a8a-b6ab26b5011e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:37:14.038519 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:14.038473    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w" podUID="6120f11d-14c8-4666-a201-469a5bca37de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded"
Nov 03 17:37:14.227577 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:14.227463    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv" podUID="7942e9d1-597f-47a6-9c70-1d174fe4c8e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:37:14.228813 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:14.228778    6059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v" podUID="2bbc9082-b2da-4198-8a8a-b6ab26b5011e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:37:14.493436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:14.441444    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl" podUID="c194a4be-0499-419f-a4a1-910dbaafd303" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:37:14.493436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:14.444717    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:37:14.644433 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:14.628337    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n" podUID="d6c1263b-e648-4f10-af7d-86397d6b207a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:37:14.644433 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:14.629061    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:37:14.970421 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:14.969658    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:37:15.156743 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:15.153583    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:37:16.332432 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.332398    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:16.718775 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.718742    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk"]
Nov 03 17:37:16.719384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.719368    6059 cpu_manager.go:414] "containerMap: removing container" podUID="76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95" containerName="revoke-keys"
Nov 03 17:37:16.719435 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.719385    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95" containerName="revoke-keys"
Nov 03 17:37:16.719435 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.719412    6059 cpu_manager.go:414] "containerMap: removing container" podUID="83ca6484-7e38-4782-b3a9-57b58bdc3920" containerName="main"
Nov 03 17:37:16.719435 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.719417    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ca6484-7e38-4782-b3a9-57b58bdc3920" containerName="main"
Nov 03 17:37:16.719529 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.719497    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ca6484-7e38-4782-b3a9-57b58bdc3920" containerName="main"
Nov 03 17:37:16.719529 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.719512    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95" containerName="revoke-keys"
Nov 03 17:37:16.724279 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.724259    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk"
Nov 03 17:37:16.781030 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.780993    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:37:16.781211 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.781091    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:37:16.781992 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.781977    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:37:16.782039 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.781992    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:37:16.788332 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.788302    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3457b5b-4c9b-457a-9275-8a10d710da0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e36c42c5-hwnbk\" (UID: \"b3457b5b-4c9b-457a-9275-8a10d710da0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk"
Nov 03 17:37:16.788528 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.788389    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3457b5b-4c9b-457a-9275-8a10d710da0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e36c42c5-hwnbk\" (UID: \"b3457b5b-4c9b-457a-9275-8a10d710da0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk"
Nov 03 17:37:16.788528 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.788503    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlqtq\" (UniqueName: \"kubernetes.io/projected/b3457b5b-4c9b-457a-9275-8a10d710da0b-kube-api-access-dlqtq\") pod \"maas-api-revoke-sub-e36c42c5-hwnbk\" (UID: \"b3457b5b-4c9b-457a-9275-8a10d710da0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk"
Nov 03 17:37:16.788944 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.788926    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk"]
Nov 03 17:37:16.890482 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.890430    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3457b5b-4c9b-457a-9275-8a10d710da0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e36c42c5-hwnbk\" (UID: \"b3457b5b-4c9b-457a-9275-8a10d710da0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk"
Nov 03 17:37:16.890695 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.890514    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3457b5b-4c9b-457a-9275-8a10d710da0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e36c42c5-hwnbk\" (UID: \"b3457b5b-4c9b-457a-9275-8a10d710da0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk"
Nov 03 17:37:16.890695 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.890546    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlqtq\" (UniqueName: \"kubernetes.io/projected/b3457b5b-4c9b-457a-9275-8a10d710da0b-kube-api-access-dlqtq\") pod \"maas-api-revoke-sub-e36c42c5-hwnbk\" (UID: \"b3457b5b-4c9b-457a-9275-8a10d710da0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk"
Nov 03 17:37:16.891176 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.891154    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3457b5b-4c9b-457a-9275-8a10d710da0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e36c42c5-hwnbk\" (UID: \"b3457b5b-4c9b-457a-9275-8a10d710da0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk"
Nov 03 17:37:16.892878 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.892857    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3457b5b-4c9b-457a-9275-8a10d710da0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e36c42c5-hwnbk\" (UID: \"b3457b5b-4c9b-457a-9275-8a10d710da0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk"
Nov 03 17:37:16.953881 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:16.953850    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlqtq\" (UniqueName: \"kubernetes.io/projected/b3457b5b-4c9b-457a-9275-8a10d710da0b-kube-api-access-dlqtq\") pod \"maas-api-revoke-sub-e36c42c5-hwnbk\" (UID: \"b3457b5b-4c9b-457a-9275-8a10d710da0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk"
Nov 03 17:37:17.037984 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:17.037893    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk"
Nov 03 17:37:17.434850 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:37:17.434814    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3457b5b_4c9b_457a_9275_8a10d710da0b.slice/crio-a8e34284bc35d6ecb6218f1acc82a5c9b8442aa2c8db25c9eb6c0aed6a2cd6eb WatchSource:0}: Error finding container a8e34284bc35d6ecb6218f1acc82a5c9b8442aa2c8db25c9eb6c0aed6a2cd6eb: Status 404 returned error can't find the container with id a8e34284bc35d6ecb6218f1acc82a5c9b8442aa2c8db25c9eb6c0aed6a2cd6eb
Nov 03 17:37:17.437578 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:17.437551    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk"]
Nov 03 17:37:17.624027 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:17.623950    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:17.834421 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:17.833709    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:17.940867 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:17.940819    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk" event={"ID":"b3457b5b-4c9b-457a-9275-8a10d710da0b","Type":"ContainerStarted","Data":"7919b8d4e26d095ac840596d9fefa1deba5bcee40965f3219edd26ab52076ffb"}
Nov 03 17:37:17.941091 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:17.940891    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk" event={"ID":"b3457b5b-4c9b-457a-9275-8a10d710da0b","Type":"ContainerStarted","Data":"a8e34284bc35d6ecb6218f1acc82a5c9b8442aa2c8db25c9eb6c0aed6a2cd6eb"}
Nov 03 17:37:18.062399 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:18.062263    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk" podStartSLOduration=2.062251122 podStartE2EDuration="2.062251122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:37:18.060786086 +0000 UTC m=+1178.858498870" watchObservedRunningTime="2026-11-03 17:37:18.062251122 +0000 UTC m=+1178.859963950"
Nov 03 17:37:18.949981 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:18.949947    6059 generic.go:356] "Generic (PLEG): container finished" podID="b3457b5b-4c9b-457a-9275-8a10d710da0b" containerID="7919b8d4e26d095ac840596d9fefa1deba5bcee40965f3219edd26ab52076ffb" exitCode=0
Nov 03 17:37:18.950415 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:18.950000    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk" event={"ID":"b3457b5b-4c9b-457a-9275-8a10d710da0b","Type":"ContainerDied","Data":"7919b8d4e26d095ac840596d9fefa1deba5bcee40965f3219edd26ab52076ffb"}
Nov 03 17:37:19.620333 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:19.620298    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:20.110107 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:20.110079    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk"
Nov 03 17:37:20.152425 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:20.152382    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3457b5b-4c9b-457a-9275-8a10d710da0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3457b5b-4c9b-457a-9275-8a10d710da0b-maas-api-xks-ca\") pod \"b3457b5b-4c9b-457a-9275-8a10d710da0b\" (UID: \"b3457b5b-4c9b-457a-9275-8a10d710da0b\") "
Nov 03 17:37:20.152645 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:20.152626    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3457b5b-4c9b-457a-9275-8a10d710da0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3457b5b-4c9b-457a-9275-8a10d710da0b-maas-api-openshift-ca\") pod \"b3457b5b-4c9b-457a-9275-8a10d710da0b\" (UID: \"b3457b5b-4c9b-457a-9275-8a10d710da0b\") "
Nov 03 17:37:20.152745 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:20.152730    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3457b5b-4c9b-457a-9275-8a10d710da0b-kube-api-access-dlqtq\" (UniqueName: \"kubernetes.io/projected/b3457b5b-4c9b-457a-9275-8a10d710da0b-kube-api-access-dlqtq\") pod \"b3457b5b-4c9b-457a-9275-8a10d710da0b\" (UID: \"b3457b5b-4c9b-457a-9275-8a10d710da0b\") "
Nov 03 17:37:20.152974 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:20.152949    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3457b5b-4c9b-457a-9275-8a10d710da0b-maas-api-openshift-ca" pod "b3457b5b-4c9b-457a-9275-8a10d710da0b" (UID: "b3457b5b-4c9b-457a-9275-8a10d710da0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:37:20.153149 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:20.153133    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3457b5b-4c9b-457a-9275-8a10d710da0b-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:20.154521 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:20.154486    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3457b5b-4c9b-457a-9275-8a10d710da0b-maas-api-xks-ca" pod "b3457b5b-4c9b-457a-9275-8a10d710da0b" (UID: "b3457b5b-4c9b-457a-9275-8a10d710da0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:37:20.154834 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:20.154812    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3457b5b-4c9b-457a-9275-8a10d710da0b-kube-api-access-dlqtq" pod "b3457b5b-4c9b-457a-9275-8a10d710da0b" (UID: "b3457b5b-4c9b-457a-9275-8a10d710da0b"). InnerVolumeSpecName "kube-api-access-dlqtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:37:20.254727 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:20.254645    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlqtq\" (UniqueName: \"kubernetes.io/projected/b3457b5b-4c9b-457a-9275-8a10d710da0b-kube-api-access-dlqtq\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:20.254727 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:20.254680    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3457b5b-4c9b-457a-9275-8a10d710da0b-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:20.966944 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:20.966915    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk"
Nov 03 17:37:20.967125 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:20.966939    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk" event={"ID":"b3457b5b-4c9b-457a-9275-8a10d710da0b","Type":"ContainerDied","Data":"a8e34284bc35d6ecb6218f1acc82a5c9b8442aa2c8db25c9eb6c0aed6a2cd6eb"}
Nov 03 17:37:20.967125 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:20.966996    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8e34284bc35d6ecb6218f1acc82a5c9b8442aa2c8db25c9eb6c0aed6a2cd6eb"
Nov 03 17:37:21.620804 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:21.620769    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"]
Nov 03 17:37:21.621402 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:21.621384    6059 cpu_manager.go:414] "containerMap: removing container" podUID="b3457b5b-4c9b-457a-9275-8a10d710da0b" containerName="revoke-keys"
Nov 03 17:37:21.621402 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:21.621401    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3457b5b-4c9b-457a-9275-8a10d710da0b" containerName="revoke-keys"
Nov 03 17:37:21.621546 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:21.621537    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3457b5b-4c9b-457a-9275-8a10d710da0b" containerName="revoke-keys"
Nov 03 17:37:21.625509 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:21.625491    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"
Nov 03 17:37:21.671039 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:21.670995    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/729411cc-0c94-4c63-9e1b-1efc7273c3df-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7\" (UID: \"729411cc-0c94-4c63-9e1b-1efc7273c3df\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"
Nov 03 17:37:21.671039 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:21.671041    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729411cc-0c94-4c63-9e1b-1efc7273c3df-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7\" (UID: \"729411cc-0c94-4c63-9e1b-1efc7273c3df\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"
Nov 03 17:37:21.671269 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:21.671247    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/729411cc-0c94-4c63-9e1b-1efc7273c3df-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7\" (UID: \"729411cc-0c94-4c63-9e1b-1efc7273c3df\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"
Nov 03 17:37:21.671433 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:21.671412    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gj9x\" (UniqueName: \"kubernetes.io/projected/729411cc-0c94-4c63-9e1b-1efc7273c3df-kube-api-access-4gj9x\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7\" (UID: \"729411cc-0c94-4c63-9e1b-1efc7273c3df\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"
Nov 03 17:37:21.732154 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:21.732125    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-dbae42e6384f\""
Nov 03 17:37:21.736215 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:21.736192    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"]
Nov 03 17:37:21.772913 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:21.772883    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/729411cc-0c94-4c63-9e1b-1efc7273c3df-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7\" (UID: \"729411cc-0c94-4c63-9e1b-1efc7273c3df\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"
Nov 03 17:37:21.773051 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:21.772919    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729411cc-0c94-4c63-9e1b-1efc7273c3df-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7\" (UID: \"729411cc-0c94-4c63-9e1b-1efc7273c3df\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"
Nov 03 17:37:21.773051 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:21.772990    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/729411cc-0c94-4c63-9e1b-1efc7273c3df-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7\" (UID: \"729411cc-0c94-4c63-9e1b-1efc7273c3df\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"
Nov 03 17:37:21.773138 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:21.773067    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gj9x\" (UniqueName: \"kubernetes.io/projected/729411cc-0c94-4c63-9e1b-1efc7273c3df-kube-api-access-4gj9x\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7\" (UID: \"729411cc-0c94-4c63-9e1b-1efc7273c3df\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"
Nov 03 17:37:21.773659 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:21.773639    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/729411cc-0c94-4c63-9e1b-1efc7273c3df-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7\" (UID: \"729411cc-0c94-4c63-9e1b-1efc7273c3df\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"
Nov 03 17:37:21.773778 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:21.773761    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729411cc-0c94-4c63-9e1b-1efc7273c3df-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7\" (UID: \"729411cc-0c94-4c63-9e1b-1efc7273c3df\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"
Nov 03 17:37:21.775534 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:21.775518    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/729411cc-0c94-4c63-9e1b-1efc7273c3df-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7\" (UID: \"729411cc-0c94-4c63-9e1b-1efc7273c3df\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"
Nov 03 17:37:21.835862 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:21.835829    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gj9x\" (UniqueName: \"kubernetes.io/projected/729411cc-0c94-4c63-9e1b-1efc7273c3df-kube-api-access-4gj9x\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7\" (UID: \"729411cc-0c94-4c63-9e1b-1efc7273c3df\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"
Nov 03 17:37:21.939786 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:21.939693    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"
Nov 03 17:37:22.321040 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:37:22.321005    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729411cc_0c94_4c63_9e1b_1efc7273c3df.slice/crio-a71cd57141ef55acf3a47c21c90c1be695a8bb63877c2642f8eaecf478496fa2 WatchSource:0}: Error finding container a71cd57141ef55acf3a47c21c90c1be695a8bb63877c2642f8eaecf478496fa2: Status 404 returned error can't find the container with id a71cd57141ef55acf3a47c21c90c1be695a8bb63877c2642f8eaecf478496fa2
Nov 03 17:37:22.327397 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:22.327370    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"]
Nov 03 17:37:22.986792 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:22.986751    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7" event={"ID":"729411cc-0c94-4c63-9e1b-1efc7273c3df","Type":"ContainerStarted","Data":"c1c73c8c795a2ef1c9530447a0deadf297d6719e04afbac38bcea5cc219fe22d"}
Nov 03 17:37:22.987209 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:22.986934    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"
Nov 03 17:37:22.987209 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:22.986956    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7" event={"ID":"729411cc-0c94-4c63-9e1b-1efc7273c3df","Type":"ContainerStarted","Data":"a71cd57141ef55acf3a47c21c90c1be695a8bb63877c2642f8eaecf478496fa2"}
Nov 03 17:37:23.109791 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:23.109739    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7" podStartSLOduration=2.109725335 podStartE2EDuration="2.109725335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:37:23.107521612 +0000 UTC m=+1183.905234393" watchObservedRunningTime="2026-11-03 17:37:23.109725335 +0000 UTC m=+1183.907438116"
Nov 03 17:37:23.735759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:23.735727    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:27.483956 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:27.483921    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl"]
Nov 03 17:37:27.488063 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:27.488043    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl"
Nov 03 17:37:27.596600 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:27.596553    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:37:27.596791 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:27.596739    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:37:27.596791 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:27.596753    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:37:27.596911 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:27.596786    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:37:27.600413 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:27.600388    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl"]
Nov 03 17:37:27.648105 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:27.648069    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb7hr\" (UniqueName: \"kubernetes.io/projected/84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181-kube-api-access-kb7hr\") pod \"maas-api-revoke-sub-1c367ba5-775zl\" (UID: \"84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl"
Nov 03 17:37:27.648265 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:27.648237    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c367ba5-775zl\" (UID: \"84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl"
Nov 03 17:37:27.648386 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:27.648341    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c367ba5-775zl\" (UID: \"84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl"
Nov 03 17:37:27.749408 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:27.749297    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c367ba5-775zl\" (UID: \"84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl"
Nov 03 17:37:27.749540 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:27.749468    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb7hr\" (UniqueName: \"kubernetes.io/projected/84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181-kube-api-access-kb7hr\") pod \"maas-api-revoke-sub-1c367ba5-775zl\" (UID: \"84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl"
Nov 03 17:37:27.749590 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:27.749578    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c367ba5-775zl\" (UID: \"84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl"
Nov 03 17:37:27.750223 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:27.750198    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c367ba5-775zl\" (UID: \"84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl"
Nov 03 17:37:27.751665 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:27.751636    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c367ba5-775zl\" (UID: \"84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl"
Nov 03 17:37:27.813127 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:27.813095    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb7hr\" (UniqueName: \"kubernetes.io/projected/84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181-kube-api-access-kb7hr\") pod \"maas-api-revoke-sub-1c367ba5-775zl\" (UID: \"84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl"
Nov 03 17:37:28.102154 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:28.102065    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl"
Nov 03 17:37:28.450865 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:28.450789    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:28.494847 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:37:28.494813    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c4e55c_ceba_4fa0_9c41_1bbb4bfd4181.slice/crio-8231a92f1abcd79baffde2e5843bac355865c2290e4048899df9d4b234edb378 WatchSource:0}: Error finding container 8231a92f1abcd79baffde2e5843bac355865c2290e4048899df9d4b234edb378: Status 404 returned error can't find the container with id 8231a92f1abcd79baffde2e5843bac355865c2290e4048899df9d4b234edb378
Nov 03 17:37:28.499820 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:28.499792    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl"]
Nov 03 17:37:29.033070 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:29.032974    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl" event={"ID":"84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181","Type":"ContainerStarted","Data":"144c663c6dc49a7945b4106f39a617a49e82263a9ae8c3b97f47e79a9b60c847"}
Nov 03 17:37:29.033070 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:29.033039    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl" event={"ID":"84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181","Type":"ContainerStarted","Data":"8231a92f1abcd79baffde2e5843bac355865c2290e4048899df9d4b234edb378"}
Nov 03 17:37:29.156533 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:29.156473    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl" podStartSLOduration=2.156456243 podStartE2EDuration="2.156456243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:37:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:37:29.15307085 +0000 UTC m=+1189.950783631" watchObservedRunningTime="2026-11-03 17:37:29.156456243 +0000 UTC m=+1189.954169027"
Nov 03 17:37:30.045410 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:30.044479    6059 generic.go:356] "Generic (PLEG): container finished" podID="84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181" containerID="144c663c6dc49a7945b4106f39a617a49e82263a9ae8c3b97f47e79a9b60c847" exitCode=0
Nov 03 17:37:30.045410 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:30.044739    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl" event={"ID":"84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181","Type":"ContainerDied","Data":"144c663c6dc49a7945b4106f39a617a49e82263a9ae8c3b97f47e79a9b60c847"}
Nov 03 17:37:31.414145 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:31.414111    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl"
Nov 03 17:37:31.513260 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:31.513203    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181-kube-api-access-kb7hr\" (UniqueName: \"kubernetes.io/projected/84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181-kube-api-access-kb7hr\") pod \"84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181\" (UID: \"84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181\") "
Nov 03 17:37:31.513492 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:31.513476    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181-maas-api-xks-ca\") pod \"84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181\" (UID: \"84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181\") "
Nov 03 17:37:31.513663 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:31.513637    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181-maas-api-openshift-ca\") pod \"84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181\" (UID: \"84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181\") "
Nov 03 17:37:31.514040 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:31.514011    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181-maas-api-openshift-ca" pod "84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181" (UID: "84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:37:31.515402 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:31.515347    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:31.516081 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:31.516050    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181-kube-api-access-kb7hr" pod "84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181" (UID: "84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181"). InnerVolumeSpecName "kube-api-access-kb7hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:37:31.516187 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:31.516141    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181-maas-api-xks-ca" pod "84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181" (UID: "84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:37:31.617023 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:31.616979    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb7hr\" (UniqueName: \"kubernetes.io/projected/84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181-kube-api-access-kb7hr\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:31.617277 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:31.617258    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:32.109383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:32.079705    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl" event={"ID":"84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181","Type":"ContainerDied","Data":"8231a92f1abcd79baffde2e5843bac355865c2290e4048899df9d4b234edb378"}
Nov 03 17:37:32.109383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:32.079774    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8231a92f1abcd79baffde2e5843bac355865c2290e4048899df9d4b234edb378"
Nov 03 17:37:32.109383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:32.079850    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl"
Nov 03 17:37:32.272382 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:37:32.268493    6059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c4e55c_ceba_4fa0_9c41_1bbb4bfd4181.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 17:37:33.353225 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:33.353169    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:33.996759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:33.996727    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"
Nov 03 17:37:34.175802 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:34.175768    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"]
Nov 03 17:37:36.861556 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:37:36.861520    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:37:36.993888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:36.993839    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:38.090811 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:38.090779    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:39.176716 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.176650    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq" podUID="bfc7b061-6dbc-4368-91e9-911e982ae50e" containerName="payload-processing" containerID="cri-o://35a76971ef094ed2f54a9e1d21c97b04ad2605a105b48388e69ef5fd3c384ad5" gracePeriod=25
Nov 03 17:37:39.229559 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.229528    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c6a27d-kd5zr"]
Nov 03 17:37:39.230160 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.230147    6059 cpu_manager.go:414] "containerMap: removing container" podUID="84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181" containerName="revoke-keys"
Nov 03 17:37:39.230210 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.230162    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181" containerName="revoke-keys"
Nov 03 17:37:39.230266 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.230257    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181" containerName="revoke-keys"
Nov 03 17:37:39.234123 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.234097    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c6a27d-kd5zr"
Nov 03 17:37:39.341035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.341004    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:37:39.341035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.341033    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:37:39.341234 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.341045    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:37:39.341234 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.341082    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:37:39.345050 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.345027    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c6a27d-kd5zr"]
Nov 03 17:37:39.411559 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.411517    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/766f8ea9-89c1-459e-be88-dcdf8411abdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76c6a27d-kd5zr\" (UID: \"766f8ea9-89c1-459e-be88-dcdf8411abdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c6a27d-kd5zr"
Nov 03 17:37:39.411791 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.411568    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/766f8ea9-89c1-459e-be88-dcdf8411abdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76c6a27d-kd5zr\" (UID: \"766f8ea9-89c1-459e-be88-dcdf8411abdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c6a27d-kd5zr"
Nov 03 17:37:39.411791 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.411678    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhk5m\" (UniqueName: \"kubernetes.io/projected/766f8ea9-89c1-459e-be88-dcdf8411abdd-kube-api-access-nhk5m\") pod \"maas-api-revoke-sub-76c6a27d-kd5zr\" (UID: \"766f8ea9-89c1-459e-be88-dcdf8411abdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c6a27d-kd5zr"
Nov 03 17:37:39.464444 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.464417    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"
Nov 03 17:37:39.500379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.500342    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 17:37:39.510213 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.510184    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 17:37:39.512642 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.512616    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/766f8ea9-89c1-459e-be88-dcdf8411abdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76c6a27d-kd5zr\" (UID: \"766f8ea9-89c1-459e-be88-dcdf8411abdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c6a27d-kd5zr"
Nov 03 17:37:39.512800 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.512653    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/766f8ea9-89c1-459e-be88-dcdf8411abdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76c6a27d-kd5zr\" (UID: \"766f8ea9-89c1-459e-be88-dcdf8411abdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c6a27d-kd5zr"
Nov 03 17:37:39.512800 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.512758    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhk5m\" (UniqueName: \"kubernetes.io/projected/766f8ea9-89c1-459e-be88-dcdf8411abdd-kube-api-access-nhk5m\") pod \"maas-api-revoke-sub-76c6a27d-kd5zr\" (UID: \"766f8ea9-89c1-459e-be88-dcdf8411abdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c6a27d-kd5zr"
Nov 03 17:37:39.513298 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.513275    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/766f8ea9-89c1-459e-be88-dcdf8411abdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76c6a27d-kd5zr\" (UID: \"766f8ea9-89c1-459e-be88-dcdf8411abdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c6a27d-kd5zr"
Nov 03 17:37:39.515676 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.515635    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/766f8ea9-89c1-459e-be88-dcdf8411abdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76c6a27d-kd5zr\" (UID: \"766f8ea9-89c1-459e-be88-dcdf8411abdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c6a27d-kd5zr"
Nov 03 17:37:39.519621 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.519601    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:37:39.531345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.531323    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:37:39.576302 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.576276    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhk5m\" (UniqueName: \"kubernetes.io/projected/766f8ea9-89c1-459e-be88-dcdf8411abdd-kube-api-access-nhk5m\") pod \"maas-api-revoke-sub-76c6a27d-kd5zr\" (UID: \"766f8ea9-89c1-459e-be88-dcdf8411abdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c6a27d-kd5zr"
Nov 03 17:37:39.577011 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.576992    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c6a27d-kd5zr"
Nov 03 17:37:39.613886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.613854    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfc7b061-6dbc-4368-91e9-911e982ae50e-kube-api-access-86dtt\" (UniqueName: \"kubernetes.io/projected/bfc7b061-6dbc-4368-91e9-911e982ae50e-kube-api-access-86dtt\") pod \"bfc7b061-6dbc-4368-91e9-911e982ae50e\" (UID: \"bfc7b061-6dbc-4368-91e9-911e982ae50e\") "
Nov 03 17:37:39.614102 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.613938    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfc7b061-6dbc-4368-91e9-911e982ae50e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bfc7b061-6dbc-4368-91e9-911e982ae50e-routing-overlay-volume\") pod \"bfc7b061-6dbc-4368-91e9-911e982ae50e\" (UID: \"bfc7b061-6dbc-4368-91e9-911e982ae50e\") "
Nov 03 17:37:39.614102 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.613967    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfc7b061-6dbc-4368-91e9-911e982ae50e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc7b061-6dbc-4368-91e9-911e982ae50e-plugins-config-volume\") pod \"bfc7b061-6dbc-4368-91e9-911e982ae50e\" (UID: \"bfc7b061-6dbc-4368-91e9-911e982ae50e\") "
Nov 03 17:37:39.614102 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.614062    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfc7b061-6dbc-4368-91e9-911e982ae50e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bfc7b061-6dbc-4368-91e9-911e982ae50e-provider-credentials\") pod \"bfc7b061-6dbc-4368-91e9-911e982ae50e\" (UID: \"bfc7b061-6dbc-4368-91e9-911e982ae50e\") "
Nov 03 17:37:39.614383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.614343    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfc7b061-6dbc-4368-91e9-911e982ae50e-routing-overlay-volume" pod "bfc7b061-6dbc-4368-91e9-911e982ae50e" (UID: "bfc7b061-6dbc-4368-91e9-911e982ae50e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:37:39.614697 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.614668    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfc7b061-6dbc-4368-91e9-911e982ae50e-plugins-config-volume" pod "bfc7b061-6dbc-4368-91e9-911e982ae50e" (UID: "bfc7b061-6dbc-4368-91e9-911e982ae50e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:37:39.614812 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.614797    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bfc7b061-6dbc-4368-91e9-911e982ae50e-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:39.614898 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.614817    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc7b061-6dbc-4368-91e9-911e982ae50e-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:39.619791 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.619760    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc7b061-6dbc-4368-91e9-911e982ae50e-kube-api-access-86dtt" pod "bfc7b061-6dbc-4368-91e9-911e982ae50e" (UID: "bfc7b061-6dbc-4368-91e9-911e982ae50e"). InnerVolumeSpecName "kube-api-access-86dtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:37:39.621052 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.621023    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc7b061-6dbc-4368-91e9-911e982ae50e-provider-credentials" pod "bfc7b061-6dbc-4368-91e9-911e982ae50e" (UID: "bfc7b061-6dbc-4368-91e9-911e982ae50e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:37:39.645156 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.645126    6059 scope.go:122] "RemoveContainer" containerID="7ce580a5a776a21f2405c5875b3af60cf4d4711c82a7dea6fa7bd1be594a79f4"
Nov 03 17:37:39.660016 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.659979    6059 scope.go:122] "RemoveContainer" containerID="577f91781d042b6e6b83748ab5362abdbe79013a48fcb4c568ed24fe7256166a"
Nov 03 17:37:39.676055 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.676022    6059 scope.go:122] "RemoveContainer" containerID="4e7bd80fa30c4315df002eae0abca15340f2d6c629aef68bd6de2c2484f3edb7"
Nov 03 17:37:39.716344 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.716317    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86dtt\" (UniqueName: \"kubernetes.io/projected/bfc7b061-6dbc-4368-91e9-911e982ae50e-kube-api-access-86dtt\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:39.716344 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.716344    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bfc7b061-6dbc-4368-91e9-911e982ae50e-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:39.979199 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:37:39.979166    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766f8ea9_89c1_459e_be88_dcdf8411abdd.slice/crio-4ff0dbf3406e195924d76224fff33c2f437bea314ba21b669e1a86311c7c08a7 WatchSource:0}: Error finding container 4ff0dbf3406e195924d76224fff33c2f437bea314ba21b669e1a86311c7c08a7: Status 404 returned error can't find the container with id 4ff0dbf3406e195924d76224fff33c2f437bea314ba21b669e1a86311c7c08a7
Nov 03 17:37:39.979720 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:39.979699    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c6a27d-kd5zr"]
Nov 03 17:37:40.149431 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:40.149396    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c6a27d-kd5zr" event={"ID":"766f8ea9-89c1-459e-be88-dcdf8411abdd","Type":"ContainerStarted","Data":"4ff0dbf3406e195924d76224fff33c2f437bea314ba21b669e1a86311c7c08a7"}
Nov 03 17:37:40.150952 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:40.150929    6059 generic.go:356] "Generic (PLEG): container finished" podID="bfc7b061-6dbc-4368-91e9-911e982ae50e" containerID="35a76971ef094ed2f54a9e1d21c97b04ad2605a105b48388e69ef5fd3c384ad5" exitCode=0
Nov 03 17:37:40.151077 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:40.150986    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq" event={"ID":"bfc7b061-6dbc-4368-91e9-911e982ae50e","Type":"ContainerDied","Data":"35a76971ef094ed2f54a9e1d21c97b04ad2605a105b48388e69ef5fd3c384ad5"}
Nov 03 17:37:40.151077 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:40.150996    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"
Nov 03 17:37:40.151077 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:40.151018    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq" event={"ID":"bfc7b061-6dbc-4368-91e9-911e982ae50e","Type":"ContainerDied","Data":"db228137b57be0141634614e56a22517c1bebdf2cc1ba8c850eb7a48b8522afd"}
Nov 03 17:37:40.151077 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:40.151040    6059 scope.go:122] "RemoveContainer" containerID="35a76971ef094ed2f54a9e1d21c97b04ad2605a105b48388e69ef5fd3c384ad5"
Nov 03 17:37:40.162226 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:40.162203    6059 scope.go:122] "RemoveContainer" containerID="35a76971ef094ed2f54a9e1d21c97b04ad2605a105b48388e69ef5fd3c384ad5"
Nov 03 17:37:40.162594 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:37:40.162567    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a76971ef094ed2f54a9e1d21c97b04ad2605a105b48388e69ef5fd3c384ad5\": container with ID starting with 35a76971ef094ed2f54a9e1d21c97b04ad2605a105b48388e69ef5fd3c384ad5 not found: ID does not exist" containerID="35a76971ef094ed2f54a9e1d21c97b04ad2605a105b48388e69ef5fd3c384ad5"
Nov 03 17:37:40.162685 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:40.162598    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a76971ef094ed2f54a9e1d21c97b04ad2605a105b48388e69ef5fd3c384ad5"} err="failed to get container status \"35a76971ef094ed2f54a9e1d21c97b04ad2605a105b48388e69ef5fd3c384ad5\": rpc error: code = NotFound desc = could not find container \"35a76971ef094ed2f54a9e1d21c97b04ad2605a105b48388e69ef5fd3c384ad5\": container with ID starting with 35a76971ef094ed2f54a9e1d21c97b04ad2605a105b48388e69ef5fd3c384ad5 not found: ID does not exist"
Nov 03 17:37:40.279571 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:40.279520    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:40.325654 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:40.325618    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"]
Nov 03 17:37:40.329189 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:40.329160    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b42n6cq"]
Nov 03 17:37:40.589733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:40.589641    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:41.161291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:41.161197    6059 generic.go:356] "Generic (PLEG): container finished" podID="766f8ea9-89c1-459e-be88-dcdf8411abdd" containerID="cb2a892a4bc364f4ed976b362872798182c0fcc266532347d93b6b19d133a5c6" exitCode=0
Nov 03 17:37:41.161291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:41.161277    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c6a27d-kd5zr" event={"ID":"766f8ea9-89c1-459e-be88-dcdf8411abdd","Type":"ContainerDied","Data":"cb2a892a4bc364f4ed976b362872798182c0fcc266532347d93b6b19d133a5c6"}
Nov 03 17:37:41.337293 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:41.337260    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfc7b061-6dbc-4368-91e9-911e982ae50e" path="/var/lib/kubelet/pods/bfc7b061-6dbc-4368-91e9-911e982ae50e/volumes"
Nov 03 17:37:42.081975 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:42.081934    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f"]
Nov 03 17:37:42.133556 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:42.133518    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71750fe5-xnw9f"]
Nov 03 17:37:42.392930 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:42.392904    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c6a27d-kd5zr"
Nov 03 17:37:42.415281 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:42.415253    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/766f8ea9-89c1-459e-be88-dcdf8411abdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/766f8ea9-89c1-459e-be88-dcdf8411abdd-maas-api-openshift-ca\") pod \"766f8ea9-89c1-459e-be88-dcdf8411abdd\" (UID: \"766f8ea9-89c1-459e-be88-dcdf8411abdd\") "
Nov 03 17:37:42.415433 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:42.415305    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/766f8ea9-89c1-459e-be88-dcdf8411abdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/766f8ea9-89c1-459e-be88-dcdf8411abdd-maas-api-xks-ca\") pod \"766f8ea9-89c1-459e-be88-dcdf8411abdd\" (UID: \"766f8ea9-89c1-459e-be88-dcdf8411abdd\") "
Nov 03 17:37:42.415499 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:42.415478    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/766f8ea9-89c1-459e-be88-dcdf8411abdd-kube-api-access-nhk5m\" (UniqueName: \"kubernetes.io/projected/766f8ea9-89c1-459e-be88-dcdf8411abdd-kube-api-access-nhk5m\") pod \"766f8ea9-89c1-459e-be88-dcdf8411abdd\" (UID: \"766f8ea9-89c1-459e-be88-dcdf8411abdd\") "
Nov 03 17:37:42.415707 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:42.415681    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/766f8ea9-89c1-459e-be88-dcdf8411abdd-maas-api-openshift-ca" pod "766f8ea9-89c1-459e-be88-dcdf8411abdd" (UID: "766f8ea9-89c1-459e-be88-dcdf8411abdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:37:42.415944 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:42.415925    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/766f8ea9-89c1-459e-be88-dcdf8411abdd-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:42.417494 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:42.417467    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/766f8ea9-89c1-459e-be88-dcdf8411abdd-maas-api-xks-ca" pod "766f8ea9-89c1-459e-be88-dcdf8411abdd" (UID: "766f8ea9-89c1-459e-be88-dcdf8411abdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:37:42.417494 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:42.417488    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766f8ea9-89c1-459e-be88-dcdf8411abdd-kube-api-access-nhk5m" pod "766f8ea9-89c1-459e-be88-dcdf8411abdd" (UID: "766f8ea9-89c1-459e-be88-dcdf8411abdd"). InnerVolumeSpecName "kube-api-access-nhk5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:37:42.517179 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:42.517137    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/766f8ea9-89c1-459e-be88-dcdf8411abdd-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:42.517179 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:42.517172    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhk5m\" (UniqueName: \"kubernetes.io/projected/766f8ea9-89c1-459e-be88-dcdf8411abdd-kube-api-access-nhk5m\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:42.947338 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:42.947298    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:43.145332 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.145282    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:43.181911 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.181878    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c6a27d-kd5zr"
Nov 03 17:37:43.181911 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.181888    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76c6a27d-kd5zr" event={"ID":"766f8ea9-89c1-459e-be88-dcdf8411abdd","Type":"ContainerDied","Data":"4ff0dbf3406e195924d76224fff33c2f437bea314ba21b669e1a86311c7c08a7"}
Nov 03 17:37:43.182109 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.181943    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ff0dbf3406e195924d76224fff33c2f437bea314ba21b669e1a86311c7c08a7"
Nov 03 17:37:43.336250 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.336214    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76e54841-75f1-4d4c-846f-c99ea9051b4e" path="/var/lib/kubelet/pods/76e54841-75f1-4d4c-846f-c99ea9051b4e/volumes"
Nov 03 17:37:43.430042 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.429999    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"]
Nov 03 17:37:43.430928 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.430895    6059 cpu_manager.go:414] "containerMap: removing container" podUID="bfc7b061-6dbc-4368-91e9-911e982ae50e" containerName="payload-processing"
Nov 03 17:37:43.430928 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.430923    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfc7b061-6dbc-4368-91e9-911e982ae50e" containerName="payload-processing"
Nov 03 17:37:43.431104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.430973    6059 cpu_manager.go:414] "containerMap: removing container" podUID="766f8ea9-89c1-459e-be88-dcdf8411abdd" containerName="revoke-keys"
Nov 03 17:37:43.431104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.430982    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="766f8ea9-89c1-459e-be88-dcdf8411abdd" containerName="revoke-keys"
Nov 03 17:37:43.431104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.431093    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfc7b061-6dbc-4368-91e9-911e982ae50e" containerName="payload-processing"
Nov 03 17:37:43.431208 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.431108    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="766f8ea9-89c1-459e-be88-dcdf8411abdd" containerName="revoke-keys"
Nov 03 17:37:43.434940 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.434920    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"
Nov 03 17:37:43.491756 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.491712    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-029ca0\""
Nov 03 17:37:43.491982 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.491716    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"openshift-service-ca.crt\""
Nov 03 17:37:43.491982 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.491746    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"agc-cred-9dd652f47699\""
Nov 03 17:37:43.492149 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.491789    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"routing-overlay\""
Nov 03 17:37:43.492149 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.491954    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"kube-root-ca.crt\""
Nov 03 17:37:43.492259 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.492242    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"payload-processing-external-model-e2e--58d254fa-dockercfg-9bdmr\""
Nov 03 17:37:43.496372 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.496333    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"]
Nov 03 17:37:43.532083 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.532048    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhh5v\" (UniqueName: \"kubernetes.io/projected/96b48f17-328a-4914-af1f-600f02b88c26-kube-api-access-jhh5v\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj\" (UID: \"96b48f17-328a-4914-af1f-600f02b88c26\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"
Nov 03 17:37:43.532257 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.532119    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96b48f17-328a-4914-af1f-600f02b88c26-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj\" (UID: \"96b48f17-328a-4914-af1f-600f02b88c26\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"
Nov 03 17:37:43.532257 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.532225    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96b48f17-328a-4914-af1f-600f02b88c26-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj\" (UID: \"96b48f17-328a-4914-af1f-600f02b88c26\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"
Nov 03 17:37:43.532396 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.532262    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96b48f17-328a-4914-af1f-600f02b88c26-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj\" (UID: \"96b48f17-328a-4914-af1f-600f02b88c26\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"
Nov 03 17:37:43.633306 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.633222    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhh5v\" (UniqueName: \"kubernetes.io/projected/96b48f17-328a-4914-af1f-600f02b88c26-kube-api-access-jhh5v\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj\" (UID: \"96b48f17-328a-4914-af1f-600f02b88c26\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"
Nov 03 17:37:43.633476 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.633395    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96b48f17-328a-4914-af1f-600f02b88c26-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj\" (UID: \"96b48f17-328a-4914-af1f-600f02b88c26\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"
Nov 03 17:37:43.633521 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.633479    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96b48f17-328a-4914-af1f-600f02b88c26-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj\" (UID: \"96b48f17-328a-4914-af1f-600f02b88c26\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"
Nov 03 17:37:43.633521 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.633506    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96b48f17-328a-4914-af1f-600f02b88c26-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj\" (UID: \"96b48f17-328a-4914-af1f-600f02b88c26\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"
Nov 03 17:37:43.634025 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.634000    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96b48f17-328a-4914-af1f-600f02b88c26-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj\" (UID: \"96b48f17-328a-4914-af1f-600f02b88c26\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"
Nov 03 17:37:43.634178 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.634159    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96b48f17-328a-4914-af1f-600f02b88c26-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj\" (UID: \"96b48f17-328a-4914-af1f-600f02b88c26\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"
Nov 03 17:37:43.635940 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.635918    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96b48f17-328a-4914-af1f-600f02b88c26-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj\" (UID: \"96b48f17-328a-4914-af1f-600f02b88c26\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"
Nov 03 17:37:43.696495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.696456    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhh5v\" (UniqueName: \"kubernetes.io/projected/96b48f17-328a-4914-af1f-600f02b88c26-kube-api-access-jhh5v\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj\" (UID: \"96b48f17-328a-4914-af1f-600f02b88c26\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"
Nov 03 17:37:43.749326 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:43.749291    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"
Nov 03 17:37:44.131583 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:37:44.131552    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b48f17_328a_4914_af1f_600f02b88c26.slice/crio-2ee6f923c271a6f97afbbe68109931bf9f032977da6569f6e52836211a933409 WatchSource:0}: Error finding container 2ee6f923c271a6f97afbbe68109931bf9f032977da6569f6e52836211a933409: Status 404 returned error can't find the container with id 2ee6f923c271a6f97afbbe68109931bf9f032977da6569f6e52836211a933409
Nov 03 17:37:44.137411 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:44.137381    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"]
Nov 03 17:37:44.192895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:44.192865    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj" event={"ID":"96b48f17-328a-4914-af1f-600f02b88c26","Type":"ContainerStarted","Data":"2ee6f923c271a6f97afbbe68109931bf9f032977da6569f6e52836211a933409"}
Nov 03 17:37:45.025238 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.025206    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp"]
Nov 03 17:37:45.029629 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.029607    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp"
Nov 03 17:37:45.058206 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.058177    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec7cf9d3-8eb7-4398-a248-11712584d152-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3e6d015-bs6kp\" (UID: \"ec7cf9d3-8eb7-4398-a248-11712584d152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp"
Nov 03 17:37:45.058345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.058290    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7zf7\" (UniqueName: \"kubernetes.io/projected/ec7cf9d3-8eb7-4398-a248-11712584d152-kube-api-access-g7zf7\") pod \"maas-api-revoke-sub-a3e6d015-bs6kp\" (UID: \"ec7cf9d3-8eb7-4398-a248-11712584d152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp"
Nov 03 17:37:45.058345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.058313    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec7cf9d3-8eb7-4398-a248-11712584d152-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3e6d015-bs6kp\" (UID: \"ec7cf9d3-8eb7-4398-a248-11712584d152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp"
Nov 03 17:37:45.086839 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.086809    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:37:45.088061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.088040    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:37:45.094171 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.094140    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp"]
Nov 03 17:37:45.136817 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.136785    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:37:45.137004 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.136911    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:37:45.159206 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.159175    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec7cf9d3-8eb7-4398-a248-11712584d152-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3e6d015-bs6kp\" (UID: \"ec7cf9d3-8eb7-4398-a248-11712584d152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp"
Nov 03 17:37:45.159391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.159241    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7zf7\" (UniqueName: \"kubernetes.io/projected/ec7cf9d3-8eb7-4398-a248-11712584d152-kube-api-access-g7zf7\") pod \"maas-api-revoke-sub-a3e6d015-bs6kp\" (UID: \"ec7cf9d3-8eb7-4398-a248-11712584d152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp"
Nov 03 17:37:45.159391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.159260    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec7cf9d3-8eb7-4398-a248-11712584d152-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3e6d015-bs6kp\" (UID: \"ec7cf9d3-8eb7-4398-a248-11712584d152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp"
Nov 03 17:37:45.159906 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.159884    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec7cf9d3-8eb7-4398-a248-11712584d152-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3e6d015-bs6kp\" (UID: \"ec7cf9d3-8eb7-4398-a248-11712584d152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp"
Nov 03 17:37:45.161655 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.161632    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec7cf9d3-8eb7-4398-a248-11712584d152-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3e6d015-bs6kp\" (UID: \"ec7cf9d3-8eb7-4398-a248-11712584d152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp"
Nov 03 17:37:45.204050 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.204010    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj" event={"ID":"96b48f17-328a-4914-af1f-600f02b88c26","Type":"ContainerStarted","Data":"05b2a039ecbaa8dbda3c57d72847dcae0a7b66fdad0f6c6438ca910c222a945c"}
Nov 03 17:37:45.204181 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.204079    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"
Nov 03 17:37:45.229582 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.229554    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7zf7\" (UniqueName: \"kubernetes.io/projected/ec7cf9d3-8eb7-4398-a248-11712584d152-kube-api-access-g7zf7\") pod \"maas-api-revoke-sub-a3e6d015-bs6kp\" (UID: \"ec7cf9d3-8eb7-4398-a248-11712584d152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp"
Nov 03 17:37:45.262372 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.262332    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"]
Nov 03 17:37:45.266705 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.266687    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"
Nov 03 17:37:45.327306 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.326054    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-198bcf954428\""
Nov 03 17:37:45.336597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.336544    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj" podStartSLOduration=2.3365312129999998 podStartE2EDuration="2.336531213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:37:45.332107748 +0000 UTC m=+1206.129820530" watchObservedRunningTime="2026-11-03 17:37:45.336531213 +0000 UTC m=+1206.134243994"
Nov 03 17:37:45.340443 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.340410    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"]
Nov 03 17:37:45.346640 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.346613    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp"
Nov 03 17:37:45.362451 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.362337    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx\" (UID: \"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"
Nov 03 17:37:45.362590 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.362519    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx\" (UID: \"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"
Nov 03 17:37:45.362651 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.362637    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw5hq\" (UniqueName: \"kubernetes.io/projected/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-kube-api-access-xw5hq\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx\" (UID: \"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"
Nov 03 17:37:45.362789 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.362759    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx\" (UID: \"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"
Nov 03 17:37:45.464514 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.464472    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw5hq\" (UniqueName: \"kubernetes.io/projected/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-kube-api-access-xw5hq\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx\" (UID: \"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"
Nov 03 17:37:45.464784 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.464616    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx\" (UID: \"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"
Nov 03 17:37:45.464784 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.464746    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx\" (UID: \"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"
Nov 03 17:37:45.465039 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.465018    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx\" (UID: \"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"
Nov 03 17:37:45.465904 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.465876    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx\" (UID: \"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"
Nov 03 17:37:45.467008 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.466979    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx\" (UID: \"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"
Nov 03 17:37:45.469606 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.469583    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx\" (UID: \"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"
Nov 03 17:37:45.528240 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.528206    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw5hq\" (UniqueName: \"kubernetes.io/projected/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-kube-api-access-xw5hq\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx\" (UID: \"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"
Nov 03 17:37:45.581566 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.581485    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"
Nov 03 17:37:45.734231 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:37:45.734187    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7cf9d3_8eb7_4398_a248_11712584d152.slice/crio-7e0384bd1e1172ff03a59e344dfd04068fa8c42c7c787bbec2df8511c92cbe29 WatchSource:0}: Error finding container 7e0384bd1e1172ff03a59e344dfd04068fa8c42c7c787bbec2df8511c92cbe29: Status 404 returned error can't find the container with id 7e0384bd1e1172ff03a59e344dfd04068fa8c42c7c787bbec2df8511c92cbe29
Nov 03 17:37:45.736231 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.736204    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp"]
Nov 03 17:37:45.964018 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:37:45.963981    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a7b2c4_a543_4cfd_bb16_9cd0b03e0853.slice/crio-1809a4355de510954a2ecc7880107883e99bcdec4636f80d7ab6930208d7569d WatchSource:0}: Error finding container 1809a4355de510954a2ecc7880107883e99bcdec4636f80d7ab6930208d7569d: Status 404 returned error can't find the container with id 1809a4355de510954a2ecc7880107883e99bcdec4636f80d7ab6930208d7569d
Nov 03 17:37:45.969631 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:45.969602    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"]
Nov 03 17:37:46.213904 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:46.213866    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx" event={"ID":"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853","Type":"ContainerStarted","Data":"43ab9a3dcb4dcc1a64aec263f036f2666300e0a1966e6e06866ca2fc63544e0f"}
Nov 03 17:37:46.214345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:46.213922    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx" event={"ID":"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853","Type":"ContainerStarted","Data":"1809a4355de510954a2ecc7880107883e99bcdec4636f80d7ab6930208d7569d"}
Nov 03 17:37:46.214345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:46.214072    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"
Nov 03 17:37:46.215876 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:46.215836    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp" event={"ID":"ec7cf9d3-8eb7-4398-a248-11712584d152","Type":"ContainerStarted","Data":"7910e8bdd85bbebd54b67d2e38601450f2d4fe9a218711bad72cf299180ff0d5"}
Nov 03 17:37:46.215995 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:46.215890    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp" event={"ID":"ec7cf9d3-8eb7-4398-a248-11712584d152","Type":"ContainerStarted","Data":"7e0384bd1e1172ff03a59e344dfd04068fa8c42c7c787bbec2df8511c92cbe29"}
Nov 03 17:37:46.348849 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:46.348791    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx" podStartSLOduration=1.348772864 podStartE2EDuration="1.348772864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:37:46.347773498 +0000 UTC m=+1207.145486279" watchObservedRunningTime="2026-11-03 17:37:46.348772864 +0000 UTC m=+1207.146485648"
Nov 03 17:37:46.479526 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:46.479423    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp" podStartSLOduration=2.47940943 podStartE2EDuration="2.47940943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:37:46.476152003 +0000 UTC m=+1207.273864783" watchObservedRunningTime="2026-11-03 17:37:46.47940943 +0000 UTC m=+1207.277122207"
Nov 03 17:37:47.207626 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:47.207582    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:47.224955 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:47.224919    6059 generic.go:356] "Generic (PLEG): container finished" podID="ec7cf9d3-8eb7-4398-a248-11712584d152" containerID="7910e8bdd85bbebd54b67d2e38601450f2d4fe9a218711bad72cf299180ff0d5" exitCode=0
Nov 03 17:37:47.225348 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:47.224999    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp" event={"ID":"ec7cf9d3-8eb7-4398-a248-11712584d152","Type":"ContainerDied","Data":"7910e8bdd85bbebd54b67d2e38601450f2d4fe9a218711bad72cf299180ff0d5"}
Nov 03 17:37:48.456626 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:48.456599    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp"
Nov 03 17:37:48.514453 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:48.514306    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec7cf9d3-8eb7-4398-a248-11712584d152-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec7cf9d3-8eb7-4398-a248-11712584d152-maas-api-openshift-ca\") pod \"ec7cf9d3-8eb7-4398-a248-11712584d152\" (UID: \"ec7cf9d3-8eb7-4398-a248-11712584d152\") "
Nov 03 17:37:48.514453 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:48.514406    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec7cf9d3-8eb7-4398-a248-11712584d152-kube-api-access-g7zf7\" (UniqueName: \"kubernetes.io/projected/ec7cf9d3-8eb7-4398-a248-11712584d152-kube-api-access-g7zf7\") pod \"ec7cf9d3-8eb7-4398-a248-11712584d152\" (UID: \"ec7cf9d3-8eb7-4398-a248-11712584d152\") "
Nov 03 17:37:48.514453 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:48.514451    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec7cf9d3-8eb7-4398-a248-11712584d152-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec7cf9d3-8eb7-4398-a248-11712584d152-maas-api-xks-ca\") pod \"ec7cf9d3-8eb7-4398-a248-11712584d152\" (UID: \"ec7cf9d3-8eb7-4398-a248-11712584d152\") "
Nov 03 17:37:48.514745 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:48.514692    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec7cf9d3-8eb7-4398-a248-11712584d152-maas-api-openshift-ca" pod "ec7cf9d3-8eb7-4398-a248-11712584d152" (UID: "ec7cf9d3-8eb7-4398-a248-11712584d152"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:37:48.515022 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:48.515001    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec7cf9d3-8eb7-4398-a248-11712584d152-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:48.516668 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:48.516640    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec7cf9d3-8eb7-4398-a248-11712584d152-maas-api-xks-ca" pod "ec7cf9d3-8eb7-4398-a248-11712584d152" (UID: "ec7cf9d3-8eb7-4398-a248-11712584d152"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:37:48.516768 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:48.516657    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7cf9d3-8eb7-4398-a248-11712584d152-kube-api-access-g7zf7" pod "ec7cf9d3-8eb7-4398-a248-11712584d152" (UID: "ec7cf9d3-8eb7-4398-a248-11712584d152"). InnerVolumeSpecName "kube-api-access-g7zf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:37:48.616644 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:48.616602    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7zf7\" (UniqueName: \"kubernetes.io/projected/ec7cf9d3-8eb7-4398-a248-11712584d152-kube-api-access-g7zf7\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:48.616644 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:48.616636    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec7cf9d3-8eb7-4398-a248-11712584d152-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:49.025559 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:49.025518    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:49.100790 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:49.094369    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"]
Nov 03 17:37:49.136383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:49.135592    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"
Nov 03 17:37:49.244470 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:49.244433    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp" event={"ID":"ec7cf9d3-8eb7-4398-a248-11712584d152","Type":"ContainerDied","Data":"7e0384bd1e1172ff03a59e344dfd04068fa8c42c7c787bbec2df8511c92cbe29"}
Nov 03 17:37:49.244645 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:49.244486    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e0384bd1e1172ff03a59e344dfd04068fa8c42c7c787bbec2df8511c92cbe29"
Nov 03 17:37:49.244645 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:49.244443    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp"
Nov 03 17:37:49.259697 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:49.259664    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:51.982925 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:51.982456    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v" podUID="2bbc9082-b2da-4198-8a8a-b6ab26b5011e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 17:37:51.984015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:51.983704    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n" podUID="d6c1263b-e648-4f10-af7d-86397d6b207a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:37:52.032280 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:52.032246    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:52.033701 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:52.033285    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w" podUID="6120f11d-14c8-4666-a201-469a5bca37de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:37:52.184379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:52.184097    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl" podUID="c194a4be-0499-419f-a4a1-910dbaafd303" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:37:52.220385 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:52.219531    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv" podUID="7942e9d1-597f-47a6-9c70-1d174fe4c8e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:37:52.274440 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:52.270962    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:52.987023 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:52.985455    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v" podUID="2bbc9082-b2da-4198-8a8a-b6ab26b5011e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:37:52.987023 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:52.986218    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:37:53.074381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:53.072204    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:37:54.094741 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.094698    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj" podUID="96b48f17-328a-4914-af1f-600f02b88c26" containerName="payload-processing" containerID="cri-o://05b2a039ecbaa8dbda3c57d72847dcae0a7b66fdad0f6c6438ca910c222a945c" gracePeriod=25
Nov 03 17:37:54.316568 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.316538    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:54.358726 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.358665    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"
Nov 03 17:37:54.484034 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.483991    6059 generic.go:356] "Generic (PLEG): container finished" podID="96b48f17-328a-4914-af1f-600f02b88c26" containerID="05b2a039ecbaa8dbda3c57d72847dcae0a7b66fdad0f6c6438ca910c222a945c" exitCode=0
Nov 03 17:37:54.484224 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.484038    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"
Nov 03 17:37:54.484224 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.484058    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj" event={"ID":"96b48f17-328a-4914-af1f-600f02b88c26","Type":"ContainerDied","Data":"05b2a039ecbaa8dbda3c57d72847dcae0a7b66fdad0f6c6438ca910c222a945c"}
Nov 03 17:37:54.484224 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.484124    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj" event={"ID":"96b48f17-328a-4914-af1f-600f02b88c26","Type":"ContainerDied","Data":"2ee6f923c271a6f97afbbe68109931bf9f032977da6569f6e52836211a933409"}
Nov 03 17:37:54.484224 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.484157    6059 scope.go:122] "RemoveContainer" containerID="05b2a039ecbaa8dbda3c57d72847dcae0a7b66fdad0f6c6438ca910c222a945c"
Nov 03 17:37:54.485925 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.485904    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b48f17-328a-4914-af1f-600f02b88c26-kube-api-access-jhh5v\" (UniqueName: \"kubernetes.io/projected/96b48f17-328a-4914-af1f-600f02b88c26-kube-api-access-jhh5v\") pod \"96b48f17-328a-4914-af1f-600f02b88c26\" (UID: \"96b48f17-328a-4914-af1f-600f02b88c26\") "
Nov 03 17:37:54.486067 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.486050    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96b48f17-328a-4914-af1f-600f02b88c26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96b48f17-328a-4914-af1f-600f02b88c26-plugins-config-volume\") pod \"96b48f17-328a-4914-af1f-600f02b88c26\" (UID: \"96b48f17-328a-4914-af1f-600f02b88c26\") "
Nov 03 17:37:54.486157 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.486145    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96b48f17-328a-4914-af1f-600f02b88c26-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96b48f17-328a-4914-af1f-600f02b88c26-routing-overlay-volume\") pod \"96b48f17-328a-4914-af1f-600f02b88c26\" (UID: \"96b48f17-328a-4914-af1f-600f02b88c26\") "
Nov 03 17:37:54.486218 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.486186    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b48f17-328a-4914-af1f-600f02b88c26-provider-credentials\" (UniqueName: \"kubernetes.io/projected/96b48f17-328a-4914-af1f-600f02b88c26-provider-credentials\") pod \"96b48f17-328a-4914-af1f-600f02b88c26\" (UID: \"96b48f17-328a-4914-af1f-600f02b88c26\") "
Nov 03 17:37:54.486971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.486509    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b48f17-328a-4914-af1f-600f02b88c26-plugins-config-volume" pod "96b48f17-328a-4914-af1f-600f02b88c26" (UID: "96b48f17-328a-4914-af1f-600f02b88c26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:37:54.486971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.486529    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b48f17-328a-4914-af1f-600f02b88c26-routing-overlay-volume" pod "96b48f17-328a-4914-af1f-600f02b88c26" (UID: "96b48f17-328a-4914-af1f-600f02b88c26"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:37:54.486971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.486925    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96b48f17-328a-4914-af1f-600f02b88c26-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:54.486971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.486948    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96b48f17-328a-4914-af1f-600f02b88c26-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:54.488833 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.488802    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b48f17-328a-4914-af1f-600f02b88c26-provider-credentials" pod "96b48f17-328a-4914-af1f-600f02b88c26" (UID: "96b48f17-328a-4914-af1f-600f02b88c26"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:37:54.488833 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.488825    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b48f17-328a-4914-af1f-600f02b88c26-kube-api-access-jhh5v" pod "96b48f17-328a-4914-af1f-600f02b88c26" (UID: "96b48f17-328a-4914-af1f-600f02b88c26"). InnerVolumeSpecName "kube-api-access-jhh5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:37:54.518063 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.518036    6059 scope.go:122] "RemoveContainer" containerID="05b2a039ecbaa8dbda3c57d72847dcae0a7b66fdad0f6c6438ca910c222a945c"
Nov 03 17:37:54.518484 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:37:54.518460    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b2a039ecbaa8dbda3c57d72847dcae0a7b66fdad0f6c6438ca910c222a945c\": container with ID starting with 05b2a039ecbaa8dbda3c57d72847dcae0a7b66fdad0f6c6438ca910c222a945c not found: ID does not exist" containerID="05b2a039ecbaa8dbda3c57d72847dcae0a7b66fdad0f6c6438ca910c222a945c"
Nov 03 17:37:54.518591 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.518491    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b2a039ecbaa8dbda3c57d72847dcae0a7b66fdad0f6c6438ca910c222a945c"} err="failed to get container status \"05b2a039ecbaa8dbda3c57d72847dcae0a7b66fdad0f6c6438ca910c222a945c\": rpc error: code = NotFound desc = could not find container \"05b2a039ecbaa8dbda3c57d72847dcae0a7b66fdad0f6c6438ca910c222a945c\": container with ID starting with 05b2a039ecbaa8dbda3c57d72847dcae0a7b66fdad0f6c6438ca910c222a945c not found: ID does not exist"
Nov 03 17:37:54.588287 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.588256    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96b48f17-328a-4914-af1f-600f02b88c26-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:54.588287 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.588285    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhh5v\" (UniqueName: \"kubernetes.io/projected/96b48f17-328a-4914-af1f-600f02b88c26-kube-api-access-jhh5v\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:54.784805 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.784769    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"]
Nov 03 17:37:54.785779 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.785012    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg" podUID="eb7314fe-6d72-4e51-8cdc-acfcce172a1c" containerName="istio-proxy" containerID="cri-o://308332826f9a21a6a96bae42e28098c17477478b49f07103a8829c9893a287db" gracePeriod=30
Nov 03 17:37:54.860008 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:54.859969    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:55.046891 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:55.046806    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"]
Nov 03 17:37:55.051839 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:55.051813    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-64c4scbtj"]
Nov 03 17:37:55.336720 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:55.336641    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96b48f17-328a-4914-af1f-600f02b88c26" path="/var/lib/kubelet/pods/96b48f17-328a-4914-af1f-600f02b88c26/volumes"
Nov 03 17:37:55.682842 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:55.682744    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:55.785926 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:55.785894    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a87a9d-th4lr"]
Nov 03 17:37:55.786490 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:55.786476    6059 cpu_manager.go:414] "containerMap: removing container" podUID="96b48f17-328a-4914-af1f-600f02b88c26" containerName="payload-processing"
Nov 03 17:37:55.786588 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:55.786492    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b48f17-328a-4914-af1f-600f02b88c26" containerName="payload-processing"
Nov 03 17:37:55.786588 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:55.786507    6059 cpu_manager.go:414] "containerMap: removing container" podUID="ec7cf9d3-8eb7-4398-a248-11712584d152" containerName="revoke-keys"
Nov 03 17:37:55.786588 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:55.786512    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec7cf9d3-8eb7-4398-a248-11712584d152" containerName="revoke-keys"
Nov 03 17:37:55.786684 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:55.786625    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b48f17-328a-4914-af1f-600f02b88c26" containerName="payload-processing"
Nov 03 17:37:55.786684 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:55.786636    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec7cf9d3-8eb7-4398-a248-11712584d152" containerName="revoke-keys"
Nov 03 17:37:55.791449 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:55.791430    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a87a9d-th4lr"
Nov 03 17:37:55.846763 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:55.846731    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:37:55.846763 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:55.846742    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:37:55.851472 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:55.851442    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a87a9d-th4lr"]
Nov 03 17:37:55.869412 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:55.869350    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:37:55.897163 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:55.897123    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:37:55.897568 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:55.897549    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:37:55.905960 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:55.905935    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edd16912-5916-4520-a958-943c07fd6626-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51a87a9d-th4lr\" (UID: \"edd16912-5916-4520-a958-943c07fd6626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a87a9d-th4lr"
Nov 03 17:37:55.906101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:55.906084    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edd16912-5916-4520-a958-943c07fd6626-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51a87a9d-th4lr\" (UID: \"edd16912-5916-4520-a958-943c07fd6626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a87a9d-th4lr"
Nov 03 17:37:55.906157 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:55.906122    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh6ct\" (UniqueName: \"kubernetes.io/projected/edd16912-5916-4520-a958-943c07fd6626-kube-api-access-fh6ct\") pod \"maas-api-revoke-sub-51a87a9d-th4lr\" (UID: \"edd16912-5916-4520-a958-943c07fd6626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a87a9d-th4lr"
Nov 03 17:37:56.007670 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:56.007626    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edd16912-5916-4520-a958-943c07fd6626-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51a87a9d-th4lr\" (UID: \"edd16912-5916-4520-a958-943c07fd6626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a87a9d-th4lr"
Nov 03 17:37:56.007888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:56.007731    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edd16912-5916-4520-a958-943c07fd6626-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51a87a9d-th4lr\" (UID: \"edd16912-5916-4520-a958-943c07fd6626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a87a9d-th4lr"
Nov 03 17:37:56.007888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:56.007756    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh6ct\" (UniqueName: \"kubernetes.io/projected/edd16912-5916-4520-a958-943c07fd6626-kube-api-access-fh6ct\") pod \"maas-api-revoke-sub-51a87a9d-th4lr\" (UID: \"edd16912-5916-4520-a958-943c07fd6626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a87a9d-th4lr"
Nov 03 17:37:56.008388 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:56.008343    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edd16912-5916-4520-a958-943c07fd6626-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51a87a9d-th4lr\" (UID: \"edd16912-5916-4520-a958-943c07fd6626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a87a9d-th4lr"
Nov 03 17:37:56.010045 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:56.010022    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edd16912-5916-4520-a958-943c07fd6626-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51a87a9d-th4lr\" (UID: \"edd16912-5916-4520-a958-943c07fd6626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a87a9d-th4lr"
Nov 03 17:37:56.071263 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:56.071234    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh6ct\" (UniqueName: \"kubernetes.io/projected/edd16912-5916-4520-a958-943c07fd6626-kube-api-access-fh6ct\") pod \"maas-api-revoke-sub-51a87a9d-th4lr\" (UID: \"edd16912-5916-4520-a958-943c07fd6626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a87a9d-th4lr"
Nov 03 17:37:56.105497 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:56.105465    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a87a9d-th4lr"
Nov 03 17:37:56.481558 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:37:56.481522    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd16912_5916_4520_a958_943c07fd6626.slice/crio-999ab321ab816da9e94459679126a7038f63a1189ccad0085d0998d9b88949f5 WatchSource:0}: Error finding container 999ab321ab816da9e94459679126a7038f63a1189ccad0085d0998d9b88949f5: Status 404 returned error can't find the container with id 999ab321ab816da9e94459679126a7038f63a1189ccad0085d0998d9b88949f5
Nov 03 17:37:56.485216 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:56.485189    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a87a9d-th4lr"]
Nov 03 17:37:56.502171 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:56.502140    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a87a9d-th4lr" event={"ID":"edd16912-5916-4520-a958-943c07fd6626","Type":"ContainerStarted","Data":"999ab321ab816da9e94459679126a7038f63a1189ccad0085d0998d9b88949f5"}
Nov 03 17:37:57.116688 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:57.116598    6059 patch_prober.go:29] interesting pod/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:37:57.116688 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:57.116664    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg" podUID="eb7314fe-6d72-4e51-8cdc-acfcce172a1c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:37:57.227988 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:57.227958    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"
Nov 03 17:37:57.442388 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:57.442297    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"]
Nov 03 17:37:57.511445 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:57.511403    6059 generic.go:356] "Generic (PLEG): container finished" podID="edd16912-5916-4520-a958-943c07fd6626" containerID="c7b15f24dcd8245ab547509af27bf8c5cc2dc01f93cc8aef4d10ec5d88a71586" exitCode=0
Nov 03 17:37:57.511830 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:57.511494    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a87a9d-th4lr" event={"ID":"edd16912-5916-4520-a958-943c07fd6626","Type":"ContainerDied","Data":"c7b15f24dcd8245ab547509af27bf8c5cc2dc01f93cc8aef4d10ec5d88a71586"}
Nov 03 17:37:58.662502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:58.662472    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a87a9d-th4lr"
Nov 03 17:37:58.742472 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:58.742437    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edd16912-5916-4520-a958-943c07fd6626-kube-api-access-fh6ct\" (UniqueName: \"kubernetes.io/projected/edd16912-5916-4520-a958-943c07fd6626-kube-api-access-fh6ct\") pod \"edd16912-5916-4520-a958-943c07fd6626\" (UID: \"edd16912-5916-4520-a958-943c07fd6626\") "
Nov 03 17:37:58.742643 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:58.742508    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edd16912-5916-4520-a958-943c07fd6626-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edd16912-5916-4520-a958-943c07fd6626-maas-api-openshift-ca\") pod \"edd16912-5916-4520-a958-943c07fd6626\" (UID: \"edd16912-5916-4520-a958-943c07fd6626\") "
Nov 03 17:37:58.742643 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:58.742542    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edd16912-5916-4520-a958-943c07fd6626-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edd16912-5916-4520-a958-943c07fd6626-maas-api-xks-ca\") pod \"edd16912-5916-4520-a958-943c07fd6626\" (UID: \"edd16912-5916-4520-a958-943c07fd6626\") "
Nov 03 17:37:58.742877 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:58.742853    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edd16912-5916-4520-a958-943c07fd6626-maas-api-openshift-ca" pod "edd16912-5916-4520-a958-943c07fd6626" (UID: "edd16912-5916-4520-a958-943c07fd6626"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:37:58.743024 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:58.743008    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edd16912-5916-4520-a958-943c07fd6626-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:58.744654 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:58.744630    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edd16912-5916-4520-a958-943c07fd6626-maas-api-xks-ca" pod "edd16912-5916-4520-a958-943c07fd6626" (UID: "edd16912-5916-4520-a958-943c07fd6626"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:37:58.744759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:58.744689    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edd16912-5916-4520-a958-943c07fd6626-kube-api-access-fh6ct" pod "edd16912-5916-4520-a958-943c07fd6626" (UID: "edd16912-5916-4520-a958-943c07fd6626"). InnerVolumeSpecName "kube-api-access-fh6ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:37:58.844310 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:58.844233    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edd16912-5916-4520-a958-943c07fd6626-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:58.844310 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:58.844262    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh6ct\" (UniqueName: \"kubernetes.io/projected/edd16912-5916-4520-a958-943c07fd6626-kube-api-access-fh6ct\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:37:59.527080 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:59.527053    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a87a9d-th4lr"
Nov 03 17:37:59.527244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:59.527090    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a87a9d-th4lr" event={"ID":"edd16912-5916-4520-a958-943c07fd6626","Type":"ContainerDied","Data":"999ab321ab816da9e94459679126a7038f63a1189ccad0085d0998d9b88949f5"}
Nov 03 17:37:59.527244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:37:59.527139    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="999ab321ab816da9e94459679126a7038f63a1189ccad0085d0998d9b88949f5"
Nov 03 17:38:00.076008 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.075982    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:38:00.158699 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.158669    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-kube-api-access-v8zx2\" (UniqueName: \"kubernetes.io/projected/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-kube-api-access-v8zx2\") pod \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") "
Nov 03 17:38:00.158879 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.158850    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-workload-socket\") pod \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") "
Nov 03 17:38:00.158931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.158890    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-workload-certs\") pod \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") "
Nov 03 17:38:00.158931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.158917    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-token\" (UniqueName: \"kubernetes.io/projected/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-token\") pod \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") "
Nov 03 17:38:00.159015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.158946    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-envoy\") pod \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") "
Nov 03 17:38:00.159015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.158975    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-podinfo\") pod \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") "
Nov 03 17:38:00.159120 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.159023    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-credential-socket\") pod \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") "
Nov 03 17:38:00.159120 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.159065    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istiod-ca-cert\") pod \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") "
Nov 03 17:38:00.159120 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.159093    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-data\") pod \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\" (UID: \"eb7314fe-6d72-4e51-8cdc-acfcce172a1c\") "
Nov 03 17:38:00.159325 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.159193    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-workload-socket" pod "eb7314fe-6d72-4e51-8cdc-acfcce172a1c" (UID: "eb7314fe-6d72-4e51-8cdc-acfcce172a1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:38:00.159325 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.159208    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-workload-certs" pod "eb7314fe-6d72-4e51-8cdc-acfcce172a1c" (UID: "eb7314fe-6d72-4e51-8cdc-acfcce172a1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:38:00.159478 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.159454    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-credential-socket" pod "eb7314fe-6d72-4e51-8cdc-acfcce172a1c" (UID: "eb7314fe-6d72-4e51-8cdc-acfcce172a1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:38:00.159611 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.159559    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-data" pod "eb7314fe-6d72-4e51-8cdc-acfcce172a1c" (UID: "eb7314fe-6d72-4e51-8cdc-acfcce172a1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:38:00.159696 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.159665    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istiod-ca-cert" pod "eb7314fe-6d72-4e51-8cdc-acfcce172a1c" (UID: "eb7314fe-6d72-4e51-8cdc-acfcce172a1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:38:00.160017 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.159991    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:00.160117 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.160021    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:00.160117 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.160036    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:00.160117 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.160054    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:00.160117 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.160068    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:00.161134 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.161109    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-kube-api-access-v8zx2" pod "eb7314fe-6d72-4e51-8cdc-acfcce172a1c" (UID: "eb7314fe-6d72-4e51-8cdc-acfcce172a1c"). InnerVolumeSpecName "kube-api-access-v8zx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:38:00.161299 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.161283    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-envoy" pod "eb7314fe-6d72-4e51-8cdc-acfcce172a1c" (UID: "eb7314fe-6d72-4e51-8cdc-acfcce172a1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:38:00.161545 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.161525    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-podinfo" pod "eb7314fe-6d72-4e51-8cdc-acfcce172a1c" (UID: "eb7314fe-6d72-4e51-8cdc-acfcce172a1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:38:00.161634 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.161591    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-token" pod "eb7314fe-6d72-4e51-8cdc-acfcce172a1c" (UID: "eb7314fe-6d72-4e51-8cdc-acfcce172a1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:38:00.233851 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.233817    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:00.261772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.261744    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:00.261772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.261771    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:00.261956 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.261782    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:00.261956 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.261791    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8zx2\" (UniqueName: \"kubernetes.io/projected/eb7314fe-6d72-4e51-8cdc-acfcce172a1c-kube-api-access-v8zx2\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:00.415643 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.415560    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:00.538144 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.538104    6059 generic.go:356] "Generic (PLEG): container finished" podID="eb7314fe-6d72-4e51-8cdc-acfcce172a1c" containerID="308332826f9a21a6a96bae42e28098c17477478b49f07103a8829c9893a287db" exitCode=0
Nov 03 17:38:00.538331 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.538148    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg" event={"ID":"eb7314fe-6d72-4e51-8cdc-acfcce172a1c","Type":"ContainerDied","Data":"308332826f9a21a6a96bae42e28098c17477478b49f07103a8829c9893a287db"}
Nov 03 17:38:00.538331 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.538190    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"
Nov 03 17:38:00.538331 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.538216    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg" event={"ID":"eb7314fe-6d72-4e51-8cdc-acfcce172a1c","Type":"ContainerDied","Data":"7e0d12c6717ebf6d1c7cfa33ac45e02f971ec2bcdb35ade4913457d65eef9554"}
Nov 03 17:38:00.538331 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.538244    6059 scope.go:122] "RemoveContainer" containerID="308332826f9a21a6a96bae42e28098c17477478b49f07103a8829c9893a287db"
Nov 03 17:38:00.550662 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.550641    6059 scope.go:122] "RemoveContainer" containerID="6c6d1fd85a52eddd9360384fa3b3428da88fdf0b81e3d9ba4f4e47d8b5e80bfd"
Nov 03 17:38:00.560836 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.560816    6059 scope.go:122] "RemoveContainer" containerID="308332826f9a21a6a96bae42e28098c17477478b49f07103a8829c9893a287db"
Nov 03 17:38:00.561120 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:38:00.561099    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"308332826f9a21a6a96bae42e28098c17477478b49f07103a8829c9893a287db\": container with ID starting with 308332826f9a21a6a96bae42e28098c17477478b49f07103a8829c9893a287db not found: ID does not exist" containerID="308332826f9a21a6a96bae42e28098c17477478b49f07103a8829c9893a287db"
Nov 03 17:38:00.561161 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.561129    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308332826f9a21a6a96bae42e28098c17477478b49f07103a8829c9893a287db"} err="failed to get container status \"308332826f9a21a6a96bae42e28098c17477478b49f07103a8829c9893a287db\": rpc error: code = NotFound desc = could not find container \"308332826f9a21a6a96bae42e28098c17477478b49f07103a8829c9893a287db\": container with ID starting with 308332826f9a21a6a96bae42e28098c17477478b49f07103a8829c9893a287db not found: ID does not exist"
Nov 03 17:38:00.561161 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.561146    6059 scope.go:122] "RemoveContainer" containerID="6c6d1fd85a52eddd9360384fa3b3428da88fdf0b81e3d9ba4f4e47d8b5e80bfd"
Nov 03 17:38:00.561446 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:38:00.561426    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6d1fd85a52eddd9360384fa3b3428da88fdf0b81e3d9ba4f4e47d8b5e80bfd\": container with ID starting with 6c6d1fd85a52eddd9360384fa3b3428da88fdf0b81e3d9ba4f4e47d8b5e80bfd not found: ID does not exist" containerID="6c6d1fd85a52eddd9360384fa3b3428da88fdf0b81e3d9ba4f4e47d8b5e80bfd"
Nov 03 17:38:00.561494 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.561452    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6d1fd85a52eddd9360384fa3b3428da88fdf0b81e3d9ba4f4e47d8b5e80bfd"} err="failed to get container status \"6c6d1fd85a52eddd9360384fa3b3428da88fdf0b81e3d9ba4f4e47d8b5e80bfd\": rpc error: code = NotFound desc = could not find container \"6c6d1fd85a52eddd9360384fa3b3428da88fdf0b81e3d9ba4f4e47d8b5e80bfd\": container with ID starting with 6c6d1fd85a52eddd9360384fa3b3428da88fdf0b81e3d9ba4f4e47d8b5e80bfd not found: ID does not exist"
Nov 03 17:38:00.714973 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.714943    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"]
Nov 03 17:38:00.766554 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:00.766522    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-855xg"]
Nov 03 17:38:01.336117 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:01.336081    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb7314fe-6d72-4e51-8cdc-acfcce172a1c" path="/var/lib/kubelet/pods/eb7314fe-6d72-4e51-8cdc-acfcce172a1c/volumes"
Nov 03 17:38:01.978289 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:01.978250    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh"]
Nov 03 17:38:02.088200 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:02.088161    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68"]
Nov 03 17:38:02.091641 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:02.091617    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9806e398-cbp68"]
Nov 03 17:38:02.095011 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:02.094968    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x"]
Nov 03 17:38:02.240536 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:02.240439    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:02.443448 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:02.443407    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7" podUID="729411cc-0c94-4c63-9e1b-1efc7273c3df" containerName="payload-processing" containerID="cri-o://c1c73c8c795a2ef1c9530447a0deadf297d6719e04afbac38bcea5cc219fe22d" gracePeriod=25
Nov 03 17:38:02.525080 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:02.524969    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:02.698999 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:02.698972    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"
Nov 03 17:38:02.713496 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:02.713465    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:02.792669 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:02.792587    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/729411cc-0c94-4c63-9e1b-1efc7273c3df-provider-credentials\" (UniqueName: \"kubernetes.io/projected/729411cc-0c94-4c63-9e1b-1efc7273c3df-provider-credentials\") pod \"729411cc-0c94-4c63-9e1b-1efc7273c3df\" (UID: \"729411cc-0c94-4c63-9e1b-1efc7273c3df\") "
Nov 03 17:38:02.792831 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:02.792691    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/729411cc-0c94-4c63-9e1b-1efc7273c3df-kube-api-access-4gj9x\" (UniqueName: \"kubernetes.io/projected/729411cc-0c94-4c63-9e1b-1efc7273c3df-kube-api-access-4gj9x\") pod \"729411cc-0c94-4c63-9e1b-1efc7273c3df\" (UID: \"729411cc-0c94-4c63-9e1b-1efc7273c3df\") "
Nov 03 17:38:02.792871 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:02.792823    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/729411cc-0c94-4c63-9e1b-1efc7273c3df-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/729411cc-0c94-4c63-9e1b-1efc7273c3df-routing-overlay-volume\") pod \"729411cc-0c94-4c63-9e1b-1efc7273c3df\" (UID: \"729411cc-0c94-4c63-9e1b-1efc7273c3df\") "
Nov 03 17:38:02.792949 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:02.792931    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/729411cc-0c94-4c63-9e1b-1efc7273c3df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729411cc-0c94-4c63-9e1b-1efc7273c3df-plugins-config-volume\") pod \"729411cc-0c94-4c63-9e1b-1efc7273c3df\" (UID: \"729411cc-0c94-4c63-9e1b-1efc7273c3df\") "
Nov 03 17:38:02.793714 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:02.793260    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729411cc-0c94-4c63-9e1b-1efc7273c3df-routing-overlay-volume" pod "729411cc-0c94-4c63-9e1b-1efc7273c3df" (UID: "729411cc-0c94-4c63-9e1b-1efc7273c3df"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:38:02.793714 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:02.793336    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729411cc-0c94-4c63-9e1b-1efc7273c3df-plugins-config-volume" pod "729411cc-0c94-4c63-9e1b-1efc7273c3df" (UID: "729411cc-0c94-4c63-9e1b-1efc7273c3df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:38:02.793714 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:02.793663    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/729411cc-0c94-4c63-9e1b-1efc7273c3df-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:02.793714 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:02.793687    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729411cc-0c94-4c63-9e1b-1efc7273c3df-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:02.794838 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:02.794815    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729411cc-0c94-4c63-9e1b-1efc7273c3df-provider-credentials" pod "729411cc-0c94-4c63-9e1b-1efc7273c3df" (UID: "729411cc-0c94-4c63-9e1b-1efc7273c3df"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:38:02.795076 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:02.795056    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729411cc-0c94-4c63-9e1b-1efc7273c3df-kube-api-access-4gj9x" pod "729411cc-0c94-4c63-9e1b-1efc7273c3df" (UID: "729411cc-0c94-4c63-9e1b-1efc7273c3df"). InnerVolumeSpecName "kube-api-access-4gj9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:38:02.895160 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:02.895094    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/729411cc-0c94-4c63-9e1b-1efc7273c3df-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:02.895160 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:02.895125    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gj9x\" (UniqueName: \"kubernetes.io/projected/729411cc-0c94-4c63-9e1b-1efc7273c3df-kube-api-access-4gj9x\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:03.367217 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:03.367170    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31608ad3-ae05-4e9c-80ce-4bb324b45f0f" path="/var/lib/kubelet/pods/31608ad3-ae05-4e9c-80ce-4bb324b45f0f/volumes"
Nov 03 17:38:03.563159 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:03.563124    6059 generic.go:356] "Generic (PLEG): container finished" podID="729411cc-0c94-4c63-9e1b-1efc7273c3df" containerID="c1c73c8c795a2ef1c9530447a0deadf297d6719e04afbac38bcea5cc219fe22d" exitCode=0
Nov 03 17:38:03.563632 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:03.563173    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"
Nov 03 17:38:03.563632 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:03.563211    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7" event={"ID":"729411cc-0c94-4c63-9e1b-1efc7273c3df","Type":"ContainerDied","Data":"c1c73c8c795a2ef1c9530447a0deadf297d6719e04afbac38bcea5cc219fe22d"}
Nov 03 17:38:03.563632 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:03.563267    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7" event={"ID":"729411cc-0c94-4c63-9e1b-1efc7273c3df","Type":"ContainerDied","Data":"a71cd57141ef55acf3a47c21c90c1be695a8bb63877c2642f8eaecf478496fa2"}
Nov 03 17:38:03.563632 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:03.563296    6059 scope.go:122] "RemoveContainer" containerID="c1c73c8c795a2ef1c9530447a0deadf297d6719e04afbac38bcea5cc219fe22d"
Nov 03 17:38:03.576308 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:03.576285    6059 scope.go:122] "RemoveContainer" containerID="c1c73c8c795a2ef1c9530447a0deadf297d6719e04afbac38bcea5cc219fe22d"
Nov 03 17:38:03.576630 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:38:03.576608    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c73c8c795a2ef1c9530447a0deadf297d6719e04afbac38bcea5cc219fe22d\": container with ID starting with c1c73c8c795a2ef1c9530447a0deadf297d6719e04afbac38bcea5cc219fe22d not found: ID does not exist" containerID="c1c73c8c795a2ef1c9530447a0deadf297d6719e04afbac38bcea5cc219fe22d"
Nov 03 17:38:03.576690 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:03.576640    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c73c8c795a2ef1c9530447a0deadf297d6719e04afbac38bcea5cc219fe22d"} err="failed to get container status \"c1c73c8c795a2ef1c9530447a0deadf297d6719e04afbac38bcea5cc219fe22d\": rpc error: code = NotFound desc = could not find container \"c1c73c8c795a2ef1c9530447a0deadf297d6719e04afbac38bcea5cc219fe22d\": container with ID starting with c1c73c8c795a2ef1c9530447a0deadf297d6719e04afbac38bcea5cc219fe22d not found: ID does not exist"
Nov 03 17:38:03.735331 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:03.735298    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"]
Nov 03 17:38:03.787027 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:03.786987    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bbpq4t7"]
Nov 03 17:38:05.078761 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:05.078724    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:05.337276 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:05.337180    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="729411cc-0c94-4c63-9e1b-1efc7273c3df" path="/var/lib/kubelet/pods/729411cc-0c94-4c63-9e1b-1efc7273c3df/volumes"
Nov 03 17:38:06.862345 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:38:06.862305    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:38:06.979175 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:06.979129    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh" podUID="9f7d5da9-069f-4387-b93d-3f3687d7a43a" containerName="payload-processing" containerID="cri-o://e4de9fb28f654396dfaf7a76570b3a18e59f4595c7010acf53168a477f2c78af" gracePeriod=25
Nov 03 17:38:07.096296 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:07.096253    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x" podUID="15e65883-b429-48a8-85df-f5122c5aa67d" containerName="payload-pre-processing" containerID="cri-o://9d31a7d0d7f3a2c9674accf9cb41a6b2f50ee0f810b96d58b4b8cf4425fa535e" gracePeriod=25
Nov 03 17:38:09.100819 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:09.100783    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:10.048661 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:10.045496    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w" podUID="6120f11d-14c8-4666-a201-469a5bca37de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded"
Nov 03 17:38:10.058533 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:10.058480    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n" podUID="d6c1263b-e648-4f10-af7d-86397d6b207a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:38:10.185627 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:10.185574    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl" podUID="c194a4be-0499-419f-a4a1-910dbaafd303" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:38:10.219505 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:10.219442    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv" podUID="7942e9d1-597f-47a6-9c70-1d174fe4c8e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:38:10.587793 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:10.587749    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:38:10.588617 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:10.588585    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5" podUID="63c795d8-dfff-43d8-88a2-de65cc01a51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:38:11.005334 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:11.005298    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:11.050897 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:11.050852    6059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n" podUID="d6c1263b-e648-4f10-af7d-86397d6b207a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:38:11.051540 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:11.051505    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w" podUID="6120f11d-14c8-4666-a201-469a5bca37de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:38:11.052076 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:11.052054    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:38:11.070379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:11.066600    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n" podUID="d6c1263b-e648-4f10-af7d-86397d6b207a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded"
Nov 03 17:38:11.070379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:11.067119    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:38:11.150829 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:11.150797    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:38:11.151821 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:11.151783    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:38:13.015777 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.015741    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj"]
Nov 03 17:38:13.016333 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.016320    6059 cpu_manager.go:414] "containerMap: removing container" podUID="eb7314fe-6d72-4e51-8cdc-acfcce172a1c" containerName="istio-proxy"
Nov 03 17:38:13.016403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.016334    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb7314fe-6d72-4e51-8cdc-acfcce172a1c" containerName="istio-proxy"
Nov 03 17:38:13.016403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.016350    6059 cpu_manager.go:414] "containerMap: removing container" podUID="eb7314fe-6d72-4e51-8cdc-acfcce172a1c" containerName="istio-proxy"
Nov 03 17:38:13.016403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.016369    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb7314fe-6d72-4e51-8cdc-acfcce172a1c" containerName="istio-proxy"
Nov 03 17:38:13.016403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.016378    6059 cpu_manager.go:414] "containerMap: removing container" podUID="729411cc-0c94-4c63-9e1b-1efc7273c3df" containerName="payload-processing"
Nov 03 17:38:13.016403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.016383    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="729411cc-0c94-4c63-9e1b-1efc7273c3df" containerName="payload-processing"
Nov 03 17:38:13.016403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.016395    6059 cpu_manager.go:414] "containerMap: removing container" podUID="edd16912-5916-4520-a958-943c07fd6626" containerName="revoke-keys"
Nov 03 17:38:13.016403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.016400    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edd16912-5916-4520-a958-943c07fd6626" containerName="revoke-keys"
Nov 03 17:38:13.016617 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.016484    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb7314fe-6d72-4e51-8cdc-acfcce172a1c" containerName="istio-proxy"
Nov 03 17:38:13.016617 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.016494    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb7314fe-6d72-4e51-8cdc-acfcce172a1c" containerName="istio-proxy"
Nov 03 17:38:13.016617 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.016501    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="729411cc-0c94-4c63-9e1b-1efc7273c3df" containerName="payload-processing"
Nov 03 17:38:13.016617 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.016507    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="edd16912-5916-4520-a958-943c07fd6626" containerName="revoke-keys"
Nov 03 17:38:13.021556 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.021538    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj"
Nov 03 17:38:13.080720 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.080687    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj"]
Nov 03 17:38:13.115585 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.115544    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc18e10-5b81-4b42-8e65-16f29e49b2c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17d4d4c7-mgwfj\" (UID: \"6bc18e10-5b81-4b42-8e65-16f29e49b2c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj"
Nov 03 17:38:13.115739 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.115714    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc18e10-5b81-4b42-8e65-16f29e49b2c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17d4d4c7-mgwfj\" (UID: \"6bc18e10-5b81-4b42-8e65-16f29e49b2c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj"
Nov 03 17:38:13.115776 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.115738    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xsrj\" (UniqueName: \"kubernetes.io/projected/6bc18e10-5b81-4b42-8e65-16f29e49b2c5-kube-api-access-4xsrj\") pod \"maas-api-revoke-sub-17d4d4c7-mgwfj\" (UID: \"6bc18e10-5b81-4b42-8e65-16f29e49b2c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj"
Nov 03 17:38:13.127449 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.127418    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:38:13.127449 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.127417    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:38:13.127449 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.127444    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:38:13.127683 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.127474    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:38:13.216736 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.216703    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc18e10-5b81-4b42-8e65-16f29e49b2c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17d4d4c7-mgwfj\" (UID: \"6bc18e10-5b81-4b42-8e65-16f29e49b2c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj"
Nov 03 17:38:13.216940 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.216751    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xsrj\" (UniqueName: \"kubernetes.io/projected/6bc18e10-5b81-4b42-8e65-16f29e49b2c5-kube-api-access-4xsrj\") pod \"maas-api-revoke-sub-17d4d4c7-mgwfj\" (UID: \"6bc18e10-5b81-4b42-8e65-16f29e49b2c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj"
Nov 03 17:38:13.216940 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.216897    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc18e10-5b81-4b42-8e65-16f29e49b2c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17d4d4c7-mgwfj\" (UID: \"6bc18e10-5b81-4b42-8e65-16f29e49b2c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj"
Nov 03 17:38:13.217583 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.217550    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc18e10-5b81-4b42-8e65-16f29e49b2c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17d4d4c7-mgwfj\" (UID: \"6bc18e10-5b81-4b42-8e65-16f29e49b2c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj"
Nov 03 17:38:13.219236 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.219213    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc18e10-5b81-4b42-8e65-16f29e49b2c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17d4d4c7-mgwfj\" (UID: \"6bc18e10-5b81-4b42-8e65-16f29e49b2c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj"
Nov 03 17:38:13.280333 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.280258    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xsrj\" (UniqueName: \"kubernetes.io/projected/6bc18e10-5b81-4b42-8e65-16f29e49b2c5-kube-api-access-4xsrj\") pod \"maas-api-revoke-sub-17d4d4c7-mgwfj\" (UID: \"6bc18e10-5b81-4b42-8e65-16f29e49b2c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj"
Nov 03 17:38:13.334618 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.334584    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj"
Nov 03 17:38:13.708034 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:38:13.707989    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc18e10_5b81_4b42_8e65_16f29e49b2c5.slice/crio-89d27af32280f01e20b0d3c5a02dba7871d24906058a42a94b4ea95fc47a00ee WatchSource:0}: Error finding container 89d27af32280f01e20b0d3c5a02dba7871d24906058a42a94b4ea95fc47a00ee: Status 404 returned error can't find the container with id 89d27af32280f01e20b0d3c5a02dba7871d24906058a42a94b4ea95fc47a00ee
Nov 03 17:38:13.711417 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:13.711397    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj"]
Nov 03 17:38:14.322208 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:14.322115    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj" event={"ID":"6bc18e10-5b81-4b42-8e65-16f29e49b2c5","Type":"ContainerStarted","Data":"c2bc45c1c5d7f9a1f9ebbc293142be7d2b2d6c887090a97e85d2d427a4ef738f"}
Nov 03 17:38:14.322208 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:14.322204    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj" event={"ID":"6bc18e10-5b81-4b42-8e65-16f29e49b2c5","Type":"ContainerStarted","Data":"89d27af32280f01e20b0d3c5a02dba7871d24906058a42a94b4ea95fc47a00ee"}
Nov 03 17:38:14.444387 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:14.444324    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj" podStartSLOduration=2.444310631 podStartE2EDuration="2.444310631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:38:14.441795132 +0000 UTC m=+1235.239507915" watchObservedRunningTime="2026-11-03 17:38:14.444310631 +0000 UTC m=+1235.242023408"
Nov 03 17:38:14.568508 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:14.568467    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x" podUID="15e65883-b429-48a8-85df-f5122c5aa67d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 17:38:14.568653 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:14.568543    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh" podUID="9f7d5da9-069f-4387-b93d-3f3687d7a43a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 17:38:15.333047 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:15.333008    6059 generic.go:356] "Generic (PLEG): container finished" podID="6bc18e10-5b81-4b42-8e65-16f29e49b2c5" containerID="c2bc45c1c5d7f9a1f9ebbc293142be7d2b2d6c887090a97e85d2d427a4ef738f" exitCode=0
Nov 03 17:38:15.336710 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:15.336685    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj" event={"ID":"6bc18e10-5b81-4b42-8e65-16f29e49b2c5","Type":"ContainerDied","Data":"c2bc45c1c5d7f9a1f9ebbc293142be7d2b2d6c887090a97e85d2d427a4ef738f"}
Nov 03 17:38:15.732239 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:15.732163    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:16.490367 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:16.490332    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj"
Nov 03 17:38:16.558572 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:16.558546    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc18e10-5b81-4b42-8e65-16f29e49b2c5-kube-api-access-4xsrj\" (UniqueName: \"kubernetes.io/projected/6bc18e10-5b81-4b42-8e65-16f29e49b2c5-kube-api-access-4xsrj\") pod \"6bc18e10-5b81-4b42-8e65-16f29e49b2c5\" (UID: \"6bc18e10-5b81-4b42-8e65-16f29e49b2c5\") "
Nov 03 17:38:16.558735 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:16.558664    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bc18e10-5b81-4b42-8e65-16f29e49b2c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc18e10-5b81-4b42-8e65-16f29e49b2c5-maas-api-openshift-ca\") pod \"6bc18e10-5b81-4b42-8e65-16f29e49b2c5\" (UID: \"6bc18e10-5b81-4b42-8e65-16f29e49b2c5\") "
Nov 03 17:38:16.558735 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:16.558707    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bc18e10-5b81-4b42-8e65-16f29e49b2c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc18e10-5b81-4b42-8e65-16f29e49b2c5-maas-api-xks-ca\") pod \"6bc18e10-5b81-4b42-8e65-16f29e49b2c5\" (UID: \"6bc18e10-5b81-4b42-8e65-16f29e49b2c5\") "
Nov 03 17:38:16.559048 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:16.559024    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc18e10-5b81-4b42-8e65-16f29e49b2c5-maas-api-openshift-ca" pod "6bc18e10-5b81-4b42-8e65-16f29e49b2c5" (UID: "6bc18e10-5b81-4b42-8e65-16f29e49b2c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:38:16.559141 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:16.559127    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc18e10-5b81-4b42-8e65-16f29e49b2c5-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:16.560701 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:16.560673    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc18e10-5b81-4b42-8e65-16f29e49b2c5-kube-api-access-4xsrj" pod "6bc18e10-5b81-4b42-8e65-16f29e49b2c5" (UID: "6bc18e10-5b81-4b42-8e65-16f29e49b2c5"). InnerVolumeSpecName "kube-api-access-4xsrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:38:16.560943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:16.560924    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bc18e10-5b81-4b42-8e65-16f29e49b2c5-maas-api-xks-ca" pod "6bc18e10-5b81-4b42-8e65-16f29e49b2c5" (UID: "6bc18e10-5b81-4b42-8e65-16f29e49b2c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:38:16.660936 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:16.660854    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc18e10-5b81-4b42-8e65-16f29e49b2c5-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:16.660936 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:16.660887    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xsrj\" (UniqueName: \"kubernetes.io/projected/6bc18e10-5b81-4b42-8e65-16f29e49b2c5-kube-api-access-4xsrj\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:17.349493 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:17.349464    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj"
Nov 03 17:38:17.349636 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:17.349498    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj" event={"ID":"6bc18e10-5b81-4b42-8e65-16f29e49b2c5","Type":"ContainerDied","Data":"89d27af32280f01e20b0d3c5a02dba7871d24906058a42a94b4ea95fc47a00ee"}
Nov 03 17:38:17.349636 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:17.349551    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89d27af32280f01e20b0d3c5a02dba7871d24906058a42a94b4ea95fc47a00ee"
Nov 03 17:38:20.364677 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:20.364594    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:24.567973 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:24.567930    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x" podUID="15e65883-b429-48a8-85df-f5122c5aa67d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 17:38:24.567973 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:24.567963    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh" podUID="9f7d5da9-069f-4387-b93d-3f3687d7a43a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 17:38:25.036860 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:25.036806    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:25.995519 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:25.995484    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln"]
Nov 03 17:38:25.996112 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:25.996097    6059 cpu_manager.go:414] "containerMap: removing container" podUID="6bc18e10-5b81-4b42-8e65-16f29e49b2c5" containerName="revoke-keys"
Nov 03 17:38:25.996159 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:25.996113    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc18e10-5b81-4b42-8e65-16f29e49b2c5" containerName="revoke-keys"
Nov 03 17:38:25.996215 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:25.996204    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc18e10-5b81-4b42-8e65-16f29e49b2c5" containerName="revoke-keys"
Nov 03 17:38:25.999933 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:25.999913    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln"
Nov 03 17:38:26.107750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:26.107710    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:38:26.107937 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:26.107822    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:38:26.107937 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:26.107825    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:38:26.107937 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:26.107886    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:38:26.111677 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:26.111644    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln"]
Nov 03 17:38:26.141404 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:26.141335    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4hxl\" (UniqueName: \"kubernetes.io/projected/436e4e0c-365c-42e6-a646-424b167a7686-kube-api-access-b4hxl\") pod \"maas-api-revoke-sub-7d1f969b-flgln\" (UID: \"436e4e0c-365c-42e6-a646-424b167a7686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln"
Nov 03 17:38:26.141548 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:26.141496    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/436e4e0c-365c-42e6-a646-424b167a7686-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d1f969b-flgln\" (UID: \"436e4e0c-365c-42e6-a646-424b167a7686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln"
Nov 03 17:38:26.141605 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:26.141544    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/436e4e0c-365c-42e6-a646-424b167a7686-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d1f969b-flgln\" (UID: \"436e4e0c-365c-42e6-a646-424b167a7686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln"
Nov 03 17:38:26.242436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:26.242398    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/436e4e0c-365c-42e6-a646-424b167a7686-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d1f969b-flgln\" (UID: \"436e4e0c-365c-42e6-a646-424b167a7686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln"
Nov 03 17:38:26.242614 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:26.242534    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4hxl\" (UniqueName: \"kubernetes.io/projected/436e4e0c-365c-42e6-a646-424b167a7686-kube-api-access-b4hxl\") pod \"maas-api-revoke-sub-7d1f969b-flgln\" (UID: \"436e4e0c-365c-42e6-a646-424b167a7686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln"
Nov 03 17:38:26.242614 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:26.242591    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/436e4e0c-365c-42e6-a646-424b167a7686-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d1f969b-flgln\" (UID: \"436e4e0c-365c-42e6-a646-424b167a7686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln"
Nov 03 17:38:26.243146 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:26.243122    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/436e4e0c-365c-42e6-a646-424b167a7686-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d1f969b-flgln\" (UID: \"436e4e0c-365c-42e6-a646-424b167a7686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln"
Nov 03 17:38:26.244894 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:26.244872    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/436e4e0c-365c-42e6-a646-424b167a7686-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d1f969b-flgln\" (UID: \"436e4e0c-365c-42e6-a646-424b167a7686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln"
Nov 03 17:38:26.306027 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:26.305944    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4hxl\" (UniqueName: \"kubernetes.io/projected/436e4e0c-365c-42e6-a646-424b167a7686-kube-api-access-b4hxl\") pod \"maas-api-revoke-sub-7d1f969b-flgln\" (UID: \"436e4e0c-365c-42e6-a646-424b167a7686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln"
Nov 03 17:38:26.314520 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:26.314493    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln"
Nov 03 17:38:26.693135 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:38:26.693103    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436e4e0c_365c_42e6_a646_424b167a7686.slice/crio-08c0a9379808eb1eedb453807b014986822ae4de86f1296275cb0426a8136379 WatchSource:0}: Error finding container 08c0a9379808eb1eedb453807b014986822ae4de86f1296275cb0426a8136379: Status 404 returned error can't find the container with id 08c0a9379808eb1eedb453807b014986822ae4de86f1296275cb0426a8136379
Nov 03 17:38:26.696510 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:26.696485    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln"]
Nov 03 17:38:27.234559 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.234534    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh"
Nov 03 17:38:27.358114 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.358085    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f7d5da9-069f-4387-b93d-3f3687d7a43a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7d5da9-069f-4387-b93d-3f3687d7a43a-plugins-config-volume\") pod \"9f7d5da9-069f-4387-b93d-3f3687d7a43a\" (UID: \"9f7d5da9-069f-4387-b93d-3f3687d7a43a\") "
Nov 03 17:38:27.358270 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.358180    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f7d5da9-069f-4387-b93d-3f3687d7a43a-kube-api-access-jv48g\" (UniqueName: \"kubernetes.io/projected/9f7d5da9-069f-4387-b93d-3f3687d7a43a-kube-api-access-jv48g\") pod \"9f7d5da9-069f-4387-b93d-3f3687d7a43a\" (UID: \"9f7d5da9-069f-4387-b93d-3f3687d7a43a\") "
Nov 03 17:38:27.358488 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.358467    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f7d5da9-069f-4387-b93d-3f3687d7a43a-plugins-config-volume" pod "9f7d5da9-069f-4387-b93d-3f3687d7a43a" (UID: "9f7d5da9-069f-4387-b93d-3f3687d7a43a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:38:27.358647 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.358626    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7d5da9-069f-4387-b93d-3f3687d7a43a-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:27.360845 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.360811    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7d5da9-069f-4387-b93d-3f3687d7a43a-kube-api-access-jv48g" pod "9f7d5da9-069f-4387-b93d-3f3687d7a43a" (UID: "9f7d5da9-069f-4387-b93d-3f3687d7a43a"). InnerVolumeSpecName "kube-api-access-jv48g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:38:27.378188 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.378162    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x"
Nov 03 17:38:27.459141 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.459106    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15e65883-b429-48a8-85df-f5122c5aa67d-kube-api-access-wbl8k\" (UniqueName: \"kubernetes.io/projected/15e65883-b429-48a8-85df-f5122c5aa67d-kube-api-access-wbl8k\") pod \"15e65883-b429-48a8-85df-f5122c5aa67d\" (UID: \"15e65883-b429-48a8-85df-f5122c5aa67d\") "
Nov 03 17:38:27.459316 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.459294    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15e65883-b429-48a8-85df-f5122c5aa67d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e65883-b429-48a8-85df-f5122c5aa67d-plugins-config-volume\") pod \"15e65883-b429-48a8-85df-f5122c5aa67d\" (UID: \"15e65883-b429-48a8-85df-f5122c5aa67d\") "
Nov 03 17:38:27.459646 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.459630    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv48g\" (UniqueName: \"kubernetes.io/projected/9f7d5da9-069f-4387-b93d-3f3687d7a43a-kube-api-access-jv48g\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:27.459748 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.459663    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15e65883-b429-48a8-85df-f5122c5aa67d-plugins-config-volume" pod "15e65883-b429-48a8-85df-f5122c5aa67d" (UID: "15e65883-b429-48a8-85df-f5122c5aa67d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:38:27.461231 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.461203    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e65883-b429-48a8-85df-f5122c5aa67d-kube-api-access-wbl8k" pod "15e65883-b429-48a8-85df-f5122c5aa67d" (UID: "15e65883-b429-48a8-85df-f5122c5aa67d"). InnerVolumeSpecName "kube-api-access-wbl8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:38:27.514475 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.514435    6059 generic.go:356] "Generic (PLEG): container finished" podID="15e65883-b429-48a8-85df-f5122c5aa67d" containerID="9d31a7d0d7f3a2c9674accf9cb41a6b2f50ee0f810b96d58b4b8cf4425fa535e" exitCode=0
Nov 03 17:38:27.514663 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.514536    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x" event={"ID":"15e65883-b429-48a8-85df-f5122c5aa67d","Type":"ContainerDied","Data":"9d31a7d0d7f3a2c9674accf9cb41a6b2f50ee0f810b96d58b4b8cf4425fa535e"}
Nov 03 17:38:27.514663 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.514581    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x" event={"ID":"15e65883-b429-48a8-85df-f5122c5aa67d","Type":"ContainerDied","Data":"e24e894b6be883564b9cea62b94ef9a57dd94d2aa2b4fbace10532c716fb8319"}
Nov 03 17:38:27.514663 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.514607    6059 scope.go:122] "RemoveContainer" containerID="9d31a7d0d7f3a2c9674accf9cb41a6b2f50ee0f810b96d58b4b8cf4425fa535e"
Nov 03 17:38:27.514775 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.514737    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x"
Nov 03 17:38:27.519593 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.519557    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln" event={"ID":"436e4e0c-365c-42e6-a646-424b167a7686","Type":"ContainerStarted","Data":"37fa0140d27200bb3ea0a69b9c4cb1c37aea0a57ec2968c4be8637c016a34167"}
Nov 03 17:38:27.519726 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.519626    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln" event={"ID":"436e4e0c-365c-42e6-a646-424b167a7686","Type":"ContainerStarted","Data":"08c0a9379808eb1eedb453807b014986822ae4de86f1296275cb0426a8136379"}
Nov 03 17:38:27.521189 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.521159    6059 generic.go:356] "Generic (PLEG): container finished" podID="9f7d5da9-069f-4387-b93d-3f3687d7a43a" containerID="e4de9fb28f654396dfaf7a76570b3a18e59f4595c7010acf53168a477f2c78af" exitCode=0
Nov 03 17:38:27.521326 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.521310    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh" event={"ID":"9f7d5da9-069f-4387-b93d-3f3687d7a43a","Type":"ContainerDied","Data":"e4de9fb28f654396dfaf7a76570b3a18e59f4595c7010acf53168a477f2c78af"}
Nov 03 17:38:27.521451 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.521347    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh"
Nov 03 17:38:27.521451 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.521399    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh" event={"ID":"9f7d5da9-069f-4387-b93d-3f3687d7a43a","Type":"ContainerDied","Data":"fdd7d6b481147b15deb541612e88dcbd6780171bec4b6ab85b0e87f032d5dd46"}
Nov 03 17:38:27.532032 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.531988    6059 scope.go:122] "RemoveContainer" containerID="9d31a7d0d7f3a2c9674accf9cb41a6b2f50ee0f810b96d58b4b8cf4425fa535e"
Nov 03 17:38:27.532388 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:38:27.532328    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d31a7d0d7f3a2c9674accf9cb41a6b2f50ee0f810b96d58b4b8cf4425fa535e\": container with ID starting with 9d31a7d0d7f3a2c9674accf9cb41a6b2f50ee0f810b96d58b4b8cf4425fa535e not found: ID does not exist" containerID="9d31a7d0d7f3a2c9674accf9cb41a6b2f50ee0f810b96d58b4b8cf4425fa535e"
Nov 03 17:38:27.532484 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.532379    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d31a7d0d7f3a2c9674accf9cb41a6b2f50ee0f810b96d58b4b8cf4425fa535e"} err="failed to get container status \"9d31a7d0d7f3a2c9674accf9cb41a6b2f50ee0f810b96d58b4b8cf4425fa535e\": rpc error: code = NotFound desc = could not find container \"9d31a7d0d7f3a2c9674accf9cb41a6b2f50ee0f810b96d58b4b8cf4425fa535e\": container with ID starting with 9d31a7d0d7f3a2c9674accf9cb41a6b2f50ee0f810b96d58b4b8cf4425fa535e not found: ID does not exist"
Nov 03 17:38:27.532484 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.532403    6059 scope.go:122] "RemoveContainer" containerID="e4de9fb28f654396dfaf7a76570b3a18e59f4595c7010acf53168a477f2c78af"
Nov 03 17:38:27.543118 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.543097    6059 scope.go:122] "RemoveContainer" containerID="e4de9fb28f654396dfaf7a76570b3a18e59f4595c7010acf53168a477f2c78af"
Nov 03 17:38:27.543442 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:38:27.543416    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4de9fb28f654396dfaf7a76570b3a18e59f4595c7010acf53168a477f2c78af\": container with ID starting with e4de9fb28f654396dfaf7a76570b3a18e59f4595c7010acf53168a477f2c78af not found: ID does not exist" containerID="e4de9fb28f654396dfaf7a76570b3a18e59f4595c7010acf53168a477f2c78af"
Nov 03 17:38:27.543527 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.543448    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4de9fb28f654396dfaf7a76570b3a18e59f4595c7010acf53168a477f2c78af"} err="failed to get container status \"e4de9fb28f654396dfaf7a76570b3a18e59f4595c7010acf53168a477f2c78af\": rpc error: code = NotFound desc = could not find container \"e4de9fb28f654396dfaf7a76570b3a18e59f4595c7010acf53168a477f2c78af\": container with ID starting with e4de9fb28f654396dfaf7a76570b3a18e59f4595c7010acf53168a477f2c78af not found: ID does not exist"
Nov 03 17:38:27.561232 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.561202    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e65883-b429-48a8-85df-f5122c5aa67d-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:27.561232 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.561232    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbl8k\" (UniqueName: \"kubernetes.io/projected/15e65883-b429-48a8-85df-f5122c5aa67d-kube-api-access-wbl8k\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:27.689620 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.689572    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh"]
Nov 03 17:38:27.695592 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.695565    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e34dd2-78cbc6b76c-qg6mh"]
Nov 03 17:38:27.813456 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.813331    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln" podStartSLOduration=2.813318255 podStartE2EDuration="2.813318255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:38:27.810014795 +0000 UTC m=+1248.607727576" watchObservedRunningTime="2026-11-03 17:38:27.813318255 +0000 UTC m=+1248.611031092"
Nov 03 17:38:27.927535 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.927505    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x"]
Nov 03 17:38:27.931207 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:27.931179    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e34dd2-545479d55f-nz98x"]
Nov 03 17:38:28.533155 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:28.533123    6059 generic.go:356] "Generic (PLEG): container finished" podID="436e4e0c-365c-42e6-a646-424b167a7686" containerID="37fa0140d27200bb3ea0a69b9c4cb1c37aea0a57ec2968c4be8637c016a34167" exitCode=0
Nov 03 17:38:28.533590 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:28.533165    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln" event={"ID":"436e4e0c-365c-42e6-a646-424b167a7686","Type":"ContainerDied","Data":"37fa0140d27200bb3ea0a69b9c4cb1c37aea0a57ec2968c4be8637c016a34167"}
Nov 03 17:38:29.031305 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:29.031272    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:29.338906 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:29.338834    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15e65883-b429-48a8-85df-f5122c5aa67d" path="/var/lib/kubelet/pods/15e65883-b429-48a8-85df-f5122c5aa67d/volumes"
Nov 03 17:38:29.339208 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:29.339196    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f7d5da9-069f-4387-b93d-3f3687d7a43a" path="/var/lib/kubelet/pods/9f7d5da9-069f-4387-b93d-3f3687d7a43a/volumes"
Nov 03 17:38:29.711368 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:29.711337    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln"
Nov 03 17:38:29.789343 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:29.789310    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/436e4e0c-365c-42e6-a646-424b167a7686-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/436e4e0c-365c-42e6-a646-424b167a7686-maas-api-openshift-ca\") pod \"436e4e0c-365c-42e6-a646-424b167a7686\" (UID: \"436e4e0c-365c-42e6-a646-424b167a7686\") "
Nov 03 17:38:29.789505 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:29.789478    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/436e4e0c-365c-42e6-a646-424b167a7686-kube-api-access-b4hxl\" (UniqueName: \"kubernetes.io/projected/436e4e0c-365c-42e6-a646-424b167a7686-kube-api-access-b4hxl\") pod \"436e4e0c-365c-42e6-a646-424b167a7686\" (UID: \"436e4e0c-365c-42e6-a646-424b167a7686\") "
Nov 03 17:38:29.789617 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:29.789603    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/436e4e0c-365c-42e6-a646-424b167a7686-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/436e4e0c-365c-42e6-a646-424b167a7686-maas-api-xks-ca\") pod \"436e4e0c-365c-42e6-a646-424b167a7686\" (UID: \"436e4e0c-365c-42e6-a646-424b167a7686\") "
Nov 03 17:38:29.789723 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:29.789694    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/436e4e0c-365c-42e6-a646-424b167a7686-maas-api-openshift-ca" pod "436e4e0c-365c-42e6-a646-424b167a7686" (UID: "436e4e0c-365c-42e6-a646-424b167a7686"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:38:29.790013 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:29.789988    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/436e4e0c-365c-42e6-a646-424b167a7686-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:29.791649 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:29.791627    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436e4e0c-365c-42e6-a646-424b167a7686-kube-api-access-b4hxl" pod "436e4e0c-365c-42e6-a646-424b167a7686" (UID: "436e4e0c-365c-42e6-a646-424b167a7686"). InnerVolumeSpecName "kube-api-access-b4hxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:38:29.791747 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:29.791688    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/436e4e0c-365c-42e6-a646-424b167a7686-maas-api-xks-ca" pod "436e4e0c-365c-42e6-a646-424b167a7686" (UID: "436e4e0c-365c-42e6-a646-424b167a7686"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:38:29.901476 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:29.894414    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/436e4e0c-365c-42e6-a646-424b167a7686-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:29.901476 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:29.894458    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4hxl\" (UniqueName: \"kubernetes.io/projected/436e4e0c-365c-42e6-a646-424b167a7686-kube-api-access-b4hxl\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:30.550333 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:30.550302    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln"
Nov 03 17:38:30.550551 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:30.550347    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln" event={"ID":"436e4e0c-365c-42e6-a646-424b167a7686","Type":"ContainerDied","Data":"08c0a9379808eb1eedb453807b014986822ae4de86f1296275cb0426a8136379"}
Nov 03 17:38:30.550551 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:30.550432    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08c0a9379808eb1eedb453807b014986822ae4de86f1296275cb0426a8136379"
Nov 03 17:38:30.722543 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:30.722511    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:33.774277 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:33.774242    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:34.001192 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:34.001160    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:35.279513 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:35.279468    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:35.487818 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:35.487788    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:35.800755 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:35.800718    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"]
Nov 03 17:38:35.800944 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:35.800925    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns" podUID="3db8f565-8a57-45ff-b31c-aa5a4a7093fa" containerName="istio-proxy" containerID="cri-o://96bfc86b7d81801f6bc1052bd0fd4b0e18042151e0b761f9d620f974dfab0438" gracePeriod=30
Nov 03 17:38:35.875209 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:35.875178    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:36.056884 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:36.056803    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68"]
Nov 03 17:38:36.060218 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:36.060195    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ebb2cfe-ssk68"]
Nov 03 17:38:36.862069 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:38:36.862029    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:38:37.337836 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:37.337797    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8e67b74-9b61-4832-9e2d-f79287894fa9" path="/var/lib/kubelet/pods/c8e67b74-9b61-4832-9e2d-f79287894fa9/volumes"
Nov 03 17:38:37.348337 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:37.348308    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:38.754382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:38.751088    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:38.962121 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:38.962086    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:39.481821 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:39.481758    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:39.776442 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:39.775823    6059 scope.go:122] "RemoveContainer" containerID="67505fdafc3a2a7399c2ea51ab70d33b0d5baa1b633451de3020a99b408df9e0"
Nov 03 17:38:39.998268 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:39.998145    6059 scope.go:122] "RemoveContainer" containerID="0fabdaf023c5a6c6382e145c0ad94e78c3e3123c8160c967e64df810be26db34"
Nov 03 17:38:40.078425 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:40.075636    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v" podUID="2bbc9082-b2da-4198-8a8a-b6ab26b5011e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:38:40.078425 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:40.077807    6059 scope.go:122] "RemoveContainer" containerID="4b3b1d45b6dd5cb2303c36bece8e390fcc3be874aaefd815b0f0f8093c2f0d89"
Nov 03 17:38:40.158878 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:40.158832    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w" podUID="6120f11d-14c8-4666-a201-469a5bca37de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:38:40.160395 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:40.159700    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl" podUID="c194a4be-0499-419f-a4a1-910dbaafd303" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:38:40.160395 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:40.159931    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n" podUID="d6c1263b-e648-4f10-af7d-86397d6b207a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:38:40.216533 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:40.216491    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv" podUID="7942e9d1-597f-47a6-9c70-1d174fe4c8e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:38:41.083925 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.081610    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v" podUID="2bbc9082-b2da-4198-8a8a-b6ab26b5011e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:38:41.083925 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.082398    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:38:41.178381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.177648    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w" podUID="6120f11d-14c8-4666-a201-469a5bca37de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded"
Nov 03 17:38:41.178666 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.178639    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:38:41.221396 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.218462    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl" podUID="c194a4be-0499-419f-a4a1-910dbaafd303" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:38:41.221396 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.218887    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:38:41.224274 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.223463    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv" podUID="7942e9d1-597f-47a6-9c70-1d174fe4c8e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:38:41.224274 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.223857    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:38:41.399428 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.399053    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:38:41.399428 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.399112    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:38:41.413615 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.413121    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:38:41.430507 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.430122    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl"
Nov 03 17:38:41.576404 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.576368    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:38:41.676734 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.676691    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-kube-api-access-ltnxq\" (UniqueName: \"kubernetes.io/projected/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-kube-api-access-ltnxq\") pod \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") "
Nov 03 17:38:41.676928 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.676756    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-envoy\") pod \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") "
Nov 03 17:38:41.676928 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.676790    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-workload-certs\") pod \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") "
Nov 03 17:38:41.676928 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.676855    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-workload-socket\") pod \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") "
Nov 03 17:38:41.677102 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.677010    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-podinfo\") pod \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") "
Nov 03 17:38:41.677102 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.677067    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-token\" (UniqueName: \"kubernetes.io/projected/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-token\") pod \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") "
Nov 03 17:38:41.677224 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.677101    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-credential-socket\") pod \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") "
Nov 03 17:38:41.677224 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.677156    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istiod-ca-cert\") pod \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") "
Nov 03 17:38:41.677224 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.677177    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-workload-certs" pod "3db8f565-8a57-45ff-b31c-aa5a4a7093fa" (UID: "3db8f565-8a57-45ff-b31c-aa5a4a7093fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:38:41.677389 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.677257    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-data\") pod \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\" (UID: \"3db8f565-8a57-45ff-b31c-aa5a4a7093fa\") "
Nov 03 17:38:41.678001 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.677561    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istiod-ca-cert" pod "3db8f565-8a57-45ff-b31c-aa5a4a7093fa" (UID: "3db8f565-8a57-45ff-b31c-aa5a4a7093fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:38:41.678001 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.677872    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:41.678001 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.677897    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:41.678001 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.677894    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-data" pod "3db8f565-8a57-45ff-b31c-aa5a4a7093fa" (UID: "3db8f565-8a57-45ff-b31c-aa5a4a7093fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:38:41.678001 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.677970    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-workload-socket" pod "3db8f565-8a57-45ff-b31c-aa5a4a7093fa" (UID: "3db8f565-8a57-45ff-b31c-aa5a4a7093fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:38:41.678382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.678113    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-credential-socket" pod "3db8f565-8a57-45ff-b31c-aa5a4a7093fa" (UID: "3db8f565-8a57-45ff-b31c-aa5a4a7093fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:38:41.679943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.679914    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-envoy" pod "3db8f565-8a57-45ff-b31c-aa5a4a7093fa" (UID: "3db8f565-8a57-45ff-b31c-aa5a4a7093fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:38:41.680167 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.680138    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-kube-api-access-ltnxq" pod "3db8f565-8a57-45ff-b31c-aa5a4a7093fa" (UID: "3db8f565-8a57-45ff-b31c-aa5a4a7093fa"). InnerVolumeSpecName "kube-api-access-ltnxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:38:41.680856 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.680824    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-token" pod "3db8f565-8a57-45ff-b31c-aa5a4a7093fa" (UID: "3db8f565-8a57-45ff-b31c-aa5a4a7093fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:38:41.680856 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.680843    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-podinfo" pod "3db8f565-8a57-45ff-b31c-aa5a4a7093fa" (UID: "3db8f565-8a57-45ff-b31c-aa5a4a7093fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:38:41.779230 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.779188    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:41.779230 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.779230    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:41.779473 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.779245    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:41.779473 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.779258    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:41.779473 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.779273    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltnxq\" (UniqueName: \"kubernetes.io/projected/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-kube-api-access-ltnxq\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:41.779473 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.779290    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:41.779473 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.779304    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db8f565-8a57-45ff-b31c-aa5a4a7093fa-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:41.944960 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.944871    6059 generic.go:356] "Generic (PLEG): container finished" podID="3db8f565-8a57-45ff-b31c-aa5a4a7093fa" containerID="96bfc86b7d81801f6bc1052bd0fd4b0e18042151e0b761f9d620f974dfab0438" exitCode=0
Nov 03 17:38:41.944960 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.944921    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"
Nov 03 17:38:41.945199 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.944955    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns" event={"ID":"3db8f565-8a57-45ff-b31c-aa5a4a7093fa","Type":"ContainerDied","Data":"96bfc86b7d81801f6bc1052bd0fd4b0e18042151e0b761f9d620f974dfab0438"}
Nov 03 17:38:41.945199 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.945021    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns" event={"ID":"3db8f565-8a57-45ff-b31c-aa5a4a7093fa","Type":"ContainerDied","Data":"412a3b626564d7467d90816dd8a70a3f2562d08e0d634acce91a44de3734b690"}
Nov 03 17:38:41.945199 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.945066    6059 scope.go:122] "RemoveContainer" containerID="96bfc86b7d81801f6bc1052bd0fd4b0e18042151e0b761f9d620f974dfab0438"
Nov 03 17:38:41.956844 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.956823    6059 scope.go:122] "RemoveContainer" containerID="d1f7556315653fbd1a28346e5983a15d8ec09118a3ee7cd9543a599f8e0d004f"
Nov 03 17:38:41.966882 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.966863    6059 scope.go:122] "RemoveContainer" containerID="96bfc86b7d81801f6bc1052bd0fd4b0e18042151e0b761f9d620f974dfab0438"
Nov 03 17:38:41.967195 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:38:41.967174    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96bfc86b7d81801f6bc1052bd0fd4b0e18042151e0b761f9d620f974dfab0438\": container with ID starting with 96bfc86b7d81801f6bc1052bd0fd4b0e18042151e0b761f9d620f974dfab0438 not found: ID does not exist" containerID="96bfc86b7d81801f6bc1052bd0fd4b0e18042151e0b761f9d620f974dfab0438"
Nov 03 17:38:41.967275 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.967209    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96bfc86b7d81801f6bc1052bd0fd4b0e18042151e0b761f9d620f974dfab0438"} err="failed to get container status \"96bfc86b7d81801f6bc1052bd0fd4b0e18042151e0b761f9d620f974dfab0438\": rpc error: code = NotFound desc = could not find container \"96bfc86b7d81801f6bc1052bd0fd4b0e18042151e0b761f9d620f974dfab0438\": container with ID starting with 96bfc86b7d81801f6bc1052bd0fd4b0e18042151e0b761f9d620f974dfab0438 not found: ID does not exist"
Nov 03 17:38:41.967275 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.967233    6059 scope.go:122] "RemoveContainer" containerID="d1f7556315653fbd1a28346e5983a15d8ec09118a3ee7cd9543a599f8e0d004f"
Nov 03 17:38:41.967547 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:38:41.967520    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1f7556315653fbd1a28346e5983a15d8ec09118a3ee7cd9543a599f8e0d004f\": container with ID starting with d1f7556315653fbd1a28346e5983a15d8ec09118a3ee7cd9543a599f8e0d004f not found: ID does not exist" containerID="d1f7556315653fbd1a28346e5983a15d8ec09118a3ee7cd9543a599f8e0d004f"
Nov 03 17:38:41.967626 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:41.967557    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f7556315653fbd1a28346e5983a15d8ec09118a3ee7cd9543a599f8e0d004f"} err="failed to get container status \"d1f7556315653fbd1a28346e5983a15d8ec09118a3ee7cd9543a599f8e0d004f\": rpc error: code = NotFound desc = could not find container \"d1f7556315653fbd1a28346e5983a15d8ec09118a3ee7cd9543a599f8e0d004f\": container with ID starting with d1f7556315653fbd1a28346e5983a15d8ec09118a3ee7cd9543a599f8e0d004f not found: ID does not exist"
Nov 03 17:38:42.113782 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:42.113745    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:42.221676 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:42.221646    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"]
Nov 03 17:38:42.239658 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:42.239621    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-7nlns"]
Nov 03 17:38:43.337297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:43.337263    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3db8f565-8a57-45ff-b31c-aa5a4a7093fa" path="/var/lib/kubelet/pods/3db8f565-8a57-45ff-b31c-aa5a4a7093fa/volumes"
Nov 03 17:38:44.685671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.685629    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c"]
Nov 03 17:38:44.686403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.686378    6059 cpu_manager.go:414] "containerMap: removing container" podUID="9f7d5da9-069f-4387-b93d-3f3687d7a43a" containerName="payload-processing"
Nov 03 17:38:44.686514 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.686403    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f7d5da9-069f-4387-b93d-3f3687d7a43a" containerName="payload-processing"
Nov 03 17:38:44.686514 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.686422    6059 cpu_manager.go:414] "containerMap: removing container" podUID="15e65883-b429-48a8-85df-f5122c5aa67d" containerName="payload-pre-processing"
Nov 03 17:38:44.686514 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.686430    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15e65883-b429-48a8-85df-f5122c5aa67d" containerName="payload-pre-processing"
Nov 03 17:38:44.686514 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.686464    6059 cpu_manager.go:414] "containerMap: removing container" podUID="3db8f565-8a57-45ff-b31c-aa5a4a7093fa" containerName="istio-proxy"
Nov 03 17:38:44.686514 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.686472    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db8f565-8a57-45ff-b31c-aa5a4a7093fa" containerName="istio-proxy"
Nov 03 17:38:44.686514 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.686511    6059 cpu_manager.go:414] "containerMap: removing container" podUID="436e4e0c-365c-42e6-a646-424b167a7686" containerName="revoke-keys"
Nov 03 17:38:44.686817 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.686519    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436e4e0c-365c-42e6-a646-424b167a7686" containerName="revoke-keys"
Nov 03 17:38:44.686817 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.686661    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3db8f565-8a57-45ff-b31c-aa5a4a7093fa" containerName="istio-proxy"
Nov 03 17:38:44.686817 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.686674    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="436e4e0c-365c-42e6-a646-424b167a7686" containerName="revoke-keys"
Nov 03 17:38:44.686817 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.686690    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f7d5da9-069f-4387-b93d-3f3687d7a43a" containerName="payload-processing"
Nov 03 17:38:44.686817 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.686702    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="15e65883-b429-48a8-85df-f5122c5aa67d" containerName="payload-pre-processing"
Nov 03 17:38:44.686817 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.686715    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3db8f565-8a57-45ff-b31c-aa5a4a7093fa" containerName="istio-proxy"
Nov 03 17:38:44.691895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.691870    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c"
Nov 03 17:38:44.756800 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.756757    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:38:44.756998 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.756844    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:38:44.756998 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.756886    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:38:44.811914 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.811886    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:38:44.815988 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.815964    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j745q\" (UniqueName: \"kubernetes.io/projected/7a6a7bae-214b-4470-af84-7d5301433e6d-kube-api-access-j745q\") pod \"maas-api-revoke-sub-c4bf8108-d4j9c\" (UID: \"7a6a7bae-214b-4470-af84-7d5301433e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c"
Nov 03 17:38:44.815988 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.815965    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c"]
Nov 03 17:38:44.816106 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.816067    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a6a7bae-214b-4470-af84-7d5301433e6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4bf8108-d4j9c\" (UID: \"7a6a7bae-214b-4470-af84-7d5301433e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c"
Nov 03 17:38:44.816140 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.816112    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a6a7bae-214b-4470-af84-7d5301433e6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4bf8108-d4j9c\" (UID: \"7a6a7bae-214b-4470-af84-7d5301433e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c"
Nov 03 17:38:44.917757 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.917724    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j745q\" (UniqueName: \"kubernetes.io/projected/7a6a7bae-214b-4470-af84-7d5301433e6d-kube-api-access-j745q\") pod \"maas-api-revoke-sub-c4bf8108-d4j9c\" (UID: \"7a6a7bae-214b-4470-af84-7d5301433e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c"
Nov 03 17:38:44.917945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.917808    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a6a7bae-214b-4470-af84-7d5301433e6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4bf8108-d4j9c\" (UID: \"7a6a7bae-214b-4470-af84-7d5301433e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c"
Nov 03 17:38:44.917945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.917853    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a6a7bae-214b-4470-af84-7d5301433e6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4bf8108-d4j9c\" (UID: \"7a6a7bae-214b-4470-af84-7d5301433e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c"
Nov 03 17:38:44.918499 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.918476    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a6a7bae-214b-4470-af84-7d5301433e6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4bf8108-d4j9c\" (UID: \"7a6a7bae-214b-4470-af84-7d5301433e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c"
Nov 03 17:38:44.920164 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.920134    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a6a7bae-214b-4470-af84-7d5301433e6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4bf8108-d4j9c\" (UID: \"7a6a7bae-214b-4470-af84-7d5301433e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c"
Nov 03 17:38:44.988124 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:44.988076    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j745q\" (UniqueName: \"kubernetes.io/projected/7a6a7bae-214b-4470-af84-7d5301433e6d-kube-api-access-j745q\") pod \"maas-api-revoke-sub-c4bf8108-d4j9c\" (UID: \"7a6a7bae-214b-4470-af84-7d5301433e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c"
Nov 03 17:38:45.004934 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:45.004908    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c"
Nov 03 17:38:45.390865 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:38:45.390828    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6a7bae_214b_4470_af84_7d5301433e6d.slice/crio-a72daa34d328c626025054f0a67873b8bf41d88ebe403b4da0ebc6858d588c90 WatchSource:0}: Error finding container a72daa34d328c626025054f0a67873b8bf41d88ebe403b4da0ebc6858d588c90: Status 404 returned error can't find the container with id a72daa34d328c626025054f0a67873b8bf41d88ebe403b4da0ebc6858d588c90
Nov 03 17:38:45.394494 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:45.394468    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c"]
Nov 03 17:38:45.985500 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:45.985464    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c" event={"ID":"7a6a7bae-214b-4470-af84-7d5301433e6d","Type":"ContainerStarted","Data":"4e78efa48f473360f68952f45c99c1c6b10ef31486f46d7672b84150cd814287"}
Nov 03 17:38:45.985880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:45.985543    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c" event={"ID":"7a6a7bae-214b-4470-af84-7d5301433e6d","Type":"ContainerStarted","Data":"a72daa34d328c626025054f0a67873b8bf41d88ebe403b4da0ebc6858d588c90"}
Nov 03 17:38:46.107644 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:46.107583    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c" podStartSLOduration=2.107567849 podStartE2EDuration="2.107567849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:38:46.104650466 +0000 UTC m=+1266.902363247" watchObservedRunningTime="2026-11-03 17:38:46.107567849 +0000 UTC m=+1266.905280674"
Nov 03 17:38:46.116008 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:46.115977    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:46.308826 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:46.308794    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:46.509509 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:46.509479    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:46.744631 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:46.744599    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:46.998712 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:46.998618    6059 generic.go:356] "Generic (PLEG): container finished" podID="7a6a7bae-214b-4470-af84-7d5301433e6d" containerID="4e78efa48f473360f68952f45c99c1c6b10ef31486f46d7672b84150cd814287" exitCode=0
Nov 03 17:38:46.999083 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:46.998707    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c" event={"ID":"7a6a7bae-214b-4470-af84-7d5301433e6d","Type":"ContainerDied","Data":"4e78efa48f473360f68952f45c99c1c6b10ef31486f46d7672b84150cd814287"}
Nov 03 17:38:47.168553 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.168517    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"]
Nov 03 17:38:47.169838 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.169806    6059 cpu_manager.go:414] "containerMap: removing container" podUID="3db8f565-8a57-45ff-b31c-aa5a4a7093fa" containerName="istio-proxy"
Nov 03 17:38:47.173459 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.173425    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db8f565-8a57-45ff-b31c-aa5a4a7093fa" containerName="istio-proxy"
Nov 03 17:38:47.178255 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.178222    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.182872 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.182840    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"]
Nov 03 17:38:47.207318 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.207263    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:47.234504 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.234427    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1ed607ea-openshift-default-dockercfg-2sgnw\""
Nov 03 17:38:47.248057 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.248022    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-credential-socket\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.248057 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.248067    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e53c74d-ceff-4b92-b579-35ef2b76de83-istiod-ca-cert\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.248264 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.248194    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-data\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.248310 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.248282    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-workload-certs\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.248349 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.248323    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-envoy\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.248429 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.248416    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-token\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.248471 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.248463    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-podinfo\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.248566 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.248541    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bjqp\" (UniqueName: \"kubernetes.io/projected/3e53c74d-ceff-4b92-b579-35ef2b76de83-kube-api-access-5bjqp\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.248618 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.248601    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-workload-socket\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.349781 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.349700    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bjqp\" (UniqueName: \"kubernetes.io/projected/3e53c74d-ceff-4b92-b579-35ef2b76de83-kube-api-access-5bjqp\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.349781 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.349744    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-workload-socket\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.349977 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.349827    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-credential-socket\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.349977 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.349860    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e53c74d-ceff-4b92-b579-35ef2b76de83-istiod-ca-cert\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.349977 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.349900    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-data\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.349977 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.349934    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-workload-certs\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.349977 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.349971    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-envoy\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.350188 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.349999    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-token\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.350188 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.350069    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-podinfo\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.350262 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.350203    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-workload-socket\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.350262 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.350250    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-credential-socket\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.350386 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.350317    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-workload-certs\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.350558 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.350525    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-data\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.350689 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.350669    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e53c74d-ceff-4b92-b579-35ef2b76de83-istiod-ca-cert\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.352482 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.352455    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-envoy\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.352618 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.352595    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-podinfo\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.417528 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.417486    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bjqp\" (UniqueName: \"kubernetes.io/projected/3e53c74d-ceff-4b92-b579-35ef2b76de83-kube-api-access-5bjqp\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.418675 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.418625    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-token\") pod \"e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") " pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.497289 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.497257    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:47.523725 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.523697    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:47.929913 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:38:47.929885    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e53c74d_ceff_4b92_b579_35ef2b76de83.slice/crio-9f965a28f39a3902b6ea2dbf29e3205c09d1ce9bd544ee3598fa4643fb8bae69 WatchSource:0}: Error finding container 9f965a28f39a3902b6ea2dbf29e3205c09d1ce9bd544ee3598fa4643fb8bae69: Status 404 returned error can't find the container with id 9f965a28f39a3902b6ea2dbf29e3205c09d1ce9bd544ee3598fa4643fb8bae69
Nov 03 17:38:47.932684 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.932649    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:38:47.932776 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.932761    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:38:47.932818 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.932804    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:38:47.933553 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:47.933531    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"]
Nov 03 17:38:48.010129 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:48.010089    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq" event={"ID":"3e53c74d-ceff-4b92-b579-35ef2b76de83","Type":"ContainerStarted","Data":"9f965a28f39a3902b6ea2dbf29e3205c09d1ce9bd544ee3598fa4643fb8bae69"}
Nov 03 17:38:48.138208 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:48.138152    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq" podStartSLOduration=1.138137145 podStartE2EDuration="1.138137145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:38:48.134182221 +0000 UTC m=+1268.931895001" watchObservedRunningTime="2026-11-03 17:38:48.138137145 +0000 UTC m=+1268.935849926"
Nov 03 17:38:48.154777 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:48.154752    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c"
Nov 03 17:38:48.260123 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:48.260093    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a6a7bae-214b-4470-af84-7d5301433e6d-kube-api-access-j745q\" (UniqueName: \"kubernetes.io/projected/7a6a7bae-214b-4470-af84-7d5301433e6d-kube-api-access-j745q\") pod \"7a6a7bae-214b-4470-af84-7d5301433e6d\" (UID: \"7a6a7bae-214b-4470-af84-7d5301433e6d\") "
Nov 03 17:38:48.260426 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:48.260294    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a6a7bae-214b-4470-af84-7d5301433e6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a6a7bae-214b-4470-af84-7d5301433e6d-maas-api-openshift-ca\") pod \"7a6a7bae-214b-4470-af84-7d5301433e6d\" (UID: \"7a6a7bae-214b-4470-af84-7d5301433e6d\") "
Nov 03 17:38:48.260525 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:48.260469    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a6a7bae-214b-4470-af84-7d5301433e6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a6a7bae-214b-4470-af84-7d5301433e6d-maas-api-xks-ca\") pod \"7a6a7bae-214b-4470-af84-7d5301433e6d\" (UID: \"7a6a7bae-214b-4470-af84-7d5301433e6d\") "
Nov 03 17:38:48.261478 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:48.261107    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a6a7bae-214b-4470-af84-7d5301433e6d-maas-api-openshift-ca" pod "7a6a7bae-214b-4470-af84-7d5301433e6d" (UID: "7a6a7bae-214b-4470-af84-7d5301433e6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:38:48.264008 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:48.263978    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a6a7bae-214b-4470-af84-7d5301433e6d-kube-api-access-j745q" pod "7a6a7bae-214b-4470-af84-7d5301433e6d" (UID: "7a6a7bae-214b-4470-af84-7d5301433e6d"). InnerVolumeSpecName "kube-api-access-j745q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:38:48.267637 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:48.267573    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a6a7bae-214b-4470-af84-7d5301433e6d-maas-api-xks-ca" pod "7a6a7bae-214b-4470-af84-7d5301433e6d" (UID: "7a6a7bae-214b-4470-af84-7d5301433e6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:38:48.361931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:48.361891    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j745q\" (UniqueName: \"kubernetes.io/projected/7a6a7bae-214b-4470-af84-7d5301433e6d-kube-api-access-j745q\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:48.361931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:48.361925    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a6a7bae-214b-4470-af84-7d5301433e6d-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:48.361931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:48.361938    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a6a7bae-214b-4470-af84-7d5301433e6d-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:48.498138 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:48.498102    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:49.019529 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:49.019501    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c"
Nov 03 17:38:49.020027 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:49.019528    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c" event={"ID":"7a6a7bae-214b-4470-af84-7d5301433e6d","Type":"ContainerDied","Data":"a72daa34d328c626025054f0a67873b8bf41d88ebe403b4da0ebc6858d588c90"}
Nov 03 17:38:49.020027 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:49.019578    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a72daa34d328c626025054f0a67873b8bf41d88ebe403b4da0ebc6858d588c90"
Nov 03 17:38:49.032786 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:49.032668    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq" event={"ID":"3e53c74d-ceff-4b92-b579-35ef2b76de83","Type":"ContainerStarted","Data":"c29ac6f3f77f4fdf3a59636f7804eac7311724a5feb38d17d8b8248cc0e12fb6"}
Nov 03 17:38:49.503330 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:49.503298    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:50.039936 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:50.039902    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:50.041007 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:50.040983    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:38:50.176718 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:50.176673    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:50.400659 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:50.400567    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:51.579221 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:51.579190    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:53.055141 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.055097    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a92b5d-bvl8d"]
Nov 03 17:38:53.104519 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.104485    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a92b5d-bvl8d"]
Nov 03 17:38:53.283605 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.283563    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:53.338859 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.338740    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e3b666b-5066-4524-a552-cbc829bd11b4" path="/var/lib/kubelet/pods/4e3b666b-5066-4524-a552-cbc829bd11b4/volumes"
Nov 03 17:38:53.481985 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.481953    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:53.661286 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.661197    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w"]
Nov 03 17:38:53.661887 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.661866    6059 cpu_manager.go:414] "containerMap: removing container" podUID="7a6a7bae-214b-4470-af84-7d5301433e6d" containerName="revoke-keys"
Nov 03 17:38:53.662005 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.661890    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a6a7bae-214b-4470-af84-7d5301433e6d" containerName="revoke-keys"
Nov 03 17:38:53.662075 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.662059    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a6a7bae-214b-4470-af84-7d5301433e6d" containerName="revoke-keys"
Nov 03 17:38:53.666853 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.666830    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w"
Nov 03 17:38:53.726510 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.726471    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w"]
Nov 03 17:38:53.757520 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.732216    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62446602-8e77-48fc-baff-a974c95e75a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w\" (UID: \"62446602-8e77-48fc-baff-a974c95e75a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w"
Nov 03 17:38:53.757520 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.732426    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hqvt\" (UniqueName: \"kubernetes.io/projected/62446602-8e77-48fc-baff-a974c95e75a4-kube-api-access-4hqvt\") pod \"payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w\" (UID: \"62446602-8e77-48fc-baff-a974c95e75a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w"
Nov 03 17:38:53.775407 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.775334    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1ed607ea\""
Nov 03 17:38:53.775689 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.775536    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1ed607ea-dockercfg-fwwpx\""
Nov 03 17:38:53.828557 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.828521    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh"]
Nov 03 17:38:53.833537 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.833509    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh"
Nov 03 17:38:53.834343 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.834311    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62446602-8e77-48fc-baff-a974c95e75a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w\" (UID: \"62446602-8e77-48fc-baff-a974c95e75a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w"
Nov 03 17:38:53.834492 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.834441    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hqvt\" (UniqueName: \"kubernetes.io/projected/62446602-8e77-48fc-baff-a974c95e75a4-kube-api-access-4hqvt\") pod \"payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w\" (UID: \"62446602-8e77-48fc-baff-a974c95e75a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w"
Nov 03 17:38:53.835861 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.835838    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62446602-8e77-48fc-baff-a974c95e75a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w\" (UID: \"62446602-8e77-48fc-baff-a974c95e75a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w"
Nov 03 17:38:53.887507 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.887474    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh"]
Nov 03 17:38:53.897771 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.897738    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hqvt\" (UniqueName: \"kubernetes.io/projected/62446602-8e77-48fc-baff-a974c95e75a4-kube-api-access-4hqvt\") pod \"payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w\" (UID: \"62446602-8e77-48fc-baff-a974c95e75a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w"
Nov 03 17:38:53.935749 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.935666    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgr9w\" (UniqueName: \"kubernetes.io/projected/b4bcd4ea-d922-4475-8e2e-c899be07ae49-kube-api-access-zgr9w\") pod \"payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh\" (UID: \"b4bcd4ea-d922-4475-8e2e-c899be07ae49\") " pod="openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh"
Nov 03 17:38:53.935749 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.935718    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bcd4ea-d922-4475-8e2e-c899be07ae49-plugins-config-volume\") pod \"payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh\" (UID: \"b4bcd4ea-d922-4475-8e2e-c899be07ae49\") " pod="openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh"
Nov 03 17:38:53.980350 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:53.980312    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w"
Nov 03 17:38:54.037723 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:54.037683    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgr9w\" (UniqueName: \"kubernetes.io/projected/b4bcd4ea-d922-4475-8e2e-c899be07ae49-kube-api-access-zgr9w\") pod \"payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh\" (UID: \"b4bcd4ea-d922-4475-8e2e-c899be07ae49\") " pod="openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh"
Nov 03 17:38:54.037893 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:54.037757    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bcd4ea-d922-4475-8e2e-c899be07ae49-plugins-config-volume\") pod \"payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh\" (UID: \"b4bcd4ea-d922-4475-8e2e-c899be07ae49\") " pod="openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh"
Nov 03 17:38:54.039405 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:54.038992    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bcd4ea-d922-4475-8e2e-c899be07ae49-plugins-config-volume\") pod \"payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh\" (UID: \"b4bcd4ea-d922-4475-8e2e-c899be07ae49\") " pod="openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh"
Nov 03 17:38:54.102141 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:54.102106    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgr9w\" (UniqueName: \"kubernetes.io/projected/b4bcd4ea-d922-4475-8e2e-c899be07ae49-kube-api-access-zgr9w\") pod \"payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh\" (UID: \"b4bcd4ea-d922-4475-8e2e-c899be07ae49\") " pod="openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh"
Nov 03 17:38:54.150321 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:54.150292    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh"
Nov 03 17:38:54.359038 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:38:54.359001    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62446602_8e77_48fc_baff_a974c95e75a4.slice/crio-68a739f47873e511522ae65d643de71f71268d952a8f451e8dbf26230d6a035a WatchSource:0}: Error finding container 68a739f47873e511522ae65d643de71f71268d952a8f451e8dbf26230d6a035a: Status 404 returned error can't find the container with id 68a739f47873e511522ae65d643de71f71268d952a8f451e8dbf26230d6a035a
Nov 03 17:38:54.363139 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:54.363116    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w"]
Nov 03 17:38:54.524183 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:38:54.524125    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4bcd4ea_d922_4475_8e2e_c899be07ae49.slice/crio-5f5f0caa9fd65c15b601c352bdf4c557f03c4b17308331931b0d24fef7d600cb WatchSource:0}: Error finding container 5f5f0caa9fd65c15b601c352bdf4c557f03c4b17308331931b0d24fef7d600cb: Status 404 returned error can't find the container with id 5f5f0caa9fd65c15b601c352bdf4c557f03c4b17308331931b0d24fef7d600cb
Nov 03 17:38:54.528221 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:54.528194    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh"]
Nov 03 17:38:55.080763 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.080731    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w" event={"ID":"62446602-8e77-48fc-baff-a974c95e75a4","Type":"ContainerStarted","Data":"307104d533cf8984d8ec34005f935cf70ce5dfaee77a21ac70636120bdafb0d4"}
Nov 03 17:38:55.080975 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.080781    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w" event={"ID":"62446602-8e77-48fc-baff-a974c95e75a4","Type":"ContainerStarted","Data":"68a739f47873e511522ae65d643de71f71268d952a8f451e8dbf26230d6a035a"}
Nov 03 17:38:55.080975 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.080826    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w"
Nov 03 17:38:55.082599 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.082574    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh" event={"ID":"b4bcd4ea-d922-4475-8e2e-c899be07ae49","Type":"ContainerStarted","Data":"587e045046bf3f3af74b812d924d59b70a12ec96e1975b83424fd952bda37b94"}
Nov 03 17:38:55.082729 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.082612    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh" event={"ID":"b4bcd4ea-d922-4475-8e2e-c899be07ae49","Type":"ContainerStarted","Data":"5f5f0caa9fd65c15b601c352bdf4c557f03c4b17308331931b0d24fef7d600cb"}
Nov 03 17:38:55.082729 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.082646    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh"
Nov 03 17:38:55.201960 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.201902    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w" podStartSLOduration=2.201888405 podStartE2EDuration="2.201888405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:38:55.20117319 +0000 UTC m=+1275.998885974" watchObservedRunningTime="2026-11-03 17:38:55.201888405 +0000 UTC m=+1275.999601204"
Nov 03 17:38:55.322112 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.322050    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh" podStartSLOduration=2.322033514 podStartE2EDuration="2.322033514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:38:55.3207575 +0000 UTC m=+1276.118470280" watchObservedRunningTime="2026-11-03 17:38:55.322033514 +0000 UTC m=+1276.119746295"
Nov 03 17:38:55.371942 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.371863    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ab418c-prc9k"]
Nov 03 17:38:55.376081 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.376060    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ab418c-prc9k"
Nov 03 17:38:55.431294 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.431205    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:38:55.431527 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.431328    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:38:55.431660 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.431641    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:38:55.431837 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.431161    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:38:55.436144 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.436121    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ab418c-prc9k"]
Nov 03 17:38:55.456796 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.456763    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdeae13a-f70e-4e1d-8202-d6558ce9b81a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86ab418c-prc9k\" (UID: \"fdeae13a-f70e-4e1d-8202-d6558ce9b81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ab418c-prc9k"
Nov 03 17:38:55.456955 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.456937    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdeae13a-f70e-4e1d-8202-d6558ce9b81a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86ab418c-prc9k\" (UID: \"fdeae13a-f70e-4e1d-8202-d6558ce9b81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ab418c-prc9k"
Nov 03 17:38:55.457114 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.457091    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p8rt\" (UniqueName: \"kubernetes.io/projected/fdeae13a-f70e-4e1d-8202-d6558ce9b81a-kube-api-access-5p8rt\") pod \"maas-api-revoke-sub-86ab418c-prc9k\" (UID: \"fdeae13a-f70e-4e1d-8202-d6558ce9b81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ab418c-prc9k"
Nov 03 17:38:55.558345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.558301    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p8rt\" (UniqueName: \"kubernetes.io/projected/fdeae13a-f70e-4e1d-8202-d6558ce9b81a-kube-api-access-5p8rt\") pod \"maas-api-revoke-sub-86ab418c-prc9k\" (UID: \"fdeae13a-f70e-4e1d-8202-d6558ce9b81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ab418c-prc9k"
Nov 03 17:38:55.559804 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.559762    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdeae13a-f70e-4e1d-8202-d6558ce9b81a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86ab418c-prc9k\" (UID: \"fdeae13a-f70e-4e1d-8202-d6558ce9b81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ab418c-prc9k"
Nov 03 17:38:55.560678 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.560646    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdeae13a-f70e-4e1d-8202-d6558ce9b81a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86ab418c-prc9k\" (UID: \"fdeae13a-f70e-4e1d-8202-d6558ce9b81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ab418c-prc9k"
Nov 03 17:38:55.561931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.561901    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdeae13a-f70e-4e1d-8202-d6558ce9b81a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86ab418c-prc9k\" (UID: \"fdeae13a-f70e-4e1d-8202-d6558ce9b81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ab418c-prc9k"
Nov 03 17:38:55.572754 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.572716    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdeae13a-f70e-4e1d-8202-d6558ce9b81a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86ab418c-prc9k\" (UID: \"fdeae13a-f70e-4e1d-8202-d6558ce9b81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ab418c-prc9k"
Nov 03 17:38:55.635171 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.635084    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p8rt\" (UniqueName: \"kubernetes.io/projected/fdeae13a-f70e-4e1d-8202-d6558ce9b81a-kube-api-access-5p8rt\") pod \"maas-api-revoke-sub-86ab418c-prc9k\" (UID: \"fdeae13a-f70e-4e1d-8202-d6558ce9b81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ab418c-prc9k"
Nov 03 17:38:55.692256 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:55.692211    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ab418c-prc9k"
Nov 03 17:38:56.373005 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:56.372961    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ab418c-prc9k"]
Nov 03 17:38:56.490381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:56.486819    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:38:57.198508 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:57.198462    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ab418c-prc9k" event={"ID":"fdeae13a-f70e-4e1d-8202-d6558ce9b81a","Type":"ContainerStarted","Data":"6d484d3a5a707b1cc50ef52c28452132ca6f8639b5583eeb943d3f411bf68a6b"}
Nov 03 17:38:57.198955 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:57.198540    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ab418c-prc9k" event={"ID":"fdeae13a-f70e-4e1d-8202-d6558ce9b81a","Type":"ContainerStarted","Data":"73398d9c84b0f5c8025321df82dcd54ab7e179c4ba27c924297ba72c000a803a"}
Nov 03 17:38:58.265383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:58.264861    6059 generic.go:356] "Generic (PLEG): container finished" podID="fdeae13a-f70e-4e1d-8202-d6558ce9b81a" containerID="6d484d3a5a707b1cc50ef52c28452132ca6f8639b5583eeb943d3f411bf68a6b" exitCode=0
Nov 03 17:38:58.265383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:58.265104    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ab418c-prc9k" event={"ID":"fdeae13a-f70e-4e1d-8202-d6558ce9b81a","Type":"ContainerDied","Data":"6d484d3a5a707b1cc50ef52c28452132ca6f8639b5583eeb943d3f411bf68a6b"}
Nov 03 17:38:59.480629 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:59.480603    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ab418c-prc9k"
Nov 03 17:38:59.532312 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:59.532220    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdeae13a-f70e-4e1d-8202-d6558ce9b81a-kube-api-access-5p8rt\" (UniqueName: \"kubernetes.io/projected/fdeae13a-f70e-4e1d-8202-d6558ce9b81a-kube-api-access-5p8rt\") pod \"fdeae13a-f70e-4e1d-8202-d6558ce9b81a\" (UID: \"fdeae13a-f70e-4e1d-8202-d6558ce9b81a\") "
Nov 03 17:38:59.532529 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:59.532418    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdeae13a-f70e-4e1d-8202-d6558ce9b81a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdeae13a-f70e-4e1d-8202-d6558ce9b81a-maas-api-xks-ca\") pod \"fdeae13a-f70e-4e1d-8202-d6558ce9b81a\" (UID: \"fdeae13a-f70e-4e1d-8202-d6558ce9b81a\") "
Nov 03 17:38:59.532529 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:59.532464    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdeae13a-f70e-4e1d-8202-d6558ce9b81a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdeae13a-f70e-4e1d-8202-d6558ce9b81a-maas-api-openshift-ca\") pod \"fdeae13a-f70e-4e1d-8202-d6558ce9b81a\" (UID: \"fdeae13a-f70e-4e1d-8202-d6558ce9b81a\") "
Nov 03 17:38:59.532885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:59.532859    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdeae13a-f70e-4e1d-8202-d6558ce9b81a-maas-api-openshift-ca" pod "fdeae13a-f70e-4e1d-8202-d6558ce9b81a" (UID: "fdeae13a-f70e-4e1d-8202-d6558ce9b81a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:38:59.533069 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:59.533054    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdeae13a-f70e-4e1d-8202-d6558ce9b81a-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:59.534309 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:59.534287    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdeae13a-f70e-4e1d-8202-d6558ce9b81a-kube-api-access-5p8rt" pod "fdeae13a-f70e-4e1d-8202-d6558ce9b81a" (UID: "fdeae13a-f70e-4e1d-8202-d6558ce9b81a"). InnerVolumeSpecName "kube-api-access-5p8rt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:38:59.534406 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:59.534335    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdeae13a-f70e-4e1d-8202-d6558ce9b81a-maas-api-xks-ca" pod "fdeae13a-f70e-4e1d-8202-d6558ce9b81a" (UID: "fdeae13a-f70e-4e1d-8202-d6558ce9b81a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:38:59.634742 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:59.634705    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p8rt\" (UniqueName: \"kubernetes.io/projected/fdeae13a-f70e-4e1d-8202-d6558ce9b81a-kube-api-access-5p8rt\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:38:59.634742 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:38:59.634736    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdeae13a-f70e-4e1d-8202-d6558ce9b81a-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:39:00.287383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:00.287329    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ab418c-prc9k"
Nov 03 17:39:00.287571 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:00.287377    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ab418c-prc9k" event={"ID":"fdeae13a-f70e-4e1d-8202-d6558ce9b81a","Type":"ContainerDied","Data":"73398d9c84b0f5c8025321df82dcd54ab7e179c4ba27c924297ba72c000a803a"}
Nov 03 17:39:00.287571 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:00.287431    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73398d9c84b0f5c8025321df82dcd54ab7e179c4ba27c924297ba72c000a803a"
Nov 03 17:39:00.699327 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:00.699247    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw"]
Nov 03 17:39:00.699918 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:00.699898    6059 cpu_manager.go:414] "containerMap: removing container" podUID="fdeae13a-f70e-4e1d-8202-d6558ce9b81a" containerName="revoke-keys"
Nov 03 17:39:00.699963 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:00.699920    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdeae13a-f70e-4e1d-8202-d6558ce9b81a" containerName="revoke-keys"
Nov 03 17:39:00.700017 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:00.700008    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdeae13a-f70e-4e1d-8202-d6558ce9b81a" containerName="revoke-keys"
Nov 03 17:39:00.703935 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:00.703917    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw"
Nov 03 17:39:00.746593 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:00.746559    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn7ls\" (UniqueName: \"kubernetes.io/projected/2234b5bb-0e99-44f3-b050-8f190df93d6e-kube-api-access-qn7ls\") pod \"maas-api-revoke-sub-fd19cff0-rmtdw\" (UID: \"2234b5bb-0e99-44f3-b050-8f190df93d6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw"
Nov 03 17:39:00.746810 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:00.746621    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2234b5bb-0e99-44f3-b050-8f190df93d6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd19cff0-rmtdw\" (UID: \"2234b5bb-0e99-44f3-b050-8f190df93d6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw"
Nov 03 17:39:00.746810 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:00.746758    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2234b5bb-0e99-44f3-b050-8f190df93d6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd19cff0-rmtdw\" (UID: \"2234b5bb-0e99-44f3-b050-8f190df93d6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw"
Nov 03 17:39:00.760310 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:00.760276    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:39:00.760310 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:00.760302    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:39:00.760620 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:00.760342    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:39:00.763680 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:00.763654    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw"]
Nov 03 17:39:00.809828 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:00.809786    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:39:00.847622 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:00.847585    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2234b5bb-0e99-44f3-b050-8f190df93d6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd19cff0-rmtdw\" (UID: \"2234b5bb-0e99-44f3-b050-8f190df93d6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw"
Nov 03 17:39:00.847805 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:00.847643    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2234b5bb-0e99-44f3-b050-8f190df93d6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd19cff0-rmtdw\" (UID: \"2234b5bb-0e99-44f3-b050-8f190df93d6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw"
Nov 03 17:39:00.847923 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:00.847900    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7ls\" (UniqueName: \"kubernetes.io/projected/2234b5bb-0e99-44f3-b050-8f190df93d6e-kube-api-access-qn7ls\") pod \"maas-api-revoke-sub-fd19cff0-rmtdw\" (UID: \"2234b5bb-0e99-44f3-b050-8f190df93d6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw"
Nov 03 17:39:00.848265 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:00.848244    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2234b5bb-0e99-44f3-b050-8f190df93d6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd19cff0-rmtdw\" (UID: \"2234b5bb-0e99-44f3-b050-8f190df93d6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw"
Nov 03 17:39:00.850078 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:00.850053    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2234b5bb-0e99-44f3-b050-8f190df93d6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd19cff0-rmtdw\" (UID: \"2234b5bb-0e99-44f3-b050-8f190df93d6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw"
Nov 03 17:39:00.911515 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:00.911482    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn7ls\" (UniqueName: \"kubernetes.io/projected/2234b5bb-0e99-44f3-b050-8f190df93d6e-kube-api-access-qn7ls\") pod \"maas-api-revoke-sub-fd19cff0-rmtdw\" (UID: \"2234b5bb-0e99-44f3-b050-8f190df93d6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw"
Nov 03 17:39:01.019313 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:01.019279    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw"
Nov 03 17:39:01.394795 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:39:01.394760    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2234b5bb_0e99_44f3_b050_8f190df93d6e.slice/crio-7af938360d2c05527e81f897812de26cec9964a9154c630a7af2a3009b88adb6 WatchSource:0}: Error finding container 7af938360d2c05527e81f897812de26cec9964a9154c630a7af2a3009b88adb6: Status 404 returned error can't find the container with id 7af938360d2c05527e81f897812de26cec9964a9154c630a7af2a3009b88adb6
Nov 03 17:39:01.396998 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:01.396972    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw"]
Nov 03 17:39:01.835882 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:01.835847    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:02.303978 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:02.303944    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw" event={"ID":"2234b5bb-0e99-44f3-b050-8f190df93d6e","Type":"ContainerStarted","Data":"a236d0f363bffcd13a49973e4983cb6955e9f284c32522e7a10cb2863cb3a379"}
Nov 03 17:39:02.304142 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:02.304003    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw" event={"ID":"2234b5bb-0e99-44f3-b050-8f190df93d6e","Type":"ContainerStarted","Data":"7af938360d2c05527e81f897812de26cec9964a9154c630a7af2a3009b88adb6"}
Nov 03 17:39:02.424555 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:02.424495    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw" podStartSLOduration=2.424479713 podStartE2EDuration="2.424479713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:39:02.422576297 +0000 UTC m=+1283.220289078" watchObservedRunningTime="2026-11-03 17:39:02.424479713 +0000 UTC m=+1283.222192493"
Nov 03 17:39:03.312748 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:03.312715    6059 generic.go:356] "Generic (PLEG): container finished" podID="2234b5bb-0e99-44f3-b050-8f190df93d6e" containerID="a236d0f363bffcd13a49973e4983cb6955e9f284c32522e7a10cb2863cb3a379" exitCode=0
Nov 03 17:39:03.313157 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:03.312808    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw" event={"ID":"2234b5bb-0e99-44f3-b050-8f190df93d6e","Type":"ContainerDied","Data":"a236d0f363bffcd13a49973e4983cb6955e9f284c32522e7a10cb2863cb3a379"}
Nov 03 17:39:04.059529 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:04.059495    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf"]
Nov 03 17:39:04.064244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:04.064221    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8f3dc8-l7pzf"]
Nov 03 17:39:04.483484 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:04.483459    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw"
Nov 03 17:39:04.594337 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:04.594248    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2234b5bb-0e99-44f3-b050-8f190df93d6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2234b5bb-0e99-44f3-b050-8f190df93d6e-maas-api-openshift-ca\") pod \"2234b5bb-0e99-44f3-b050-8f190df93d6e\" (UID: \"2234b5bb-0e99-44f3-b050-8f190df93d6e\") "
Nov 03 17:39:04.594337 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:04.594324    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2234b5bb-0e99-44f3-b050-8f190df93d6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2234b5bb-0e99-44f3-b050-8f190df93d6e-maas-api-xks-ca\") pod \"2234b5bb-0e99-44f3-b050-8f190df93d6e\" (UID: \"2234b5bb-0e99-44f3-b050-8f190df93d6e\") "
Nov 03 17:39:04.594605 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:04.594454    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2234b5bb-0e99-44f3-b050-8f190df93d6e-kube-api-access-qn7ls\" (UniqueName: \"kubernetes.io/projected/2234b5bb-0e99-44f3-b050-8f190df93d6e-kube-api-access-qn7ls\") pod \"2234b5bb-0e99-44f3-b050-8f190df93d6e\" (UID: \"2234b5bb-0e99-44f3-b050-8f190df93d6e\") "
Nov 03 17:39:04.594685 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:04.594659    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2234b5bb-0e99-44f3-b050-8f190df93d6e-maas-api-openshift-ca" pod "2234b5bb-0e99-44f3-b050-8f190df93d6e" (UID: "2234b5bb-0e99-44f3-b050-8f190df93d6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:39:04.594841 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:04.594826    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2234b5bb-0e99-44f3-b050-8f190df93d6e-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:39:04.596559 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:04.596537    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2234b5bb-0e99-44f3-b050-8f190df93d6e-maas-api-xks-ca" pod "2234b5bb-0e99-44f3-b050-8f190df93d6e" (UID: "2234b5bb-0e99-44f3-b050-8f190df93d6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:39:04.596691 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:04.596672    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2234b5bb-0e99-44f3-b050-8f190df93d6e-kube-api-access-qn7ls" pod "2234b5bb-0e99-44f3-b050-8f190df93d6e" (UID: "2234b5bb-0e99-44f3-b050-8f190df93d6e"). InnerVolumeSpecName "kube-api-access-qn7ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:39:04.696533 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:04.696499    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn7ls\" (UniqueName: \"kubernetes.io/projected/2234b5bb-0e99-44f3-b050-8f190df93d6e-kube-api-access-qn7ls\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:39:04.696533 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:04.696529    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2234b5bb-0e99-44f3-b050-8f190df93d6e-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:39:05.328208 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:05.328174    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw"
Nov 03 17:39:05.335986 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:05.335953    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="031c7311-179c-4fdc-9911-fd6c0743ce8a" path="/var/lib/kubelet/pods/031c7311-179c-4fdc-9911-fd6c0743ce8a/volumes"
Nov 03 17:39:05.336560 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:05.336533    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw" event={"ID":"2234b5bb-0e99-44f3-b050-8f190df93d6e","Type":"ContainerDied","Data":"7af938360d2c05527e81f897812de26cec9964a9154c630a7af2a3009b88adb6"}
Nov 03 17:39:05.336656 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:05.336582    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7af938360d2c05527e81f897812de26cec9964a9154c630a7af2a3009b88adb6"
Nov 03 17:39:06.150327 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:06.150297    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w"
Nov 03 17:39:06.150719 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:06.150383    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh"
Nov 03 17:39:06.417251 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:06.417164    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:06.862105 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:39:06.862065    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:39:10.959934 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:10.959898    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:13.167950 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:13.167894    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n" podUID="d6c1263b-e648-4f10-af7d-86397d6b207a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:39:13.412487 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:13.412436    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w" podUID="6120f11d-14c8-4666-a201-469a5bca37de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:39:13.424565 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:13.424467    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v" podUID="2bbc9082-b2da-4198-8a8a-b6ab26b5011e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:39:13.477379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:13.453462    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl" podUID="c194a4be-0499-419f-a4a1-910dbaafd303" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:39:14.171393 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:14.170457    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n" podUID="d6c1263b-e648-4f10-af7d-86397d6b207a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:39:14.171393 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:14.170893    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:39:14.221381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:14.217460    6059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v" podUID="2bbc9082-b2da-4198-8a8a-b6ab26b5011e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:39:14.421455 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:14.398456    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv" podUID="7942e9d1-597f-47a6-9c70-1d174fe4c8e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:39:14.595381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:14.572877    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w" podUID="6120f11d-14c8-4666-a201-469a5bca37de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded"
Nov 03 17:39:14.595875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:14.590156    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v" podUID="2bbc9082-b2da-4198-8a8a-b6ab26b5011e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 17:39:14.596643 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:14.596621    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:39:14.596855 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:14.596835    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:39:14.620660 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:14.620629    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:39:14.827844 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:14.827799    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:39:14.913965 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:14.913502    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:39:14.964381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:14.954509    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:39:14.964381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:14.954577    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c" podUID="84690202-b89e-4135-aebb-aa53c0e18463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:39:16.203088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:16.203048    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl"]
Nov 03 17:39:16.203919 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:16.203896    6059 cpu_manager.go:414] "containerMap: removing container" podUID="2234b5bb-0e99-44f3-b050-8f190df93d6e" containerName="revoke-keys"
Nov 03 17:39:16.204022 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:16.203919    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2234b5bb-0e99-44f3-b050-8f190df93d6e" containerName="revoke-keys"
Nov 03 17:39:16.204107 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:16.204094    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2234b5bb-0e99-44f3-b050-8f190df93d6e" containerName="revoke-keys"
Nov 03 17:39:16.208857 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:16.208836    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl"
Nov 03 17:39:16.277621 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:16.277587    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl"]
Nov 03 17:39:16.277895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:16.277602    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:39:16.278075 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:16.277647    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:39:16.278075 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:16.277693    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:39:16.294804 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:16.294773    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlf44\" (UniqueName: \"kubernetes.io/projected/3ff1efbd-eeae-4b26-9cb8-160edf44d4b3-kube-api-access-vlf44\") pod \"maas-api-revoke-sub-80282998-zdfcl\" (UID: \"3ff1efbd-eeae-4b26-9cb8-160edf44d4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl"
Nov 03 17:39:16.294963 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:16.294838    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ff1efbd-eeae-4b26-9cb8-160edf44d4b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80282998-zdfcl\" (UID: \"3ff1efbd-eeae-4b26-9cb8-160edf44d4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl"
Nov 03 17:39:16.295083 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:16.295062    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ff1efbd-eeae-4b26-9cb8-160edf44d4b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80282998-zdfcl\" (UID: \"3ff1efbd-eeae-4b26-9cb8-160edf44d4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl"
Nov 03 17:39:16.320732 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:16.320698    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:39:16.376517 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:16.376471    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:16.396380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:16.396330    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlf44\" (UniqueName: \"kubernetes.io/projected/3ff1efbd-eeae-4b26-9cb8-160edf44d4b3-kube-api-access-vlf44\") pod \"maas-api-revoke-sub-80282998-zdfcl\" (UID: \"3ff1efbd-eeae-4b26-9cb8-160edf44d4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl"
Nov 03 17:39:16.396540 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:16.396445    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ff1efbd-eeae-4b26-9cb8-160edf44d4b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80282998-zdfcl\" (UID: \"3ff1efbd-eeae-4b26-9cb8-160edf44d4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl"
Nov 03 17:39:16.396613 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:16.396598    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ff1efbd-eeae-4b26-9cb8-160edf44d4b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80282998-zdfcl\" (UID: \"3ff1efbd-eeae-4b26-9cb8-160edf44d4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl"
Nov 03 17:39:16.397101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:16.397080    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ff1efbd-eeae-4b26-9cb8-160edf44d4b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80282998-zdfcl\" (UID: \"3ff1efbd-eeae-4b26-9cb8-160edf44d4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl"
Nov 03 17:39:16.398993 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:16.398969    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ff1efbd-eeae-4b26-9cb8-160edf44d4b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80282998-zdfcl\" (UID: \"3ff1efbd-eeae-4b26-9cb8-160edf44d4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl"
Nov 03 17:39:16.459755 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:16.459728    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlf44\" (UniqueName: \"kubernetes.io/projected/3ff1efbd-eeae-4b26-9cb8-160edf44d4b3-kube-api-access-vlf44\") pod \"maas-api-revoke-sub-80282998-zdfcl\" (UID: \"3ff1efbd-eeae-4b26-9cb8-160edf44d4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl"
Nov 03 17:39:16.523071 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:16.523037    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl"
Nov 03 17:39:16.898734 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:39:16.898701    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff1efbd_eeae_4b26_9cb8_160edf44d4b3.slice/crio-c26c2012d80e191e0b756c601933fae54fae4e82a757543ff8f384cc743097ec WatchSource:0}: Error finding container c26c2012d80e191e0b756c601933fae54fae4e82a757543ff8f384cc743097ec: Status 404 returned error can't find the container with id c26c2012d80e191e0b756c601933fae54fae4e82a757543ff8f384cc743097ec
Nov 03 17:39:16.902745 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:16.902713    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl"]
Nov 03 17:39:17.626904 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:17.626859    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl" event={"ID":"3ff1efbd-eeae-4b26-9cb8-160edf44d4b3","Type":"ContainerStarted","Data":"8c87a2c1de7b4ef354fc8a84264d5771ac91179a03806839abb335ce654ca2e1"}
Nov 03 17:39:17.627395 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:17.626932    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl" event={"ID":"3ff1efbd-eeae-4b26-9cb8-160edf44d4b3","Type":"ContainerStarted","Data":"c26c2012d80e191e0b756c601933fae54fae4e82a757543ff8f384cc743097ec"}
Nov 03 17:39:17.749096 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:17.749044    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl" podStartSLOduration=1.74903071 podStartE2EDuration="1.74903071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:39:17.746578748 +0000 UTC m=+1298.544291530" watchObservedRunningTime="2026-11-03 17:39:17.74903071 +0000 UTC m=+1298.546743488"
Nov 03 17:39:18.635437 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:18.635398    6059 generic.go:356] "Generic (PLEG): container finished" podID="3ff1efbd-eeae-4b26-9cb8-160edf44d4b3" containerID="8c87a2c1de7b4ef354fc8a84264d5771ac91179a03806839abb335ce654ca2e1" exitCode=0
Nov 03 17:39:18.635834 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:18.635471    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl" event={"ID":"3ff1efbd-eeae-4b26-9cb8-160edf44d4b3","Type":"ContainerDied","Data":"8c87a2c1de7b4ef354fc8a84264d5771ac91179a03806839abb335ce654ca2e1"}
Nov 03 17:39:19.803729 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:19.803698    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl"
Nov 03 17:39:19.940853 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:19.940760    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff1efbd-eeae-4b26-9cb8-160edf44d4b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ff1efbd-eeae-4b26-9cb8-160edf44d4b3-maas-api-openshift-ca\") pod \"3ff1efbd-eeae-4b26-9cb8-160edf44d4b3\" (UID: \"3ff1efbd-eeae-4b26-9cb8-160edf44d4b3\") "
Nov 03 17:39:19.941007 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:19.940954    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff1efbd-eeae-4b26-9cb8-160edf44d4b3-kube-api-access-vlf44\" (UniqueName: \"kubernetes.io/projected/3ff1efbd-eeae-4b26-9cb8-160edf44d4b3-kube-api-access-vlf44\") pod \"3ff1efbd-eeae-4b26-9cb8-160edf44d4b3\" (UID: \"3ff1efbd-eeae-4b26-9cb8-160edf44d4b3\") "
Nov 03 17:39:19.941101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:19.941079    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff1efbd-eeae-4b26-9cb8-160edf44d4b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ff1efbd-eeae-4b26-9cb8-160edf44d4b3-maas-api-xks-ca\") pod \"3ff1efbd-eeae-4b26-9cb8-160edf44d4b3\" (UID: \"3ff1efbd-eeae-4b26-9cb8-160edf44d4b3\") "
Nov 03 17:39:19.941342 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:19.941139    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff1efbd-eeae-4b26-9cb8-160edf44d4b3-maas-api-openshift-ca" pod "3ff1efbd-eeae-4b26-9cb8-160edf44d4b3" (UID: "3ff1efbd-eeae-4b26-9cb8-160edf44d4b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:39:19.941895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:19.941819    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ff1efbd-eeae-4b26-9cb8-160edf44d4b3-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:39:19.943711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:19.943680    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff1efbd-eeae-4b26-9cb8-160edf44d4b3-maas-api-xks-ca" pod "3ff1efbd-eeae-4b26-9cb8-160edf44d4b3" (UID: "3ff1efbd-eeae-4b26-9cb8-160edf44d4b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:39:19.943849 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:19.943828    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff1efbd-eeae-4b26-9cb8-160edf44d4b3-kube-api-access-vlf44" pod "3ff1efbd-eeae-4b26-9cb8-160edf44d4b3" (UID: "3ff1efbd-eeae-4b26-9cb8-160edf44d4b3"). InnerVolumeSpecName "kube-api-access-vlf44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:39:20.042863 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:20.042831    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlf44\" (UniqueName: \"kubernetes.io/projected/3ff1efbd-eeae-4b26-9cb8-160edf44d4b3-kube-api-access-vlf44\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:39:20.042863 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:20.042861    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ff1efbd-eeae-4b26-9cb8-160edf44d4b3-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:39:20.053471 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:20.053438    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd"]
Nov 03 17:39:20.059285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:20.059258    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422063f1-f4ctd"]
Nov 03 17:39:20.660044 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:20.660016    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl"
Nov 03 17:39:20.660223 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:20.660042    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl" event={"ID":"3ff1efbd-eeae-4b26-9cb8-160edf44d4b3","Type":"ContainerDied","Data":"c26c2012d80e191e0b756c601933fae54fae4e82a757543ff8f384cc743097ec"}
Nov 03 17:39:20.660223 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:20.660085    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c26c2012d80e191e0b756c601933fae54fae4e82a757543ff8f384cc743097ec"
Nov 03 17:39:20.827950 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:20.827917    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:21.337078 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:21.337046    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="018a023a-a6d2-4189-9d3f-4c9820addd27" path="/var/lib/kubelet/pods/018a023a-a6d2-4189-9d3f-4c9820addd27/volumes"
Nov 03 17:39:25.194274 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.194237    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf"]
Nov 03 17:39:25.194879 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.194864    6059 cpu_manager.go:414] "containerMap: removing container" podUID="3ff1efbd-eeae-4b26-9cb8-160edf44d4b3" containerName="revoke-keys"
Nov 03 17:39:25.194928 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.194880    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff1efbd-eeae-4b26-9cb8-160edf44d4b3" containerName="revoke-keys"
Nov 03 17:39:25.194994 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.194984    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff1efbd-eeae-4b26-9cb8-160edf44d4b3" containerName="revoke-keys"
Nov 03 17:39:25.198627 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.198611    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf"
Nov 03 17:39:25.206479 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.206454    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7972cf5-71cb-436b-bc1c-754a24d025fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-511b856d-mmbmf\" (UID: \"d7972cf5-71cb-436b-bc1c-754a24d025fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf"
Nov 03 17:39:25.206716 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.206686    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7972cf5-71cb-436b-bc1c-754a24d025fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-511b856d-mmbmf\" (UID: \"d7972cf5-71cb-436b-bc1c-754a24d025fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf"
Nov 03 17:39:25.206803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.206741    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhxnt\" (UniqueName: \"kubernetes.io/projected/d7972cf5-71cb-436b-bc1c-754a24d025fc-kube-api-access-dhxnt\") pod \"maas-api-revoke-sub-511b856d-mmbmf\" (UID: \"d7972cf5-71cb-436b-bc1c-754a24d025fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf"
Nov 03 17:39:25.258516 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.258483    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf"]
Nov 03 17:39:25.298695 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.298663    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:25.305029 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.305000    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:39:25.305029 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.305006    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:39:25.305029 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.305028    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:39:25.305247 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.305070    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:39:25.307835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.307806    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7972cf5-71cb-436b-bc1c-754a24d025fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-511b856d-mmbmf\" (UID: \"d7972cf5-71cb-436b-bc1c-754a24d025fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf"
Nov 03 17:39:25.307956 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.307854    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhxnt\" (UniqueName: \"kubernetes.io/projected/d7972cf5-71cb-436b-bc1c-754a24d025fc-kube-api-access-dhxnt\") pod \"maas-api-revoke-sub-511b856d-mmbmf\" (UID: \"d7972cf5-71cb-436b-bc1c-754a24d025fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf"
Nov 03 17:39:25.307956 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.307917    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7972cf5-71cb-436b-bc1c-754a24d025fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-511b856d-mmbmf\" (UID: \"d7972cf5-71cb-436b-bc1c-754a24d025fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf"
Nov 03 17:39:25.308522 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.308504    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7972cf5-71cb-436b-bc1c-754a24d025fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-511b856d-mmbmf\" (UID: \"d7972cf5-71cb-436b-bc1c-754a24d025fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf"
Nov 03 17:39:25.310279 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.310258    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7972cf5-71cb-436b-bc1c-754a24d025fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-511b856d-mmbmf\" (UID: \"d7972cf5-71cb-436b-bc1c-754a24d025fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf"
Nov 03 17:39:25.373103 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.373074    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhxnt\" (UniqueName: \"kubernetes.io/projected/d7972cf5-71cb-436b-bc1c-754a24d025fc-kube-api-access-dhxnt\") pod \"maas-api-revoke-sub-511b856d-mmbmf\" (UID: \"d7972cf5-71cb-436b-bc1c-754a24d025fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf"
Nov 03 17:39:25.480634 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.480603    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:25.513512 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.513471    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf"
Nov 03 17:39:25.668551 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.668516    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:25.890840 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:39:25.890811    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7972cf5_71cb_436b_bc1c_754a24d025fc.slice/crio-d0c76c5fd4f197952f22d3b5d88d32bb589f954672c4301ea9ae6ceb963846b1 WatchSource:0}: Error finding container d0c76c5fd4f197952f22d3b5d88d32bb589f954672c4301ea9ae6ceb963846b1: Status 404 returned error can't find the container with id d0c76c5fd4f197952f22d3b5d88d32bb589f954672c4301ea9ae6ceb963846b1
Nov 03 17:39:25.895533 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:25.895508    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf"]
Nov 03 17:39:26.713277 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:26.713242    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf" event={"ID":"d7972cf5-71cb-436b-bc1c-754a24d025fc","Type":"ContainerStarted","Data":"a104a9ddd56ca96b02d8733540d318115986928bf0a40f23a3805e86e8436cb4"}
Nov 03 17:39:26.713695 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:26.713305    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf" event={"ID":"d7972cf5-71cb-436b-bc1c-754a24d025fc","Type":"ContainerStarted","Data":"d0c76c5fd4f197952f22d3b5d88d32bb589f954672c4301ea9ae6ceb963846b1"}
Nov 03 17:39:26.835313 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:26.835253    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf" podStartSLOduration=1.8352359040000001 podStartE2EDuration="1.835235904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:39:26.832595728 +0000 UTC m=+1307.630308513" watchObservedRunningTime="2026-11-03 17:39:26.835235904 +0000 UTC m=+1307.632948691"
Nov 03 17:39:27.722270 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:27.722235    6059 generic.go:356] "Generic (PLEG): container finished" podID="d7972cf5-71cb-436b-bc1c-754a24d025fc" containerID="a104a9ddd56ca96b02d8733540d318115986928bf0a40f23a3805e86e8436cb4" exitCode=0
Nov 03 17:39:27.722710 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:27.722321    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf" event={"ID":"d7972cf5-71cb-436b-bc1c-754a24d025fc","Type":"ContainerDied","Data":"a104a9ddd56ca96b02d8733540d318115986928bf0a40f23a3805e86e8436cb4"}
Nov 03 17:39:28.893451 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:28.893422    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf"
Nov 03 17:39:28.950685 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:28.950654    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7972cf5-71cb-436b-bc1c-754a24d025fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7972cf5-71cb-436b-bc1c-754a24d025fc-maas-api-xks-ca\") pod \"d7972cf5-71cb-436b-bc1c-754a24d025fc\" (UID: \"d7972cf5-71cb-436b-bc1c-754a24d025fc\") "
Nov 03 17:39:28.950890 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:28.950708    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7972cf5-71cb-436b-bc1c-754a24d025fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7972cf5-71cb-436b-bc1c-754a24d025fc-maas-api-openshift-ca\") pod \"d7972cf5-71cb-436b-bc1c-754a24d025fc\" (UID: \"d7972cf5-71cb-436b-bc1c-754a24d025fc\") "
Nov 03 17:39:28.950890 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:28.950779    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7972cf5-71cb-436b-bc1c-754a24d025fc-kube-api-access-dhxnt\" (UniqueName: \"kubernetes.io/projected/d7972cf5-71cb-436b-bc1c-754a24d025fc-kube-api-access-dhxnt\") pod \"d7972cf5-71cb-436b-bc1c-754a24d025fc\" (UID: \"d7972cf5-71cb-436b-bc1c-754a24d025fc\") "
Nov 03 17:39:28.951224 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:28.951193    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7972cf5-71cb-436b-bc1c-754a24d025fc-maas-api-openshift-ca" pod "d7972cf5-71cb-436b-bc1c-754a24d025fc" (UID: "d7972cf5-71cb-436b-bc1c-754a24d025fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:39:28.951390 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:28.951323    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7972cf5-71cb-436b-bc1c-754a24d025fc-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:39:28.952756 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:28.952723    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7972cf5-71cb-436b-bc1c-754a24d025fc-maas-api-xks-ca" pod "d7972cf5-71cb-436b-bc1c-754a24d025fc" (UID: "d7972cf5-71cb-436b-bc1c-754a24d025fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:39:28.952867 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:28.952845    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7972cf5-71cb-436b-bc1c-754a24d025fc-kube-api-access-dhxnt" pod "d7972cf5-71cb-436b-bc1c-754a24d025fc" (UID: "d7972cf5-71cb-436b-bc1c-754a24d025fc"). InnerVolumeSpecName "kube-api-access-dhxnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:39:29.061267 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:29.061168    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7972cf5-71cb-436b-bc1c-754a24d025fc-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:39:29.061267 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:29.061207    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhxnt\" (UniqueName: \"kubernetes.io/projected/d7972cf5-71cb-436b-bc1c-754a24d025fc-kube-api-access-dhxnt\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:39:29.071207 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:29.071166    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll"]
Nov 03 17:39:29.074787 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:29.074762    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc27644-ghpll"]
Nov 03 17:39:29.338878 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:29.338794    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d" path="/var/lib/kubelet/pods/4ec91acf-4d9b-43cd-a9f4-f8ce43212b9d/volumes"
Nov 03 17:39:29.739526 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:29.739495    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf"
Nov 03 17:39:29.739688 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:29.739523    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf" event={"ID":"d7972cf5-71cb-436b-bc1c-754a24d025fc","Type":"ContainerDied","Data":"d0c76c5fd4f197952f22d3b5d88d32bb589f954672c4301ea9ae6ceb963846b1"}
Nov 03 17:39:29.739688 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:29.739578    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0c76c5fd4f197952f22d3b5d88d32bb589f954672c4301ea9ae6ceb963846b1"
Nov 03 17:39:30.041327 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:30.041243    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:30.063164 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:30.063130    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c904b31-926pc"]
Nov 03 17:39:30.068667 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:30.068642    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c904b31-926pc"]
Nov 03 17:39:31.371381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:31.370725    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7676565-4063-4272-8929-6e25a934ec5d" path="/var/lib/kubelet/pods/d7676565-4063-4272-8929-6e25a934ec5d/volumes"
Nov 03 17:39:32.420382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:32.413917    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv" podUID="7942e9d1-597f-47a6-9c70-1d174fe4c8e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:39:32.437330 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:32.436507    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl" podUID="c194a4be-0499-419f-a4a1-910dbaafd303" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:39:32.630380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:32.629503    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v" podUID="2bbc9082-b2da-4198-8a8a-b6ab26b5011e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:39:32.880382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:32.879801    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w" podUID="6120f11d-14c8-4666-a201-469a5bca37de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:39:32.899615 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:32.899560    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n" podUID="d6c1263b-e648-4f10-af7d-86397d6b207a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:39:34.347415 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:34.347378    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2"]
Nov 03 17:39:34.347966 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:34.347952    6059 cpu_manager.go:414] "containerMap: removing container" podUID="d7972cf5-71cb-436b-bc1c-754a24d025fc" containerName="revoke-keys"
Nov 03 17:39:34.348018 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:34.347967    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7972cf5-71cb-436b-bc1c-754a24d025fc" containerName="revoke-keys"
Nov 03 17:39:34.348074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:34.348065    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7972cf5-71cb-436b-bc1c-754a24d025fc" containerName="revoke-keys"
Nov 03 17:39:34.352761 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:34.352744    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2"
Nov 03 17:39:34.409824 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:34.409792    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s588h\" (UniqueName: \"kubernetes.io/projected/90049890-ff52-4463-b6cf-e87f76ee3190-kube-api-access-s588h\") pod \"maas-api-revoke-sub-0c742ec8-bk7w2\" (UID: \"90049890-ff52-4463-b6cf-e87f76ee3190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2"
Nov 03 17:39:34.410285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:34.410261    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90049890-ff52-4463-b6cf-e87f76ee3190-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c742ec8-bk7w2\" (UID: \"90049890-ff52-4463-b6cf-e87f76ee3190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2"
Nov 03 17:39:34.410406 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:34.410348    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90049890-ff52-4463-b6cf-e87f76ee3190-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c742ec8-bk7w2\" (UID: \"90049890-ff52-4463-b6cf-e87f76ee3190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2"
Nov 03 17:39:34.412249 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:34.412222    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2"]
Nov 03 17:39:34.458343 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:34.458314    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:39:34.458343 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:34.458315    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:39:34.458343 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:34.458337    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:39:34.458587 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:34.458489    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:39:34.511818 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:34.511781    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s588h\" (UniqueName: \"kubernetes.io/projected/90049890-ff52-4463-b6cf-e87f76ee3190-kube-api-access-s588h\") pod \"maas-api-revoke-sub-0c742ec8-bk7w2\" (UID: \"90049890-ff52-4463-b6cf-e87f76ee3190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2"
Nov 03 17:39:34.511953 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:34.511904    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90049890-ff52-4463-b6cf-e87f76ee3190-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c742ec8-bk7w2\" (UID: \"90049890-ff52-4463-b6cf-e87f76ee3190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2"
Nov 03 17:39:34.512015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:34.511974    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90049890-ff52-4463-b6cf-e87f76ee3190-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c742ec8-bk7w2\" (UID: \"90049890-ff52-4463-b6cf-e87f76ee3190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2"
Nov 03 17:39:34.512682 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:34.512663    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90049890-ff52-4463-b6cf-e87f76ee3190-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c742ec8-bk7w2\" (UID: \"90049890-ff52-4463-b6cf-e87f76ee3190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2"
Nov 03 17:39:34.514286 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:34.514267    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90049890-ff52-4463-b6cf-e87f76ee3190-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c742ec8-bk7w2\" (UID: \"90049890-ff52-4463-b6cf-e87f76ee3190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2"
Nov 03 17:39:34.574938 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:34.574913    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s588h\" (UniqueName: \"kubernetes.io/projected/90049890-ff52-4463-b6cf-e87f76ee3190-kube-api-access-s588h\") pod \"maas-api-revoke-sub-0c742ec8-bk7w2\" (UID: \"90049890-ff52-4463-b6cf-e87f76ee3190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2"
Nov 03 17:39:34.666486 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:34.666416    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2"
Nov 03 17:39:35.045449 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:39:35.045415    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90049890_ff52_4463_b6cf_e87f76ee3190.slice/crio-420c407e4c19461243e260272d3388c6f0260c02f6e642c3b8f826c56d0a7894 WatchSource:0}: Error finding container 420c407e4c19461243e260272d3388c6f0260c02f6e642c3b8f826c56d0a7894: Status 404 returned error can't find the container with id 420c407e4c19461243e260272d3388c6f0260c02f6e642c3b8f826c56d0a7894
Nov 03 17:39:35.047683 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:35.047656    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2"]
Nov 03 17:39:35.416746 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:35.416712    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:35.854406 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:35.854344    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2" event={"ID":"90049890-ff52-4463-b6cf-e87f76ee3190","Type":"ContainerStarted","Data":"dc3f25b7b60f61a336a4d4a0824c32d8331738e6c3f1fbe69d4b84fb9fe73f20"}
Nov 03 17:39:35.854605 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:35.854422    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2" event={"ID":"90049890-ff52-4463-b6cf-e87f76ee3190","Type":"ContainerStarted","Data":"420c407e4c19461243e260272d3388c6f0260c02f6e642c3b8f826c56d0a7894"}
Nov 03 17:39:35.979260 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:35.979206    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2" podStartSLOduration=1.979190019 podStartE2EDuration="1.979190019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:39:35.977129894 +0000 UTC m=+1316.774842676" watchObservedRunningTime="2026-11-03 17:39:35.979190019 +0000 UTC m=+1316.776902800"
Nov 03 17:39:36.862249 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:39:36.862211    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:39:36.864630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:36.864607    6059 generic.go:356] "Generic (PLEG): container finished" podID="90049890-ff52-4463-b6cf-e87f76ee3190" containerID="dc3f25b7b60f61a336a4d4a0824c32d8331738e6c3f1fbe69d4b84fb9fe73f20" exitCode=0
Nov 03 17:39:36.864759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:36.864690    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2" event={"ID":"90049890-ff52-4463-b6cf-e87f76ee3190","Type":"ContainerDied","Data":"dc3f25b7b60f61a336a4d4a0824c32d8331738e6c3f1fbe69d4b84fb9fe73f20"}
Nov 03 17:39:37.690776 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:37.690739    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:38.025689 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:38.025659    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2"
Nov 03 17:39:38.055299 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:38.055269    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90049890-ff52-4463-b6cf-e87f76ee3190-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90049890-ff52-4463-b6cf-e87f76ee3190-maas-api-openshift-ca\") pod \"90049890-ff52-4463-b6cf-e87f76ee3190\" (UID: \"90049890-ff52-4463-b6cf-e87f76ee3190\") "
Nov 03 17:39:38.055531 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:38.055510    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90049890-ff52-4463-b6cf-e87f76ee3190-kube-api-access-s588h\" (UniqueName: \"kubernetes.io/projected/90049890-ff52-4463-b6cf-e87f76ee3190-kube-api-access-s588h\") pod \"90049890-ff52-4463-b6cf-e87f76ee3190\" (UID: \"90049890-ff52-4463-b6cf-e87f76ee3190\") "
Nov 03 17:39:38.055649 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:38.055624    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90049890-ff52-4463-b6cf-e87f76ee3190-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90049890-ff52-4463-b6cf-e87f76ee3190-maas-api-xks-ca\") pod \"90049890-ff52-4463-b6cf-e87f76ee3190\" (UID: \"90049890-ff52-4463-b6cf-e87f76ee3190\") "
Nov 03 17:39:38.055717 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:38.055695    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90049890-ff52-4463-b6cf-e87f76ee3190-maas-api-openshift-ca" pod "90049890-ff52-4463-b6cf-e87f76ee3190" (UID: "90049890-ff52-4463-b6cf-e87f76ee3190"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:39:38.056057 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:38.056029    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90049890-ff52-4463-b6cf-e87f76ee3190-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:39:38.057654 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:38.057624    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90049890-ff52-4463-b6cf-e87f76ee3190-kube-api-access-s588h" pod "90049890-ff52-4463-b6cf-e87f76ee3190" (UID: "90049890-ff52-4463-b6cf-e87f76ee3190"). InnerVolumeSpecName "kube-api-access-s588h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:39:38.057754 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:38.057654    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90049890-ff52-4463-b6cf-e87f76ee3190-maas-api-xks-ca" pod "90049890-ff52-4463-b6cf-e87f76ee3190" (UID: "90049890-ff52-4463-b6cf-e87f76ee3190"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:39:38.157896 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:38.157861    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s588h\" (UniqueName: \"kubernetes.io/projected/90049890-ff52-4463-b6cf-e87f76ee3190-kube-api-access-s588h\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:39:38.157896 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:38.157894    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90049890-ff52-4463-b6cf-e87f76ee3190-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:39:38.883105 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:38.883074    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2"
Nov 03 17:39:38.883277 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:38.883107    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2" event={"ID":"90049890-ff52-4463-b6cf-e87f76ee3190","Type":"ContainerDied","Data":"420c407e4c19461243e260272d3388c6f0260c02f6e642c3b8f826c56d0a7894"}
Nov 03 17:39:38.883277 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:38.883157    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="420c407e4c19461243e260272d3388c6f0260c02f6e642c3b8f826c56d0a7894"
Nov 03 17:39:39.204680 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.204582    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3dec6e8-nppxc"]
Nov 03 17:39:39.205213 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.205196    6059 cpu_manager.go:414] "containerMap: removing container" podUID="90049890-ff52-4463-b6cf-e87f76ee3190" containerName="revoke-keys"
Nov 03 17:39:39.205275 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.205214    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90049890-ff52-4463-b6cf-e87f76ee3190" containerName="revoke-keys"
Nov 03 17:39:39.205340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.205320    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="90049890-ff52-4463-b6cf-e87f76ee3190" containerName="revoke-keys"
Nov 03 17:39:39.209052 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.209032    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3dec6e8-nppxc"
Nov 03 17:39:39.263751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.263722    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:39:39.263923 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.263790    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:39:39.263923 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.263838    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:39:39.265754 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.265736    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:39:39.271552 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.271517    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq5bd\" (UniqueName: \"kubernetes.io/projected/924ff06e-596a-432d-962a-bb7cb7f64d7c-kube-api-access-gq5bd\") pod \"maas-api-revoke-sub-b3dec6e8-nppxc\" (UID: \"924ff06e-596a-432d-962a-bb7cb7f64d7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3dec6e8-nppxc"
Nov 03 17:39:39.271675 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.271609    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924ff06e-596a-432d-962a-bb7cb7f64d7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3dec6e8-nppxc\" (UID: \"924ff06e-596a-432d-962a-bb7cb7f64d7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3dec6e8-nppxc"
Nov 03 17:39:39.271865 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.271840    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924ff06e-596a-432d-962a-bb7cb7f64d7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3dec6e8-nppxc\" (UID: \"924ff06e-596a-432d-962a-bb7cb7f64d7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3dec6e8-nppxc"
Nov 03 17:39:39.319545 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.319511    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3dec6e8-nppxc"]
Nov 03 17:39:39.372925 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.372888    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924ff06e-596a-432d-962a-bb7cb7f64d7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3dec6e8-nppxc\" (UID: \"924ff06e-596a-432d-962a-bb7cb7f64d7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3dec6e8-nppxc"
Nov 03 17:39:39.373092 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.372973    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq5bd\" (UniqueName: \"kubernetes.io/projected/924ff06e-596a-432d-962a-bb7cb7f64d7c-kube-api-access-gq5bd\") pod \"maas-api-revoke-sub-b3dec6e8-nppxc\" (UID: \"924ff06e-596a-432d-962a-bb7cb7f64d7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3dec6e8-nppxc"
Nov 03 17:39:39.373092 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.372992    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924ff06e-596a-432d-962a-bb7cb7f64d7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3dec6e8-nppxc\" (UID: \"924ff06e-596a-432d-962a-bb7cb7f64d7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3dec6e8-nppxc"
Nov 03 17:39:39.426184 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.426152    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:39:39.426397 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.426275    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:39:39.434594 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.434562    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924ff06e-596a-432d-962a-bb7cb7f64d7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3dec6e8-nppxc\" (UID: \"924ff06e-596a-432d-962a-bb7cb7f64d7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3dec6e8-nppxc"
Nov 03 17:39:39.436595 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.436571    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924ff06e-596a-432d-962a-bb7cb7f64d7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3dec6e8-nppxc\" (UID: \"924ff06e-596a-432d-962a-bb7cb7f64d7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3dec6e8-nppxc"
Nov 03 17:39:39.487772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.487746    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:39:39.497164 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.497137    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq5bd\" (UniqueName: \"kubernetes.io/projected/924ff06e-596a-432d-962a-bb7cb7f64d7c-kube-api-access-gq5bd\") pod \"maas-api-revoke-sub-b3dec6e8-nppxc\" (UID: \"924ff06e-596a-432d-962a-bb7cb7f64d7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3dec6e8-nppxc"
Nov 03 17:39:39.575863 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.575833    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:39:39.582943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.582915    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3dec6e8-nppxc"
Nov 03 17:39:39.939380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.939261    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:39.966435 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:39.966396    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3dec6e8-nppxc"]
Nov 03 17:39:39.969178 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:39:39.969150    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924ff06e_596a_432d_962a_bb7cb7f64d7c.slice/crio-f4b4fa90f940ab7337e4878eb7f801ced677325890fdcc3b785e1442a4c3ce79 WatchSource:0}: Error finding container f4b4fa90f940ab7337e4878eb7f801ced677325890fdcc3b785e1442a4c3ce79: Status 404 returned error can't find the container with id f4b4fa90f940ab7337e4878eb7f801ced677325890fdcc3b785e1442a4c3ce79
Nov 03 17:39:40.055740 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:40.055711    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q"]
Nov 03 17:39:40.059868 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:40.059840    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e21c192-qsd9q"]
Nov 03 17:39:40.564534 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:40.564449    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:40.900082 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:40.899994    6059 generic.go:356] "Generic (PLEG): container finished" podID="924ff06e-596a-432d-962a-bb7cb7f64d7c" containerID="bf97440cc175998241950cd45c4d75688b66ba93e3bfc3e3c8bc880ddbbf566f" exitCode=0
Nov 03 17:39:40.900233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:40.900082    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3dec6e8-nppxc" event={"ID":"924ff06e-596a-432d-962a-bb7cb7f64d7c","Type":"ContainerDied","Data":"bf97440cc175998241950cd45c4d75688b66ba93e3bfc3e3c8bc880ddbbf566f"}
Nov 03 17:39:40.900233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:40.900134    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3dec6e8-nppxc" event={"ID":"924ff06e-596a-432d-962a-bb7cb7f64d7c","Type":"ContainerStarted","Data":"f4b4fa90f940ab7337e4878eb7f801ced677325890fdcc3b785e1442a4c3ce79"}
Nov 03 17:39:40.975539 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:40.975510    6059 scope.go:122] "RemoveContainer" containerID="371a38a396a387004193ebfe30fc564eb08975e2821b9d8f5a97fac34b154c84"
Nov 03 17:39:40.988621 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:40.988596    6059 scope.go:122] "RemoveContainer" containerID="473bb4c7d06c058400511be602750da812a217f1e8178c36bc9767a8e2903d73"
Nov 03 17:39:41.106818 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.106778    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"]
Nov 03 17:39:41.112373 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.112338    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.190431 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.184552    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-49c123d1-openshift-default-dockercfg-t6nmk\""
Nov 03 17:39:41.222110 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.218005    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-workload-socket\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.222110 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.218058    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-token\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.222110 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.218119    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-credential-socket\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.222110 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.218148    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-workload-certs\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.222110 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.218253    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h4v7\" (UniqueName: \"kubernetes.io/projected/7d3b16ed-f9c0-4c11-b7de-4f472c859170-kube-api-access-7h4v7\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.222110 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.218301    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-envoy\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.222110 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.218380    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-data\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.222110 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.218487    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-podinfo\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.222110 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.218554    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istiod-ca-cert\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.222787 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.222647    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"]
Nov 03 17:39:41.239121 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.239076    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:41.319694 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.319657    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-credential-socket\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.319863 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.319701    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-workload-certs\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.319863 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.319818    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h4v7\" (UniqueName: \"kubernetes.io/projected/7d3b16ed-f9c0-4c11-b7de-4f472c859170-kube-api-access-7h4v7\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.320001 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.319875    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-envoy\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.320001 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.319935    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-data\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.320001 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.319993    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-podinfo\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.320145 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.320053    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istiod-ca-cert\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.320145 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.320122    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-workload-socket\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.320255 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.320159    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-token\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.320255 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.320186    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-credential-socket\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.320833 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.320807    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-workload-socket\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.320995 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.320880    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-data\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.321085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.320941    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-workload-certs\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.321691 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.321665    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istiod-ca-cert\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.325082 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.324223    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-envoy\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.325429 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.325407    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-podinfo\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.352213 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.352162    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd" path="/var/lib/kubelet/pods/cc82f8bf-a4e0-44f2-bc75-a9e80ceadfbd/volumes"
Nov 03 17:39:41.384249 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.384212    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h4v7\" (UniqueName: \"kubernetes.io/projected/7d3b16ed-f9c0-4c11-b7de-4f472c859170-kube-api-access-7h4v7\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.385516 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.385487    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-token\") pod \"e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") " pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.426802 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.426752    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:41.546182 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.546150    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:41.851564 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:39:41.851512    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d3b16ed_f9c0_4c11_b7de_4f472c859170.slice/crio-a8ade10869a22e0d5cd2da540042b06cec5ad4284452f36eb4910b12cc552d3b WatchSource:0}: Error finding container a8ade10869a22e0d5cd2da540042b06cec5ad4284452f36eb4910b12cc552d3b: Status 404 returned error can't find the container with id a8ade10869a22e0d5cd2da540042b06cec5ad4284452f36eb4910b12cc552d3b
Nov 03 17:39:41.854311 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.854274    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:39:41.854445 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.854347    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:39:41.854445 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.854396    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:39:41.856570 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.856545    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"]
Nov 03 17:39:41.911123 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:41.911093    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7" event={"ID":"7d3b16ed-f9c0-4c11-b7de-4f472c859170","Type":"ContainerStarted","Data":"a8ade10869a22e0d5cd2da540042b06cec5ad4284452f36eb4910b12cc552d3b"}
Nov 03 17:39:42.071463 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:42.071435    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3dec6e8-nppxc"
Nov 03 17:39:42.130872 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:42.130783    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924ff06e-596a-432d-962a-bb7cb7f64d7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924ff06e-596a-432d-962a-bb7cb7f64d7c-maas-api-openshift-ca\") pod \"924ff06e-596a-432d-962a-bb7cb7f64d7c\" (UID: \"924ff06e-596a-432d-962a-bb7cb7f64d7c\") "
Nov 03 17:39:42.130872 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:42.130851    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/924ff06e-596a-432d-962a-bb7cb7f64d7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924ff06e-596a-432d-962a-bb7cb7f64d7c-maas-api-xks-ca\") pod \"924ff06e-596a-432d-962a-bb7cb7f64d7c\" (UID: \"924ff06e-596a-432d-962a-bb7cb7f64d7c\") "
Nov 03 17:39:42.131119 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:42.130965    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924ff06e-596a-432d-962a-bb7cb7f64d7c-kube-api-access-gq5bd\" (UniqueName: \"kubernetes.io/projected/924ff06e-596a-432d-962a-bb7cb7f64d7c-kube-api-access-gq5bd\") pod \"924ff06e-596a-432d-962a-bb7cb7f64d7c\" (UID: \"924ff06e-596a-432d-962a-bb7cb7f64d7c\") "
Nov 03 17:39:42.131202 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:42.131167    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924ff06e-596a-432d-962a-bb7cb7f64d7c-maas-api-openshift-ca" pod "924ff06e-596a-432d-962a-bb7cb7f64d7c" (UID: "924ff06e-596a-432d-962a-bb7cb7f64d7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:39:42.131448 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:42.131424    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924ff06e-596a-432d-962a-bb7cb7f64d7c-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:39:42.132961 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:42.132933    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/924ff06e-596a-432d-962a-bb7cb7f64d7c-maas-api-xks-ca" pod "924ff06e-596a-432d-962a-bb7cb7f64d7c" (UID: "924ff06e-596a-432d-962a-bb7cb7f64d7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:39:42.133084 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:42.132992    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924ff06e-596a-432d-962a-bb7cb7f64d7c-kube-api-access-gq5bd" pod "924ff06e-596a-432d-962a-bb7cb7f64d7c" (UID: "924ff06e-596a-432d-962a-bb7cb7f64d7c"). InnerVolumeSpecName "kube-api-access-gq5bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:39:42.232888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:42.232850    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq5bd\" (UniqueName: \"kubernetes.io/projected/924ff06e-596a-432d-962a-bb7cb7f64d7c-kube-api-access-gq5bd\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:39:42.232888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:42.232885    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924ff06e-596a-432d-962a-bb7cb7f64d7c-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:39:42.920426 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:42.920386    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7" event={"ID":"7d3b16ed-f9c0-4c11-b7de-4f472c859170","Type":"ContainerStarted","Data":"34f240a2a5be507ab00360ed3768aa60e74a382a394fe9b62f5dcc0638588a85"}
Nov 03 17:39:42.921931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:42.921907    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3dec6e8-nppxc"
Nov 03 17:39:42.922052 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:42.921939    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3dec6e8-nppxc" event={"ID":"924ff06e-596a-432d-962a-bb7cb7f64d7c","Type":"ContainerDied","Data":"f4b4fa90f940ab7337e4878eb7f801ced677325890fdcc3b785e1442a4c3ce79"}
Nov 03 17:39:42.922052 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:42.921983    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4b4fa90f940ab7337e4878eb7f801ced677325890fdcc3b785e1442a4c3ce79"
Nov 03 17:39:43.098135 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:43.098082    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7" podStartSLOduration=2.098068668 podStartE2EDuration="2.098068668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:39:43.096991927 +0000 UTC m=+1323.894704709" watchObservedRunningTime="2026-11-03 17:39:43.098068668 +0000 UTC m=+1323.895781448"
Nov 03 17:39:43.427285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:43.427246    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:43.433080 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:43.433055    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:43.930088 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:43.930062    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:43.930805 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:43.930787    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:39:44.620030 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:44.619979    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:45.087326 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:45.087294    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2db4500-q695g"]
Nov 03 17:39:45.088107 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:45.088083    6059 cpu_manager.go:414] "containerMap: removing container" podUID="924ff06e-596a-432d-962a-bb7cb7f64d7c" containerName="revoke-keys"
Nov 03 17:39:45.088206 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:45.088108    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924ff06e-596a-432d-962a-bb7cb7f64d7c" containerName="revoke-keys"
Nov 03 17:39:45.088299 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:45.088285    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="924ff06e-596a-432d-962a-bb7cb7f64d7c" containerName="revoke-keys"
Nov 03 17:39:45.093742 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:45.093715    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2db4500-q695g"
Nov 03 17:39:45.152435 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:45.152399    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2db4500-q695g"]
Nov 03 17:39:45.167519 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:45.167492    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7630c9bd-ea2d-45ed-9440-d53e539935d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2db4500-q695g\" (UID: \"7630c9bd-ea2d-45ed-9440-d53e539935d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2db4500-q695g"
Nov 03 17:39:45.167683 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:45.167549    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7630c9bd-ea2d-45ed-9440-d53e539935d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2db4500-q695g\" (UID: \"7630c9bd-ea2d-45ed-9440-d53e539935d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2db4500-q695g"
Nov 03 17:39:45.167769 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:45.167744    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rq65\" (UniqueName: \"kubernetes.io/projected/7630c9bd-ea2d-45ed-9440-d53e539935d3-kube-api-access-5rq65\") pod \"maas-api-revoke-sub-d2db4500-q695g\" (UID: \"7630c9bd-ea2d-45ed-9440-d53e539935d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2db4500-q695g"
Nov 03 17:39:45.198335 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:45.198304    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:39:45.198335 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:45.198323    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:39:45.198335 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:45.198320    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:39:45.200198 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:45.200173    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:39:45.269197 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:45.269168    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rq65\" (UniqueName: \"kubernetes.io/projected/7630c9bd-ea2d-45ed-9440-d53e539935d3-kube-api-access-5rq65\") pod \"maas-api-revoke-sub-d2db4500-q695g\" (UID: \"7630c9bd-ea2d-45ed-9440-d53e539935d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2db4500-q695g"
Nov 03 17:39:45.269397 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:45.269272    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7630c9bd-ea2d-45ed-9440-d53e539935d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2db4500-q695g\" (UID: \"7630c9bd-ea2d-45ed-9440-d53e539935d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2db4500-q695g"
Nov 03 17:39:45.269397 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:45.269314    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7630c9bd-ea2d-45ed-9440-d53e539935d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2db4500-q695g\" (UID: \"7630c9bd-ea2d-45ed-9440-d53e539935d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2db4500-q695g"
Nov 03 17:39:45.269915 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:45.269895    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7630c9bd-ea2d-45ed-9440-d53e539935d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2db4500-q695g\" (UID: \"7630c9bd-ea2d-45ed-9440-d53e539935d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2db4500-q695g"
Nov 03 17:39:45.271751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:45.271728    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7630c9bd-ea2d-45ed-9440-d53e539935d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2db4500-q695g\" (UID: \"7630c9bd-ea2d-45ed-9440-d53e539935d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2db4500-q695g"
Nov 03 17:39:45.332886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:45.332858    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rq65\" (UniqueName: \"kubernetes.io/projected/7630c9bd-ea2d-45ed-9440-d53e539935d3-kube-api-access-5rq65\") pod \"maas-api-revoke-sub-d2db4500-q695g\" (UID: \"7630c9bd-ea2d-45ed-9440-d53e539935d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2db4500-q695g"
Nov 03 17:39:45.411428 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:45.411339    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2db4500-q695g"
Nov 03 17:39:45.787221 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:39:45.787180    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7630c9bd_ea2d_45ed_9440_d53e539935d3.slice/crio-451c51cd88eeeda01b5eb6cefcbb348d9dccbdf2fa038e198a4cffee86f0f620 WatchSource:0}: Error finding container 451c51cd88eeeda01b5eb6cefcbb348d9dccbdf2fa038e198a4cffee86f0f620: Status 404 returned error can't find the container with id 451c51cd88eeeda01b5eb6cefcbb348d9dccbdf2fa038e198a4cffee86f0f620
Nov 03 17:39:45.790716 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:45.790690    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2db4500-q695g"]
Nov 03 17:39:45.946679 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:45.946642    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2db4500-q695g" event={"ID":"7630c9bd-ea2d-45ed-9440-d53e539935d3","Type":"ContainerStarted","Data":"451c51cd88eeeda01b5eb6cefcbb348d9dccbdf2fa038e198a4cffee86f0f620"}
Nov 03 17:39:46.061780 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:46.061756    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-312bd564-tj8mt"]
Nov 03 17:39:46.065376 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:46.065336    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-312bd564-tj8mt"]
Nov 03 17:39:46.067251 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:46.067232    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:46.313834 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:46.313747    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:46.956131 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:46.956036    6059 generic.go:356] "Generic (PLEG): container finished" podID="7630c9bd-ea2d-45ed-9440-d53e539935d3" containerID="943b3c1e3c7184b62a47d27ce650dbee4bc1e04555fc90a67d8c03c4f88d09aa" exitCode=0
Nov 03 17:39:46.956131 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:46.956115    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2db4500-q695g" event={"ID":"7630c9bd-ea2d-45ed-9440-d53e539935d3","Type":"ContainerDied","Data":"943b3c1e3c7184b62a47d27ce650dbee4bc1e04555fc90a67d8c03c4f88d09aa"}
Nov 03 17:39:47.337107 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:47.337028    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d822d650-6b28-4159-a7e8-95cf97f75d46" path="/var/lib/kubelet/pods/d822d650-6b28-4159-a7e8-95cf97f75d46/volumes"
Nov 03 17:39:48.114864 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:48.114835    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2db4500-q695g"
Nov 03 17:39:48.207994 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:48.207958    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7630c9bd-ea2d-45ed-9440-d53e539935d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7630c9bd-ea2d-45ed-9440-d53e539935d3-maas-api-xks-ca\") pod \"7630c9bd-ea2d-45ed-9440-d53e539935d3\" (UID: \"7630c9bd-ea2d-45ed-9440-d53e539935d3\") "
Nov 03 17:39:48.208147 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:48.208062    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7630c9bd-ea2d-45ed-9440-d53e539935d3-kube-api-access-5rq65\" (UniqueName: \"kubernetes.io/projected/7630c9bd-ea2d-45ed-9440-d53e539935d3-kube-api-access-5rq65\") pod \"7630c9bd-ea2d-45ed-9440-d53e539935d3\" (UID: \"7630c9bd-ea2d-45ed-9440-d53e539935d3\") "
Nov 03 17:39:48.208220 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:48.208205    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7630c9bd-ea2d-45ed-9440-d53e539935d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7630c9bd-ea2d-45ed-9440-d53e539935d3-maas-api-openshift-ca\") pod \"7630c9bd-ea2d-45ed-9440-d53e539935d3\" (UID: \"7630c9bd-ea2d-45ed-9440-d53e539935d3\") "
Nov 03 17:39:48.208653 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:48.208626    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7630c9bd-ea2d-45ed-9440-d53e539935d3-maas-api-openshift-ca" pod "7630c9bd-ea2d-45ed-9440-d53e539935d3" (UID: "7630c9bd-ea2d-45ed-9440-d53e539935d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:39:48.208787 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:48.208762    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7630c9bd-ea2d-45ed-9440-d53e539935d3-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:39:48.210074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:48.210044    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7630c9bd-ea2d-45ed-9440-d53e539935d3-maas-api-xks-ca" pod "7630c9bd-ea2d-45ed-9440-d53e539935d3" (UID: "7630c9bd-ea2d-45ed-9440-d53e539935d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:39:48.210199 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:48.210178    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7630c9bd-ea2d-45ed-9440-d53e539935d3-kube-api-access-5rq65" pod "7630c9bd-ea2d-45ed-9440-d53e539935d3" (UID: "7630c9bd-ea2d-45ed-9440-d53e539935d3"). InnerVolumeSpecName "kube-api-access-5rq65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:39:48.310242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:48.310208    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7630c9bd-ea2d-45ed-9440-d53e539935d3-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:39:48.310242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:48.310239    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rq65\" (UniqueName: \"kubernetes.io/projected/7630c9bd-ea2d-45ed-9440-d53e539935d3-kube-api-access-5rq65\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:39:49.001599 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:48.998154    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2db4500-q695g" event={"ID":"7630c9bd-ea2d-45ed-9440-d53e539935d3","Type":"ContainerDied","Data":"451c51cd88eeeda01b5eb6cefcbb348d9dccbdf2fa038e198a4cffee86f0f620"}
Nov 03 17:39:49.001599 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:48.998253    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="451c51cd88eeeda01b5eb6cefcbb348d9dccbdf2fa038e198a4cffee86f0f620"
Nov 03 17:39:49.001599 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:48.998321    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2db4500-q695g"
Nov 03 17:39:49.169880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:49.169848    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:49.360247 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:49.360167    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:49.543880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:49.543845    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:49.743378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:49.742647    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk"]
Nov 03 17:39:49.764456 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:49.743682    6059 cpu_manager.go:414] "containerMap: removing container" podUID="7630c9bd-ea2d-45ed-9440-d53e539935d3" containerName="revoke-keys"
Nov 03 17:39:49.764456 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:49.743709    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7630c9bd-ea2d-45ed-9440-d53e539935d3" containerName="revoke-keys"
Nov 03 17:39:49.764456 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:49.743915    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7630c9bd-ea2d-45ed-9440-d53e539935d3" containerName="revoke-keys"
Nov 03 17:39:49.782121 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:49.782090    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk"]
Nov 03 17:39:49.782480 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:49.782457    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk"
Nov 03 17:39:49.837679 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:49.837634    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-49c123d1\""
Nov 03 17:39:49.889350 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:49.889317    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-49c123d1-dockercfg-8c6mw\""
Nov 03 17:39:49.938830 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:49.938798    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7-plugins-config-volume\") pod \"payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk\" (UID: \"f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7\") " pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk"
Nov 03 17:39:49.938986 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:49.938865    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfxpw\" (UniqueName: \"kubernetes.io/projected/f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7-kube-api-access-qfxpw\") pod \"payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk\" (UID: \"f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7\") " pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk"
Nov 03 17:39:50.040575 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:50.040458    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7-plugins-config-volume\") pod \"payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk\" (UID: \"f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7\") " pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk"
Nov 03 17:39:50.040575 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:50.040554    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfxpw\" (UniqueName: \"kubernetes.io/projected/f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7-kube-api-access-qfxpw\") pod \"payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk\" (UID: \"f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7\") " pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk"
Nov 03 17:39:50.041410 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:50.041349    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7-plugins-config-volume\") pod \"payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk\" (UID: \"f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7\") " pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk"
Nov 03 17:39:50.112334 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:50.112297    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfxpw\" (UniqueName: \"kubernetes.io/projected/f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7-kube-api-access-qfxpw\") pod \"payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk\" (UID: \"f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7\") " pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk"
Nov 03 17:39:50.405521 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:50.405049    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk"
Nov 03 17:39:51.460319 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:51.460288    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk"]
Nov 03 17:39:52.189845 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:52.189801    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk" event={"ID":"f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7","Type":"ContainerStarted","Data":"6f60ddef030c81cfcf5e5981e7e23938180e225fe28aab91b3fff83b6c286e3b"}
Nov 03 17:39:52.189995 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:52.189888    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk" event={"ID":"f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7","Type":"ContainerStarted","Data":"5aca8d0936308d0e15f2eae16cd42c25723de8c79256c5c4139bb07650a7e558"}
Nov 03 17:39:52.189995 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:52.189916    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk"
Nov 03 17:39:52.321506 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:52.321457    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk" podStartSLOduration=3.321443255 podStartE2EDuration="3.321443255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:39:52.318580131 +0000 UTC m=+1333.116292912" watchObservedRunningTime="2026-11-03 17:39:52.321443255 +0000 UTC m=+1333.119156036"
Nov 03 17:39:53.063178 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:53.063142    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:53.857591 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:53.857552    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm"]
Nov 03 17:39:53.861997 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:53.861974    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm"
Nov 03 17:39:53.908547 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:53.908507    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65d49cb-7dea-46dd-b8fa-6d785939fb1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-695b8f95-cs2cm\" (UID: \"d65d49cb-7dea-46dd-b8fa-6d785939fb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm"
Nov 03 17:39:53.908718 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:53.908569    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65d49cb-7dea-46dd-b8fa-6d785939fb1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-695b8f95-cs2cm\" (UID: \"d65d49cb-7dea-46dd-b8fa-6d785939fb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm"
Nov 03 17:39:53.908775 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:53.908718    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwf9c\" (UniqueName: \"kubernetes.io/projected/d65d49cb-7dea-46dd-b8fa-6d785939fb1e-kube-api-access-xwf9c\") pod \"maas-api-revoke-sub-695b8f95-cs2cm\" (UID: \"d65d49cb-7dea-46dd-b8fa-6d785939fb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm"
Nov 03 17:39:53.915644 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:53.915603    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm"]
Nov 03 17:39:53.961669 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:53.961636    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:39:53.961830 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:53.961676    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:39:53.961830 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:53.961686    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:39:53.967549 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:53.967524    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:39:54.010131 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:54.010078    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65d49cb-7dea-46dd-b8fa-6d785939fb1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-695b8f95-cs2cm\" (UID: \"d65d49cb-7dea-46dd-b8fa-6d785939fb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm"
Nov 03 17:39:54.010131 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:54.010138    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65d49cb-7dea-46dd-b8fa-6d785939fb1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-695b8f95-cs2cm\" (UID: \"d65d49cb-7dea-46dd-b8fa-6d785939fb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm"
Nov 03 17:39:54.010429 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:54.010174    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwf9c\" (UniqueName: \"kubernetes.io/projected/d65d49cb-7dea-46dd-b8fa-6d785939fb1e-kube-api-access-xwf9c\") pod \"maas-api-revoke-sub-695b8f95-cs2cm\" (UID: \"d65d49cb-7dea-46dd-b8fa-6d785939fb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm"
Nov 03 17:39:54.010818 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:54.010798    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65d49cb-7dea-46dd-b8fa-6d785939fb1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-695b8f95-cs2cm\" (UID: \"d65d49cb-7dea-46dd-b8fa-6d785939fb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm"
Nov 03 17:39:54.012612 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:54.012594    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65d49cb-7dea-46dd-b8fa-6d785939fb1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-695b8f95-cs2cm\" (UID: \"d65d49cb-7dea-46dd-b8fa-6d785939fb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm"
Nov 03 17:39:54.074576 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:54.074539    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwf9c\" (UniqueName: \"kubernetes.io/projected/d65d49cb-7dea-46dd-b8fa-6d785939fb1e-kube-api-access-xwf9c\") pod \"maas-api-revoke-sub-695b8f95-cs2cm\" (UID: \"d65d49cb-7dea-46dd-b8fa-6d785939fb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm"
Nov 03 17:39:54.176390 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:54.176284    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm"
Nov 03 17:39:54.358389 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:54.358014    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:39:54.577225 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:39:54.577181    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65d49cb_7dea_46dd_b8fa_6d785939fb1e.slice/crio-b65db1a2b8a25f91345e747e44b93a8dd5bb219d4faa127abe74eac21fb2adcb WatchSource:0}: Error finding container b65db1a2b8a25f91345e747e44b93a8dd5bb219d4faa127abe74eac21fb2adcb: Status 404 returned error can't find the container with id b65db1a2b8a25f91345e747e44b93a8dd5bb219d4faa127abe74eac21fb2adcb
Nov 03 17:39:54.581045 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:54.581020    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm"]
Nov 03 17:39:55.217478 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:55.217444    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm" event={"ID":"d65d49cb-7dea-46dd-b8fa-6d785939fb1e","Type":"ContainerStarted","Data":"4adc512d67bf6febbf79294018a2064dbb713f930f6c70a264fa7143b02faf4e"}
Nov 03 17:39:55.217890 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:55.217501    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm" event={"ID":"d65d49cb-7dea-46dd-b8fa-6d785939fb1e","Type":"ContainerStarted","Data":"b65db1a2b8a25f91345e747e44b93a8dd5bb219d4faa127abe74eac21fb2adcb"}
Nov 03 17:39:55.359491 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:55.354754    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm" podStartSLOduration=2.354732414 podStartE2EDuration="2.354732414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:39:55.343530442 +0000 UTC m=+1336.141243224" watchObservedRunningTime="2026-11-03 17:39:55.354732414 +0000 UTC m=+1336.152445195"
Nov 03 17:39:56.227379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:56.227320    6059 generic.go:356] "Generic (PLEG): container finished" podID="d65d49cb-7dea-46dd-b8fa-6d785939fb1e" containerID="4adc512d67bf6febbf79294018a2064dbb713f930f6c70a264fa7143b02faf4e" exitCode=0
Nov 03 17:39:56.227875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:56.227397    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm" event={"ID":"d65d49cb-7dea-46dd-b8fa-6d785939fb1e","Type":"ContainerDied","Data":"4adc512d67bf6febbf79294018a2064dbb713f930f6c70a264fa7143b02faf4e"}
Nov 03 17:39:57.403286 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:57.403252    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm"
Nov 03 17:39:57.577342 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:57.577251    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d65d49cb-7dea-46dd-b8fa-6d785939fb1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65d49cb-7dea-46dd-b8fa-6d785939fb1e-maas-api-xks-ca\") pod \"d65d49cb-7dea-46dd-b8fa-6d785939fb1e\" (UID: \"d65d49cb-7dea-46dd-b8fa-6d785939fb1e\") "
Nov 03 17:39:57.577342 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:57.577317    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d65d49cb-7dea-46dd-b8fa-6d785939fb1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65d49cb-7dea-46dd-b8fa-6d785939fb1e-maas-api-openshift-ca\") pod \"d65d49cb-7dea-46dd-b8fa-6d785939fb1e\" (UID: \"d65d49cb-7dea-46dd-b8fa-6d785939fb1e\") "
Nov 03 17:39:57.577619 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:57.577385    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d65d49cb-7dea-46dd-b8fa-6d785939fb1e-kube-api-access-xwf9c\" (UniqueName: \"kubernetes.io/projected/d65d49cb-7dea-46dd-b8fa-6d785939fb1e-kube-api-access-xwf9c\") pod \"d65d49cb-7dea-46dd-b8fa-6d785939fb1e\" (UID: \"d65d49cb-7dea-46dd-b8fa-6d785939fb1e\") "
Nov 03 17:39:57.577785 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:57.577758    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d65d49cb-7dea-46dd-b8fa-6d785939fb1e-maas-api-openshift-ca" pod "d65d49cb-7dea-46dd-b8fa-6d785939fb1e" (UID: "d65d49cb-7dea-46dd-b8fa-6d785939fb1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:39:57.577965 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:57.577939    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65d49cb-7dea-46dd-b8fa-6d785939fb1e-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:39:57.579954 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:57.579927    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d65d49cb-7dea-46dd-b8fa-6d785939fb1e-kube-api-access-xwf9c" pod "d65d49cb-7dea-46dd-b8fa-6d785939fb1e" (UID: "d65d49cb-7dea-46dd-b8fa-6d785939fb1e"). InnerVolumeSpecName "kube-api-access-xwf9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:39:57.580186 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:57.580150    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d65d49cb-7dea-46dd-b8fa-6d785939fb1e-maas-api-xks-ca" pod "d65d49cb-7dea-46dd-b8fa-6d785939fb1e" (UID: "d65d49cb-7dea-46dd-b8fa-6d785939fb1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:39:57.679673 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:57.679628    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65d49cb-7dea-46dd-b8fa-6d785939fb1e-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:39:57.679673 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:57.679667    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwf9c\" (UniqueName: \"kubernetes.io/projected/d65d49cb-7dea-46dd-b8fa-6d785939fb1e-kube-api-access-xwf9c\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:39:58.245732 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:58.245699    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm"
Nov 03 17:39:58.245732 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:58.245721    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm" event={"ID":"d65d49cb-7dea-46dd-b8fa-6d785939fb1e","Type":"ContainerDied","Data":"b65db1a2b8a25f91345e747e44b93a8dd5bb219d4faa127abe74eac21fb2adcb"}
Nov 03 17:39:58.245984 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:58.245767    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b65db1a2b8a25f91345e747e44b93a8dd5bb219d4faa127abe74eac21fb2adcb"
Nov 03 17:39:58.821080 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:39:58.821039    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:00.264161 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:00.264127    6059 generic.go:356] "Generic (PLEG): container finished" podID="3e53c74d-ceff-4b92-b579-35ef2b76de83" containerID="c29ac6f3f77f4fdf3a59636f7804eac7311724a5feb38d17d8b8248cc0e12fb6" exitCode=0
Nov 03 17:40:00.264649 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:00.264196    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq" event={"ID":"3e53c74d-ceff-4b92-b579-35ef2b76de83","Type":"ContainerDied","Data":"c29ac6f3f77f4fdf3a59636f7804eac7311724a5feb38d17d8b8248cc0e12fb6"}
Nov 03 17:40:00.264649 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:00.264575    6059 scope.go:122] "RemoveContainer" containerID="c29ac6f3f77f4fdf3a59636f7804eac7311724a5feb38d17d8b8248cc0e12fb6"
Nov 03 17:40:00.265902 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:00.265880    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:40:00.265964 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:00.265939    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:40:00.266056 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:00.265970    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:40:01.274107 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:01.274068    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq" event={"ID":"3e53c74d-ceff-4b92-b579-35ef2b76de83","Type":"ContainerStarted","Data":"2e86ef0087015f81d2b1179ff4297ab04a7b43b4625f6d2a4e0d5915c673c901"}
Nov 03 17:40:01.498164 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:01.498128    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:40:01.499320 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:01.499287    6059 patch_prober.go:29] interesting pod/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 17:40:01.499456 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:01.499339    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq" podUID="3e53c74d-ceff-4b92-b579-35ef2b76de83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 17:40:02.498468 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:02.498434    6059 patch_prober.go:29] interesting pod/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 17:40:02.498842 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:02.498498    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq" podUID="3e53c74d-ceff-4b92-b579-35ef2b76de83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 17:40:03.200411 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:03.200382    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk"
Nov 03 17:40:03.460452 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:03.457045    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:03.498086 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:03.498049    6059 patch_prober.go:29] interesting pod/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 17:40:03.498291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:03.498112    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq" podUID="3e53c74d-ceff-4b92-b579-35ef2b76de83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 17:40:04.063920 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.063889    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m"]
Nov 03 17:40:04.067549 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.067519    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec3560d3-hdc2m"]
Nov 03 17:40:04.497653 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.497624    6059 patch_prober.go:29] interesting pod/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 17:40:04.497837 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.497672    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq" podUID="3e53c74d-ceff-4b92-b579-35ef2b76de83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 17:40:04.560334 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.560295    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7"]
Nov 03 17:40:04.560883 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.560869    6059 cpu_manager.go:414] "containerMap: removing container" podUID="d65d49cb-7dea-46dd-b8fa-6d785939fb1e" containerName="revoke-keys"
Nov 03 17:40:04.560934 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.560885    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d65d49cb-7dea-46dd-b8fa-6d785939fb1e" containerName="revoke-keys"
Nov 03 17:40:04.561016 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.561007    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d65d49cb-7dea-46dd-b8fa-6d785939fb1e" containerName="revoke-keys"
Nov 03 17:40:04.564540 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.564524    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7"
Nov 03 17:40:04.583298 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.583273    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9031a661-8d7b-4a6a-ae19-0a9b1a594b99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95f94805-pv2h7\" (UID: \"9031a661-8d7b-4a6a-ae19-0a9b1a594b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7"
Nov 03 17:40:04.583514 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.583488    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9031a661-8d7b-4a6a-ae19-0a9b1a594b99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95f94805-pv2h7\" (UID: \"9031a661-8d7b-4a6a-ae19-0a9b1a594b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7"
Nov 03 17:40:04.583584 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.583534    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zvmq\" (UniqueName: \"kubernetes.io/projected/9031a661-8d7b-4a6a-ae19-0a9b1a594b99-kube-api-access-4zvmq\") pod \"maas-api-revoke-sub-95f94805-pv2h7\" (UID: \"9031a661-8d7b-4a6a-ae19-0a9b1a594b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7"
Nov 03 17:40:04.661628 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.661588    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7"]
Nov 03 17:40:04.673842 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.673802    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:40:04.673842 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.673832    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:40:04.674078 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.673841    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:40:04.674078 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.673963    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:40:04.684462 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.684439    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9031a661-8d7b-4a6a-ae19-0a9b1a594b99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95f94805-pv2h7\" (UID: \"9031a661-8d7b-4a6a-ae19-0a9b1a594b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7"
Nov 03 17:40:04.684611 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.684601    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9031a661-8d7b-4a6a-ae19-0a9b1a594b99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95f94805-pv2h7\" (UID: \"9031a661-8d7b-4a6a-ae19-0a9b1a594b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7"
Nov 03 17:40:04.684681 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.684631    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zvmq\" (UniqueName: \"kubernetes.io/projected/9031a661-8d7b-4a6a-ae19-0a9b1a594b99-kube-api-access-4zvmq\") pod \"maas-api-revoke-sub-95f94805-pv2h7\" (UID: \"9031a661-8d7b-4a6a-ae19-0a9b1a594b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7"
Nov 03 17:40:04.685184 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.685164    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9031a661-8d7b-4a6a-ae19-0a9b1a594b99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95f94805-pv2h7\" (UID: \"9031a661-8d7b-4a6a-ae19-0a9b1a594b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7"
Nov 03 17:40:04.686856 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.686833    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9031a661-8d7b-4a6a-ae19-0a9b1a594b99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95f94805-pv2h7\" (UID: \"9031a661-8d7b-4a6a-ae19-0a9b1a594b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7"
Nov 03 17:40:04.757863 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.757789    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zvmq\" (UniqueName: \"kubernetes.io/projected/9031a661-8d7b-4a6a-ae19-0a9b1a594b99-kube-api-access-4zvmq\") pod \"maas-api-revoke-sub-95f94805-pv2h7\" (UID: \"9031a661-8d7b-4a6a-ae19-0a9b1a594b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7"
Nov 03 17:40:04.877814 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:04.877785    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7"
Nov 03 17:40:05.054669 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:05.052453    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:40:05.273666 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:40:05.273632    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9031a661_8d7b_4a6a_ae19_0a9b1a594b99.slice/crio-00a8ec45d07604d5f0971dda5c8fd6d84f661ed9968e3a62abf175e95025ffbb WatchSource:0}: Error finding container 00a8ec45d07604d5f0971dda5c8fd6d84f661ed9968e3a62abf175e95025ffbb: Status 404 returned error can't find the container with id 00a8ec45d07604d5f0971dda5c8fd6d84f661ed9968e3a62abf175e95025ffbb
Nov 03 17:40:05.277788 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:05.277764    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7"]
Nov 03 17:40:05.305890 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:05.305837    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7" event={"ID":"9031a661-8d7b-4a6a-ae19-0a9b1a594b99","Type":"ContainerStarted","Data":"00a8ec45d07604d5f0971dda5c8fd6d84f661ed9968e3a62abf175e95025ffbb"}
Nov 03 17:40:05.360941 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:05.360901    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c41881f-5e83-4546-b412-8923b7a50af5" path="/var/lib/kubelet/pods/3c41881f-5e83-4546-b412-8923b7a50af5/volumes"
Nov 03 17:40:05.497963 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:05.497933    6059 patch_prober.go:29] interesting pod/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 17:40:05.498112 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:05.497980    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq" podUID="3e53c74d-ceff-4b92-b579-35ef2b76de83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 17:40:06.316367 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:06.316315    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7" event={"ID":"9031a661-8d7b-4a6a-ae19-0a9b1a594b99","Type":"ContainerStarted","Data":"ec904366f6c9e5935da62d7ef27d75ebffc271c46727287f706909ca1bd1430b"}
Nov 03 17:40:06.501414 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:06.501382    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:40:06.502593 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:06.502572    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:40:06.630583 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:06.630480    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7" podStartSLOduration=2.630465334 podStartE2EDuration="2.630465334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:06.443857896 +0000 UTC m=+1347.241570679" watchObservedRunningTime="2026-11-03 17:40:06.630465334 +0000 UTC m=+1347.428178163"
Nov 03 17:40:06.861323 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:40:06.861289    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:40:07.333710 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:07.332593    6059 generic.go:356] "Generic (PLEG): container finished" podID="9031a661-8d7b-4a6a-ae19-0a9b1a594b99" containerID="ec904366f6c9e5935da62d7ef27d75ebffc271c46727287f706909ca1bd1430b" exitCode=0
Nov 03 17:40:07.340251 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:07.340220    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7" event={"ID":"9031a661-8d7b-4a6a-ae19-0a9b1a594b99","Type":"ContainerDied","Data":"ec904366f6c9e5935da62d7ef27d75ebffc271c46727287f706909ca1bd1430b"}
Nov 03 17:40:08.030256 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:08.030214    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:08.485826 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:08.485798    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7"
Nov 03 17:40:08.535466 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:08.535430    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9031a661-8d7b-4a6a-ae19-0a9b1a594b99-kube-api-access-4zvmq\" (UniqueName: \"kubernetes.io/projected/9031a661-8d7b-4a6a-ae19-0a9b1a594b99-kube-api-access-4zvmq\") pod \"9031a661-8d7b-4a6a-ae19-0a9b1a594b99\" (UID: \"9031a661-8d7b-4a6a-ae19-0a9b1a594b99\") "
Nov 03 17:40:08.535649 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:08.535477    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9031a661-8d7b-4a6a-ae19-0a9b1a594b99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9031a661-8d7b-4a6a-ae19-0a9b1a594b99-maas-api-openshift-ca\") pod \"9031a661-8d7b-4a6a-ae19-0a9b1a594b99\" (UID: \"9031a661-8d7b-4a6a-ae19-0a9b1a594b99\") "
Nov 03 17:40:08.535649 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:08.535598    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9031a661-8d7b-4a6a-ae19-0a9b1a594b99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9031a661-8d7b-4a6a-ae19-0a9b1a594b99-maas-api-xks-ca\") pod \"9031a661-8d7b-4a6a-ae19-0a9b1a594b99\" (UID: \"9031a661-8d7b-4a6a-ae19-0a9b1a594b99\") "
Nov 03 17:40:08.535878 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:08.535844    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9031a661-8d7b-4a6a-ae19-0a9b1a594b99-maas-api-openshift-ca" pod "9031a661-8d7b-4a6a-ae19-0a9b1a594b99" (UID: "9031a661-8d7b-4a6a-ae19-0a9b1a594b99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:40:08.536190 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:08.536165    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9031a661-8d7b-4a6a-ae19-0a9b1a594b99-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:08.537684 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:08.537660    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9031a661-8d7b-4a6a-ae19-0a9b1a594b99-kube-api-access-4zvmq" pod "9031a661-8d7b-4a6a-ae19-0a9b1a594b99" (UID: "9031a661-8d7b-4a6a-ae19-0a9b1a594b99"). InnerVolumeSpecName "kube-api-access-4zvmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:40:08.537684 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:08.537676    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9031a661-8d7b-4a6a-ae19-0a9b1a594b99-maas-api-xks-ca" pod "9031a661-8d7b-4a6a-ae19-0a9b1a594b99" (UID: "9031a661-8d7b-4a6a-ae19-0a9b1a594b99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:40:08.638079 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:08.637995    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9031a661-8d7b-4a6a-ae19-0a9b1a594b99-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:08.638079 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:08.638027    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zvmq\" (UniqueName: \"kubernetes.io/projected/9031a661-8d7b-4a6a-ae19-0a9b1a594b99-kube-api-access-4zvmq\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:09.351509 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:09.351479    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7"
Nov 03 17:40:09.351509 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:09.351498    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7" event={"ID":"9031a661-8d7b-4a6a-ae19-0a9b1a594b99","Type":"ContainerDied","Data":"00a8ec45d07604d5f0971dda5c8fd6d84f661ed9968e3a62abf175e95025ffbb"}
Nov 03 17:40:09.351707 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:09.351548    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00a8ec45d07604d5f0971dda5c8fd6d84f661ed9968e3a62abf175e95025ffbb"
Nov 03 17:40:13.358372 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:13.358317    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ed607ea-n2477"]
Nov 03 17:40:13.360058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:13.360020    6059 cpu_manager.go:414] "containerMap: removing container" podUID="9031a661-8d7b-4a6a-ae19-0a9b1a594b99" containerName="revoke-keys"
Nov 03 17:40:13.360243 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:13.360227    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9031a661-8d7b-4a6a-ae19-0a9b1a594b99" containerName="revoke-keys"
Nov 03 17:40:13.360922 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:13.360905    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9031a661-8d7b-4a6a-ae19-0a9b1a594b99" containerName="revoke-keys"
Nov 03 17:40:13.612485 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:13.608538    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ed607ea-n2477"]
Nov 03 17:40:13.612485 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:13.608676    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ed607ea-n2477"
Nov 03 17:40:13.616638 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:13.616598    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:13.664382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:13.663852    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbbdb77-9224-4eff-9937-e060f53137f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1ed607ea-n2477\" (UID: \"2bbbdb77-9224-4eff-9937-e060f53137f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ed607ea-n2477"
Nov 03 17:40:13.664382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:13.663965    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmfpk\" (UniqueName: \"kubernetes.io/projected/2bbbdb77-9224-4eff-9937-e060f53137f5-kube-api-access-fmfpk\") pod \"maas-api-revoke-keys-e2e-ait-1ed607ea-n2477\" (UID: \"2bbbdb77-9224-4eff-9937-e060f53137f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ed607ea-n2477"
Nov 03 17:40:13.664382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:13.664067    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbbdb77-9224-4eff-9937-e060f53137f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1ed607ea-n2477\" (UID: \"2bbbdb77-9224-4eff-9937-e060f53137f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ed607ea-n2477"
Nov 03 17:40:13.685383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:13.665970    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:40:13.716124 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:13.716080    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:40:13.717143 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:13.717023    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:40:13.717546 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:13.717514    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:40:13.766378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:13.765569    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbbdb77-9224-4eff-9937-e060f53137f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1ed607ea-n2477\" (UID: \"2bbbdb77-9224-4eff-9937-e060f53137f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ed607ea-n2477"
Nov 03 17:40:13.766378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:13.765703    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmfpk\" (UniqueName: \"kubernetes.io/projected/2bbbdb77-9224-4eff-9937-e060f53137f5-kube-api-access-fmfpk\") pod \"maas-api-revoke-keys-e2e-ait-1ed607ea-n2477\" (UID: \"2bbbdb77-9224-4eff-9937-e060f53137f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ed607ea-n2477"
Nov 03 17:40:13.766378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:13.765817    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbbdb77-9224-4eff-9937-e060f53137f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1ed607ea-n2477\" (UID: \"2bbbdb77-9224-4eff-9937-e060f53137f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ed607ea-n2477"
Nov 03 17:40:13.766783 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:13.766763    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbbdb77-9224-4eff-9937-e060f53137f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1ed607ea-n2477\" (UID: \"2bbbdb77-9224-4eff-9937-e060f53137f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ed607ea-n2477"
Nov 03 17:40:13.769323 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:13.769300    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbbdb77-9224-4eff-9937-e060f53137f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1ed607ea-n2477\" (UID: \"2bbbdb77-9224-4eff-9937-e060f53137f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ed607ea-n2477"
Nov 03 17:40:13.840568 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:13.840538    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmfpk\" (UniqueName: \"kubernetes.io/projected/2bbbdb77-9224-4eff-9937-e060f53137f5-kube-api-access-fmfpk\") pod \"maas-api-revoke-keys-e2e-ait-1ed607ea-n2477\" (UID: \"2bbbdb77-9224-4eff-9937-e060f53137f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ed607ea-n2477"
Nov 03 17:40:13.927831 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:13.927724    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ed607ea-n2477"
Nov 03 17:40:14.055662 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:14.055620    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls"]
Nov 03 17:40:14.059468 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:14.059436    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e988894f-6qvls"]
Nov 03 17:40:14.305811 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:40:14.305774    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bbbdb77_9224_4eff_9937_e060f53137f5.slice/crio-588202d55fb1178d40499ab098acd7f80f6649d9d0cd1d8c6af3b012ef635749 WatchSource:0}: Error finding container 588202d55fb1178d40499ab098acd7f80f6649d9d0cd1d8c6af3b012ef635749: Status 404 returned error can't find the container with id 588202d55fb1178d40499ab098acd7f80f6649d9d0cd1d8c6af3b012ef635749
Nov 03 17:40:14.310348 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:14.310184    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ed607ea-n2477"]
Nov 03 17:40:14.584468 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:14.584386    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ed607ea-n2477" event={"ID":"2bbbdb77-9224-4eff-9937-e060f53137f5","Type":"ContainerStarted","Data":"588202d55fb1178d40499ab098acd7f80f6649d9d0cd1d8c6af3b012ef635749"}
Nov 03 17:40:15.337078 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:15.337044    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37b19cdf-69a5-4c58-84dc-7d08992c260a" path="/var/lib/kubelet/pods/37b19cdf-69a5-4c58-84dc-7d08992c260a/volumes"
Nov 03 17:40:15.593510 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:15.593474    6059 generic.go:356] "Generic (PLEG): container finished" podID="2bbbdb77-9224-4eff-9937-e060f53137f5" containerID="7f47e37d15811db9605c40b0125b7124fa19c34835ac94e5fa545d5263793c5b" exitCode=0
Nov 03 17:40:15.594134 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:15.593557    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ed607ea-n2477" event={"ID":"2bbbdb77-9224-4eff-9937-e060f53137f5","Type":"ContainerDied","Data":"7f47e37d15811db9605c40b0125b7124fa19c34835ac94e5fa545d5263793c5b"}
Nov 03 17:40:16.758805 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:16.758778    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ed607ea-n2477"
Nov 03 17:40:16.904120 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:16.904020    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bbbdb77-9224-4eff-9937-e060f53137f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbbdb77-9224-4eff-9937-e060f53137f5-maas-api-openshift-ca\") pod \"2bbbdb77-9224-4eff-9937-e060f53137f5\" (UID: \"2bbbdb77-9224-4eff-9937-e060f53137f5\") "
Nov 03 17:40:16.904345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:16.904320    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bbbdb77-9224-4eff-9937-e060f53137f5-kube-api-access-fmfpk\" (UniqueName: \"kubernetes.io/projected/2bbbdb77-9224-4eff-9937-e060f53137f5-kube-api-access-fmfpk\") pod \"2bbbdb77-9224-4eff-9937-e060f53137f5\" (UID: \"2bbbdb77-9224-4eff-9937-e060f53137f5\") "
Nov 03 17:40:16.904449 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:16.904407    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bbbdb77-9224-4eff-9937-e060f53137f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbbdb77-9224-4eff-9937-e060f53137f5-maas-api-xks-ca\") pod \"2bbbdb77-9224-4eff-9937-e060f53137f5\" (UID: \"2bbbdb77-9224-4eff-9937-e060f53137f5\") "
Nov 03 17:40:16.904520 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:16.904438    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bbbdb77-9224-4eff-9937-e060f53137f5-maas-api-openshift-ca" pod "2bbbdb77-9224-4eff-9937-e060f53137f5" (UID: "2bbbdb77-9224-4eff-9937-e060f53137f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:40:16.904906 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:16.904883    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbbdb77-9224-4eff-9937-e060f53137f5-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:16.906932 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:16.906903    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bbbdb77-9224-4eff-9937-e060f53137f5-kube-api-access-fmfpk" pod "2bbbdb77-9224-4eff-9937-e060f53137f5" (UID: "2bbbdb77-9224-4eff-9937-e060f53137f5"). InnerVolumeSpecName "kube-api-access-fmfpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:40:16.907046 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:16.906941    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bbbdb77-9224-4eff-9937-e060f53137f5-maas-api-xks-ca" pod "2bbbdb77-9224-4eff-9937-e060f53137f5" (UID: "2bbbdb77-9224-4eff-9937-e060f53137f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:40:17.006442 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:17.006404    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmfpk\" (UniqueName: \"kubernetes.io/projected/2bbbdb77-9224-4eff-9937-e060f53137f5-kube-api-access-fmfpk\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:17.006442 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:17.006439    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbbdb77-9224-4eff-9937-e060f53137f5-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:17.613751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:17.613657    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ed607ea-n2477"
Nov 03 17:40:17.613751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:17.613685    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ed607ea-n2477" event={"ID":"2bbbdb77-9224-4eff-9937-e060f53137f5","Type":"ContainerDied","Data":"588202d55fb1178d40499ab098acd7f80f6649d9d0cd1d8c6af3b012ef635749"}
Nov 03 17:40:17.613751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:17.613730    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="588202d55fb1178d40499ab098acd7f80f6649d9d0cd1d8c6af3b012ef635749"
Nov 03 17:40:18.907153 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:18.907117    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:23.123076 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.123037    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm"]
Nov 03 17:40:23.123713 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.123691    6059 cpu_manager.go:414] "containerMap: removing container" podUID="2bbbdb77-9224-4eff-9937-e060f53137f5" containerName="revoke-keys"
Nov 03 17:40:23.123781 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.123714    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bbbdb77-9224-4eff-9937-e060f53137f5" containerName="revoke-keys"
Nov 03 17:40:23.123921 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.123908    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bbbdb77-9224-4eff-9937-e060f53137f5" containerName="revoke-keys"
Nov 03 17:40:23.127796 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.127777    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm"
Nov 03 17:40:23.184007 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.183969    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:40:23.190569 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.190523    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm"]
Nov 03 17:40:23.235897 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.235857    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:40:23.235897 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.235858    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:40:23.282690 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.282656    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46171787-934f-473e-9de9-b9d4e0f8f68a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7535f460-tdccm\" (UID: \"46171787-934f-473e-9de9-b9d4e0f8f68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm"
Nov 03 17:40:23.282862 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.282722    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46171787-934f-473e-9de9-b9d4e0f8f68a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7535f460-tdccm\" (UID: \"46171787-934f-473e-9de9-b9d4e0f8f68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm"
Nov 03 17:40:23.282917 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.282860    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28ccd\" (UniqueName: \"kubernetes.io/projected/46171787-934f-473e-9de9-b9d4e0f8f68a-kube-api-access-28ccd\") pod \"maas-api-revoke-sub-7535f460-tdccm\" (UID: \"46171787-934f-473e-9de9-b9d4e0f8f68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm"
Nov 03 17:40:23.284158 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.284138    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:40:23.384440 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.384316    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46171787-934f-473e-9de9-b9d4e0f8f68a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7535f460-tdccm\" (UID: \"46171787-934f-473e-9de9-b9d4e0f8f68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm"
Nov 03 17:40:23.384440 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.384413    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46171787-934f-473e-9de9-b9d4e0f8f68a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7535f460-tdccm\" (UID: \"46171787-934f-473e-9de9-b9d4e0f8f68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm"
Nov 03 17:40:23.384656 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.384487    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28ccd\" (UniqueName: \"kubernetes.io/projected/46171787-934f-473e-9de9-b9d4e0f8f68a-kube-api-access-28ccd\") pod \"maas-api-revoke-sub-7535f460-tdccm\" (UID: \"46171787-934f-473e-9de9-b9d4e0f8f68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm"
Nov 03 17:40:23.385010 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.384988    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46171787-934f-473e-9de9-b9d4e0f8f68a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7535f460-tdccm\" (UID: \"46171787-934f-473e-9de9-b9d4e0f8f68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm"
Nov 03 17:40:23.386812 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.386781    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46171787-934f-473e-9de9-b9d4e0f8f68a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7535f460-tdccm\" (UID: \"46171787-934f-473e-9de9-b9d4e0f8f68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm"
Nov 03 17:40:23.411780 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.411747    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:23.448582 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.448553    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28ccd\" (UniqueName: \"kubernetes.io/projected/46171787-934f-473e-9de9-b9d4e0f8f68a-kube-api-access-28ccd\") pod \"maas-api-revoke-sub-7535f460-tdccm\" (UID: \"46171787-934f-473e-9de9-b9d4e0f8f68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm"
Nov 03 17:40:23.597983 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.597940    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:23.741502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.741458    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm"
Nov 03 17:40:23.784308 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:23.784278    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:24.123114 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:40:24.123076    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46171787_934f_473e_9de9_b9d4e0f8f68a.slice/crio-cbe342727fca21d273b3a0befafce54bb4144a99f58b9fd9cb3f2d588d352bcc WatchSource:0}: Error finding container cbe342727fca21d273b3a0befafce54bb4144a99f58b9fd9cb3f2d588d352bcc: Status 404 returned error can't find the container with id cbe342727fca21d273b3a0befafce54bb4144a99f58b9fd9cb3f2d588d352bcc
Nov 03 17:40:24.126708 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:24.126685    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm"]
Nov 03 17:40:24.673701 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:24.673656    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm" event={"ID":"46171787-934f-473e-9de9-b9d4e0f8f68a","Type":"ContainerStarted","Data":"2ee4c8ab6ae4bee7718d4353c1fad1b36e11c0b17e3da7620c3493e233dcc9ae"}
Nov 03 17:40:24.673887 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:24.673763    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm" event={"ID":"46171787-934f-473e-9de9-b9d4e0f8f68a","Type":"ContainerStarted","Data":"cbe342727fca21d273b3a0befafce54bb4144a99f58b9fd9cb3f2d588d352bcc"}
Nov 03 17:40:24.798653 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:24.798595    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm" podStartSLOduration=1.79858157 podStartE2EDuration="1.79858157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:24.794727041 +0000 UTC m=+1365.592439823" watchObservedRunningTime="2026-11-03 17:40:24.79858157 +0000 UTC m=+1365.596294344"
Nov 03 17:40:25.156222 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:25.156136    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh"]
Nov 03 17:40:25.257194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:25.257157    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w"]
Nov 03 17:40:25.539018 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:25.538981    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:25.684677 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:25.684635    6059 generic.go:356] "Generic (PLEG): container finished" podID="46171787-934f-473e-9de9-b9d4e0f8f68a" containerID="2ee4c8ab6ae4bee7718d4353c1fad1b36e11c0b17e3da7620c3493e233dcc9ae" exitCode=0
Nov 03 17:40:25.684677 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:25.684682    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm" event={"ID":"46171787-934f-473e-9de9-b9d4e0f8f68a","Type":"ContainerDied","Data":"2ee4c8ab6ae4bee7718d4353c1fad1b36e11c0b17e3da7620c3493e233dcc9ae"}
Nov 03 17:40:25.728793 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:25.728755    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:26.884640 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:26.884611    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm"
Nov 03 17:40:27.042412 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:27.042348    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46171787-934f-473e-9de9-b9d4e0f8f68a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46171787-934f-473e-9de9-b9d4e0f8f68a-maas-api-openshift-ca\") pod \"46171787-934f-473e-9de9-b9d4e0f8f68a\" (UID: \"46171787-934f-473e-9de9-b9d4e0f8f68a\") "
Nov 03 17:40:27.042607 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:27.042504    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46171787-934f-473e-9de9-b9d4e0f8f68a-kube-api-access-28ccd\" (UniqueName: \"kubernetes.io/projected/46171787-934f-473e-9de9-b9d4e0f8f68a-kube-api-access-28ccd\") pod \"46171787-934f-473e-9de9-b9d4e0f8f68a\" (UID: \"46171787-934f-473e-9de9-b9d4e0f8f68a\") "
Nov 03 17:40:27.042674 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:27.042641    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46171787-934f-473e-9de9-b9d4e0f8f68a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46171787-934f-473e-9de9-b9d4e0f8f68a-maas-api-xks-ca\") pod \"46171787-934f-473e-9de9-b9d4e0f8f68a\" (UID: \"46171787-934f-473e-9de9-b9d4e0f8f68a\") "
Nov 03 17:40:27.042797 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:27.042768    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46171787-934f-473e-9de9-b9d4e0f8f68a-maas-api-openshift-ca" pod "46171787-934f-473e-9de9-b9d4e0f8f68a" (UID: "46171787-934f-473e-9de9-b9d4e0f8f68a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:40:27.043596 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:27.043570    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46171787-934f-473e-9de9-b9d4e0f8f68a-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:27.044965 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:27.044933    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46171787-934f-473e-9de9-b9d4e0f8f68a-kube-api-access-28ccd" pod "46171787-934f-473e-9de9-b9d4e0f8f68a" (UID: "46171787-934f-473e-9de9-b9d4e0f8f68a"). InnerVolumeSpecName "kube-api-access-28ccd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:40:27.045248 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:27.045219    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46171787-934f-473e-9de9-b9d4e0f8f68a-maas-api-xks-ca" pod "46171787-934f-473e-9de9-b9d4e0f8f68a" (UID: "46171787-934f-473e-9de9-b9d4e0f8f68a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:40:27.144543 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:27.144504    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28ccd\" (UniqueName: \"kubernetes.io/projected/46171787-934f-473e-9de9-b9d4e0f8f68a-kube-api-access-28ccd\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:27.144543 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:27.144539    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46171787-934f-473e-9de9-b9d4e0f8f68a-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:27.702943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:27.702861    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm"
Nov 03 17:40:27.702943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:27.702894    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm" event={"ID":"46171787-934f-473e-9de9-b9d4e0f8f68a","Type":"ContainerDied","Data":"cbe342727fca21d273b3a0befafce54bb4144a99f58b9fd9cb3f2d588d352bcc"}
Nov 03 17:40:27.702943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:27.702941    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbe342727fca21d273b3a0befafce54bb4144a99f58b9fd9cb3f2d588d352bcc"
Nov 03 17:40:28.247961 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:28.247920    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:30.156762 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.156692    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh" podUID="b4bcd4ea-d922-4475-8e2e-c899be07ae49" containerName="payload-processing" containerID="cri-o://587e045046bf3f3af74b812d924d59b70a12ec96e1975b83424fd952bda37b94" gracePeriod=25
Nov 03 17:40:30.258395 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.258309    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w" podUID="62446602-8e77-48fc-baff-a974c95e75a4" containerName="payload-pre-processing" containerID="cri-o://307104d533cf8984d8ec34005f935cf70ce5dfaee77a21ac70636120bdafb0d4" gracePeriod=25
Nov 03 17:40:30.430061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.429976    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh"
Nov 03 17:40:30.522443 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.522411    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4bcd4ea-d922-4475-8e2e-c899be07ae49-kube-api-access-zgr9w\" (UniqueName: \"kubernetes.io/projected/b4bcd4ea-d922-4475-8e2e-c899be07ae49-kube-api-access-zgr9w\") pod \"b4bcd4ea-d922-4475-8e2e-c899be07ae49\" (UID: \"b4bcd4ea-d922-4475-8e2e-c899be07ae49\") "
Nov 03 17:40:30.522628 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.522572    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4bcd4ea-d922-4475-8e2e-c899be07ae49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bcd4ea-d922-4475-8e2e-c899be07ae49-plugins-config-volume\") pod \"b4bcd4ea-d922-4475-8e2e-c899be07ae49\" (UID: \"b4bcd4ea-d922-4475-8e2e-c899be07ae49\") "
Nov 03 17:40:30.522999 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.522971    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4bcd4ea-d922-4475-8e2e-c899be07ae49-plugins-config-volume" pod "b4bcd4ea-d922-4475-8e2e-c899be07ae49" (UID: "b4bcd4ea-d922-4475-8e2e-c899be07ae49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:40:30.524568 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.524545    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4bcd4ea-d922-4475-8e2e-c899be07ae49-kube-api-access-zgr9w" pod "b4bcd4ea-d922-4475-8e2e-c899be07ae49" (UID: "b4bcd4ea-d922-4475-8e2e-c899be07ae49"). InnerVolumeSpecName "kube-api-access-zgr9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:40:30.536570 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.536540    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w"
Nov 03 17:40:30.624012 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.623977    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62446602-8e77-48fc-baff-a974c95e75a4-kube-api-access-4hqvt\" (UniqueName: \"kubernetes.io/projected/62446602-8e77-48fc-baff-a974c95e75a4-kube-api-access-4hqvt\") pod \"62446602-8e77-48fc-baff-a974c95e75a4\" (UID: \"62446602-8e77-48fc-baff-a974c95e75a4\") "
Nov 03 17:40:30.624199 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.624111    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62446602-8e77-48fc-baff-a974c95e75a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62446602-8e77-48fc-baff-a974c95e75a4-plugins-config-volume\") pod \"62446602-8e77-48fc-baff-a974c95e75a4\" (UID: \"62446602-8e77-48fc-baff-a974c95e75a4\") "
Nov 03 17:40:30.624427 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.624406    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bcd4ea-d922-4475-8e2e-c899be07ae49-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:30.624427 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.624428    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgr9w\" (UniqueName: \"kubernetes.io/projected/b4bcd4ea-d922-4475-8e2e-c899be07ae49-kube-api-access-zgr9w\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:30.624595 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.624483    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62446602-8e77-48fc-baff-a974c95e75a4-plugins-config-volume" pod "62446602-8e77-48fc-baff-a974c95e75a4" (UID: "62446602-8e77-48fc-baff-a974c95e75a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:40:30.626187 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.626166    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62446602-8e77-48fc-baff-a974c95e75a4-kube-api-access-4hqvt" pod "62446602-8e77-48fc-baff-a974c95e75a4" (UID: "62446602-8e77-48fc-baff-a974c95e75a4"). InnerVolumeSpecName "kube-api-access-4hqvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:40:30.725714 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.725676    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62446602-8e77-48fc-baff-a974c95e75a4-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:30.725714 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.725710    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hqvt\" (UniqueName: \"kubernetes.io/projected/62446602-8e77-48fc-baff-a974c95e75a4-kube-api-access-4hqvt\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:30.734023 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.733989    6059 generic.go:356] "Generic (PLEG): container finished" podID="62446602-8e77-48fc-baff-a974c95e75a4" containerID="307104d533cf8984d8ec34005f935cf70ce5dfaee77a21ac70636120bdafb0d4" exitCode=0
Nov 03 17:40:30.734207 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.734042    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w"
Nov 03 17:40:30.734207 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.734076    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w" event={"ID":"62446602-8e77-48fc-baff-a974c95e75a4","Type":"ContainerDied","Data":"307104d533cf8984d8ec34005f935cf70ce5dfaee77a21ac70636120bdafb0d4"}
Nov 03 17:40:30.734207 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.734128    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w" event={"ID":"62446602-8e77-48fc-baff-a974c95e75a4","Type":"ContainerDied","Data":"68a739f47873e511522ae65d643de71f71268d952a8f451e8dbf26230d6a035a"}
Nov 03 17:40:30.734207 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.734161    6059 scope.go:122] "RemoveContainer" containerID="307104d533cf8984d8ec34005f935cf70ce5dfaee77a21ac70636120bdafb0d4"
Nov 03 17:40:30.735678 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.735651    6059 generic.go:356] "Generic (PLEG): container finished" podID="b4bcd4ea-d922-4475-8e2e-c899be07ae49" containerID="587e045046bf3f3af74b812d924d59b70a12ec96e1975b83424fd952bda37b94" exitCode=0
Nov 03 17:40:30.735805 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.735718    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh"
Nov 03 17:40:30.735805 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.735748    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh" event={"ID":"b4bcd4ea-d922-4475-8e2e-c899be07ae49","Type":"ContainerDied","Data":"587e045046bf3f3af74b812d924d59b70a12ec96e1975b83424fd952bda37b94"}
Nov 03 17:40:30.735904 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.735813    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh" event={"ID":"b4bcd4ea-d922-4475-8e2e-c899be07ae49","Type":"ContainerDied","Data":"5f5f0caa9fd65c15b601c352bdf4c557f03c4b17308331931b0d24fef7d600cb"}
Nov 03 17:40:30.749102 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.748985    6059 scope.go:122] "RemoveContainer" containerID="307104d533cf8984d8ec34005f935cf70ce5dfaee77a21ac70636120bdafb0d4"
Nov 03 17:40:30.749979 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:40:30.749757    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"307104d533cf8984d8ec34005f935cf70ce5dfaee77a21ac70636120bdafb0d4\": container with ID starting with 307104d533cf8984d8ec34005f935cf70ce5dfaee77a21ac70636120bdafb0d4 not found: ID does not exist" containerID="307104d533cf8984d8ec34005f935cf70ce5dfaee77a21ac70636120bdafb0d4"
Nov 03 17:40:30.749979 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.749795    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"307104d533cf8984d8ec34005f935cf70ce5dfaee77a21ac70636120bdafb0d4"} err="failed to get container status \"307104d533cf8984d8ec34005f935cf70ce5dfaee77a21ac70636120bdafb0d4\": rpc error: code = NotFound desc = could not find container \"307104d533cf8984d8ec34005f935cf70ce5dfaee77a21ac70636120bdafb0d4\": container with ID starting with 307104d533cf8984d8ec34005f935cf70ce5dfaee77a21ac70636120bdafb0d4 not found: ID does not exist"
Nov 03 17:40:30.749979 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.749819    6059 scope.go:122] "RemoveContainer" containerID="587e045046bf3f3af74b812d924d59b70a12ec96e1975b83424fd952bda37b94"
Nov 03 17:40:30.761455 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.761424    6059 scope.go:122] "RemoveContainer" containerID="587e045046bf3f3af74b812d924d59b70a12ec96e1975b83424fd952bda37b94"
Nov 03 17:40:30.761796 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:40:30.761771    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"587e045046bf3f3af74b812d924d59b70a12ec96e1975b83424fd952bda37b94\": container with ID starting with 587e045046bf3f3af74b812d924d59b70a12ec96e1975b83424fd952bda37b94 not found: ID does not exist" containerID="587e045046bf3f3af74b812d924d59b70a12ec96e1975b83424fd952bda37b94"
Nov 03 17:40:30.761900 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.761802    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"587e045046bf3f3af74b812d924d59b70a12ec96e1975b83424fd952bda37b94"} err="failed to get container status \"587e045046bf3f3af74b812d924d59b70a12ec96e1975b83424fd952bda37b94\": rpc error: code = NotFound desc = could not find container \"587e045046bf3f3af74b812d924d59b70a12ec96e1975b83424fd952bda37b94\": container with ID starting with 587e045046bf3f3af74b812d924d59b70a12ec96e1975b83424fd952bda37b94 not found: ID does not exist"
Nov 03 17:40:30.907950 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.907912    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh"]
Nov 03 17:40:30.912223 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:30.912193    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ed607ea-68ff7d8f97-djdzh"]
Nov 03 17:40:31.025498 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:31.025410    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w"]
Nov 03 17:40:31.032051 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:31.032011    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ed607ea-86bc4584f9-46r9w"]
Nov 03 17:40:31.336290 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:31.336200    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62446602-8e77-48fc-baff-a974c95e75a4" path="/var/lib/kubelet/pods/62446602-8e77-48fc-baff-a974c95e75a4/volumes"
Nov 03 17:40:31.336803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:31.336774    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4bcd4ea-d922-4475-8e2e-c899be07ae49" path="/var/lib/kubelet/pods/b4bcd4ea-d922-4475-8e2e-c899be07ae49/volumes"
Nov 03 17:40:32.052972 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:32.052933    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff"]
Nov 03 17:40:32.104477 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:32.104438    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-121693e7-tdjff"]
Nov 03 17:40:32.943189 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:32.943158    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:33.188443 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:33.188411    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:33.336508 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:33.336474    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7" path="/var/lib/kubelet/pods/2bd9ecd8-3fd2-44cc-8005-f30e6c0263d7/volumes"
Nov 03 17:40:33.378507 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:33.378472    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:36.862283 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:40:36.862243    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:40:37.418197 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:37.418163    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:38.027557 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:38.027494    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:38.828188 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:38.828149    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ddc4390-j9zbn"]
Nov 03 17:40:38.828864 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:38.828840    6059 cpu_manager.go:414] "containerMap: removing container" podUID="46171787-934f-473e-9de9-b9d4e0f8f68a" containerName="revoke-keys"
Nov 03 17:40:38.828864 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:38.828860    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46171787-934f-473e-9de9-b9d4e0f8f68a" containerName="revoke-keys"
Nov 03 17:40:38.829074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:38.828888    6059 cpu_manager.go:414] "containerMap: removing container" podUID="b4bcd4ea-d922-4475-8e2e-c899be07ae49" containerName="payload-processing"
Nov 03 17:40:38.829074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:38.828894    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4bcd4ea-d922-4475-8e2e-c899be07ae49" containerName="payload-processing"
Nov 03 17:40:38.829074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:38.828903    6059 cpu_manager.go:414] "containerMap: removing container" podUID="62446602-8e77-48fc-baff-a974c95e75a4" containerName="payload-pre-processing"
Nov 03 17:40:38.829074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:38.828908    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62446602-8e77-48fc-baff-a974c95e75a4" containerName="payload-pre-processing"
Nov 03 17:40:38.829074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:38.829040    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="46171787-934f-473e-9de9-b9d4e0f8f68a" containerName="revoke-keys"
Nov 03 17:40:38.829074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:38.829051    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4bcd4ea-d922-4475-8e2e-c899be07ae49" containerName="payload-processing"
Nov 03 17:40:38.829074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:38.829060    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="62446602-8e77-48fc-baff-a974c95e75a4" containerName="payload-pre-processing"
Nov 03 17:40:38.834679 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:38.834651    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ddc4390-j9zbn"
Nov 03 17:40:38.936626 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:38.936579    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49fb5c34-7a16-4094-b33c-2c511e1c1bd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ddc4390-j9zbn\" (UID: \"49fb5c34-7a16-4094-b33c-2c511e1c1bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ddc4390-j9zbn"
Nov 03 17:40:38.936776 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:38.936732    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njnz6\" (UniqueName: \"kubernetes.io/projected/49fb5c34-7a16-4094-b33c-2c511e1c1bd9-kube-api-access-njnz6\") pod \"maas-api-revoke-sub-0ddc4390-j9zbn\" (UID: \"49fb5c34-7a16-4094-b33c-2c511e1c1bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ddc4390-j9zbn"
Nov 03 17:40:38.936848 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:38.936835    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49fb5c34-7a16-4094-b33c-2c511e1c1bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ddc4390-j9zbn\" (UID: \"49fb5c34-7a16-4094-b33c-2c511e1c1bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ddc4390-j9zbn"
Nov 03 17:40:38.941539 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:38.941503    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:40:38.941539 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:38.941521    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:40:38.941737 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:38.941556    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:40:38.941737 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:38.941651    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:40:38.948209 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:38.948153    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ddc4390-j9zbn"]
Nov 03 17:40:39.038483 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:39.038445    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49fb5c34-7a16-4094-b33c-2c511e1c1bd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ddc4390-j9zbn\" (UID: \"49fb5c34-7a16-4094-b33c-2c511e1c1bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ddc4390-j9zbn"
Nov 03 17:40:39.038946 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:39.038548    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njnz6\" (UniqueName: \"kubernetes.io/projected/49fb5c34-7a16-4094-b33c-2c511e1c1bd9-kube-api-access-njnz6\") pod \"maas-api-revoke-sub-0ddc4390-j9zbn\" (UID: \"49fb5c34-7a16-4094-b33c-2c511e1c1bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ddc4390-j9zbn"
Nov 03 17:40:39.038946 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:39.038620    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49fb5c34-7a16-4094-b33c-2c511e1c1bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ddc4390-j9zbn\" (UID: \"49fb5c34-7a16-4094-b33c-2c511e1c1bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ddc4390-j9zbn"
Nov 03 17:40:39.039280 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:39.039259    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49fb5c34-7a16-4094-b33c-2c511e1c1bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ddc4390-j9zbn\" (UID: \"49fb5c34-7a16-4094-b33c-2c511e1c1bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ddc4390-j9zbn"
Nov 03 17:40:39.040886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:39.040863    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49fb5c34-7a16-4094-b33c-2c511e1c1bd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ddc4390-j9zbn\" (UID: \"49fb5c34-7a16-4094-b33c-2c511e1c1bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ddc4390-j9zbn"
Nov 03 17:40:39.103091 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:39.103016    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njnz6\" (UniqueName: \"kubernetes.io/projected/49fb5c34-7a16-4094-b33c-2c511e1c1bd9-kube-api-access-njnz6\") pod \"maas-api-revoke-sub-0ddc4390-j9zbn\" (UID: \"49fb5c34-7a16-4094-b33c-2c511e1c1bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ddc4390-j9zbn"
Nov 03 17:40:39.151335 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:39.151294    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ddc4390-j9zbn"
Nov 03 17:40:39.528013 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:40:39.527982    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49fb5c34_7a16_4094_b33c_2c511e1c1bd9.slice/crio-b961a29659a6e630a90c4431fa52ffda7d75d998edc9bae5d9d70a4d2241ebd8 WatchSource:0}: Error finding container b961a29659a6e630a90c4431fa52ffda7d75d998edc9bae5d9d70a4d2241ebd8: Status 404 returned error can't find the container with id b961a29659a6e630a90c4431fa52ffda7d75d998edc9bae5d9d70a4d2241ebd8
Nov 03 17:40:39.532943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:39.532913    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ddc4390-j9zbn"]
Nov 03 17:40:39.819527 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:39.819439    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ddc4390-j9zbn" event={"ID":"49fb5c34-7a16-4094-b33c-2c511e1c1bd9","Type":"ContainerStarted","Data":"b961a29659a6e630a90c4431fa52ffda7d75d998edc9bae5d9d70a4d2241ebd8"}
Nov 03 17:40:40.832955 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:40.832913    6059 generic.go:356] "Generic (PLEG): container finished" podID="49fb5c34-7a16-4094-b33c-2c511e1c1bd9" containerID="f4386b6bf0c68ca3438c818d8c4f92da1e467754ff75f89a33cbfe8f3a42f631" exitCode=0
Nov 03 17:40:40.833414 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:40.832957    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ddc4390-j9zbn" event={"ID":"49fb5c34-7a16-4094-b33c-2c511e1c1bd9","Type":"ContainerDied","Data":"f4386b6bf0c68ca3438c818d8c4f92da1e467754ff75f89a33cbfe8f3a42f631"}
Nov 03 17:40:41.066377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:41.059878    6059 scope.go:122] "RemoveContainer" containerID="bf101c32c926ff4037f57d5ec3911e54d53378379617637d1b1424b61963244c"
Nov 03 17:40:41.110942 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:41.110915    6059 scope.go:122] "RemoveContainer" containerID="fe4786836e46cf4a9ecad018848f5e360a14dd7bdfdcff44fe9ed1f8d2a56f7b"
Nov 03 17:40:41.130074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:41.130031    6059 scope.go:122] "RemoveContainer" containerID="77d4243ce418b3bb4f6eb65df4dd304d381fe7fe1f80b8d02687c382c26342c3"
Nov 03 17:40:41.144262 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:41.144233    6059 scope.go:122] "RemoveContainer" containerID="9fda63b2860641b2711b05f545ee68ff7c66d9c47ebf747f0626c2dbc6aa737e"
Nov 03 17:40:41.155517 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:41.155493    6059 scope.go:122] "RemoveContainer" containerID="375ffdac5ca9cafdcf30c62c70f6a99878ae4ecb728adb279462b3edd7703be5"
Nov 03 17:40:41.170047 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:41.170021    6059 scope.go:122] "RemoveContainer" containerID="30f37b5f8372446715faa033514c9cd1b762f5a829e2adffc8f04425f129e5b5"
Nov 03 17:40:42.019462 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:42.019433    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ddc4390-j9zbn"
Nov 03 17:40:42.047405 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:42.036870    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:42.130442 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:42.120793    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49fb5c34-7a16-4094-b33c-2c511e1c1bd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49fb5c34-7a16-4094-b33c-2c511e1c1bd9-maas-api-openshift-ca\") pod \"49fb5c34-7a16-4094-b33c-2c511e1c1bd9\" (UID: \"49fb5c34-7a16-4094-b33c-2c511e1c1bd9\") "
Nov 03 17:40:42.130442 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:42.120980    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49fb5c34-7a16-4094-b33c-2c511e1c1bd9-kube-api-access-njnz6\" (UniqueName: \"kubernetes.io/projected/49fb5c34-7a16-4094-b33c-2c511e1c1bd9-kube-api-access-njnz6\") pod \"49fb5c34-7a16-4094-b33c-2c511e1c1bd9\" (UID: \"49fb5c34-7a16-4094-b33c-2c511e1c1bd9\") "
Nov 03 17:40:42.130442 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:42.121038    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49fb5c34-7a16-4094-b33c-2c511e1c1bd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49fb5c34-7a16-4094-b33c-2c511e1c1bd9-maas-api-xks-ca\") pod \"49fb5c34-7a16-4094-b33c-2c511e1c1bd9\" (UID: \"49fb5c34-7a16-4094-b33c-2c511e1c1bd9\") "
Nov 03 17:40:42.131173 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:42.131142    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49fb5c34-7a16-4094-b33c-2c511e1c1bd9-maas-api-openshift-ca" pod "49fb5c34-7a16-4094-b33c-2c511e1c1bd9" (UID: "49fb5c34-7a16-4094-b33c-2c511e1c1bd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:40:42.137093 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:42.137066    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49fb5c34-7a16-4094-b33c-2c511e1c1bd9-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:42.167385 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:42.150682    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49fb5c34-7a16-4094-b33c-2c511e1c1bd9-kube-api-access-njnz6" pod "49fb5c34-7a16-4094-b33c-2c511e1c1bd9" (UID: "49fb5c34-7a16-4094-b33c-2c511e1c1bd9"). InnerVolumeSpecName "kube-api-access-njnz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:40:42.175407 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:42.173654    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49fb5c34-7a16-4094-b33c-2c511e1c1bd9-maas-api-xks-ca" pod "49fb5c34-7a16-4094-b33c-2c511e1c1bd9" (UID: "49fb5c34-7a16-4094-b33c-2c511e1c1bd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:40:42.273447 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:42.273403    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njnz6\" (UniqueName: \"kubernetes.io/projected/49fb5c34-7a16-4094-b33c-2c511e1c1bd9-kube-api-access-njnz6\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:42.273828 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:42.273807    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49fb5c34-7a16-4094-b33c-2c511e1c1bd9-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:42.867662 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:42.867609    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ddc4390-j9zbn" event={"ID":"49fb5c34-7a16-4094-b33c-2c511e1c1bd9","Type":"ContainerDied","Data":"b961a29659a6e630a90c4431fa52ffda7d75d998edc9bae5d9d70a4d2241ebd8"}
Nov 03 17:40:42.872379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:42.867963    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b961a29659a6e630a90c4431fa52ffda7d75d998edc9bae5d9d70a4d2241ebd8"
Nov 03 17:40:42.872583 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:42.868499    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ddc4390-j9zbn"
Nov 03 17:40:46.256133 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:46.256098    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:47.717371 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:47.717321    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:47.921703 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:47.921656    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:48.156101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:48.156063    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:48.198142 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:48.198101    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ed607ea-n2477"]
Nov 03 17:40:48.252668 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:48.252629    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ed607ea-n2477"]
Nov 03 17:40:48.302022 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:48.301984    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"]
Nov 03 17:40:48.302256 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:48.302233    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq" podUID="3e53c74d-ceff-4b92-b579-35ef2b76de83" containerName="istio-proxy" containerID="cri-o://2e86ef0087015f81d2b1179ff4297ab04a7b43b4625f6d2a4e0d5915c673c901" gracePeriod=30
Nov 03 17:40:48.373325 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:48.373292    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:48.563141 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:48.563110    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:48.980072 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:48.980026    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:49.061081 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:49.061044    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc"]
Nov 03 17:40:49.110602 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:49.110570    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70ffcd57-lmcgc"]
Nov 03 17:40:49.168631 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:49.168595    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:49.336867 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:49.336787    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204564fa-b79b-4040-adfa-f732c266c065" path="/var/lib/kubelet/pods/204564fa-b79b-4040-adfa-f732c266c065/volumes"
Nov 03 17:40:49.337222 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:49.337207    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bbbdb77-9224-4eff-9937-e060f53137f5" path="/var/lib/kubelet/pods/2bbbdb77-9224-4eff-9937-e060f53137f5/volumes"
Nov 03 17:40:49.350277 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:49.350241    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:49.884232 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:49.884192    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"]
Nov 03 17:40:49.885079 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:49.885051    6059 cpu_manager.go:414] "containerMap: removing container" podUID="49fb5c34-7a16-4094-b33c-2c511e1c1bd9" containerName="revoke-keys"
Nov 03 17:40:49.885254 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:49.885239    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49fb5c34-7a16-4094-b33c-2c511e1c1bd9" containerName="revoke-keys"
Nov 03 17:40:49.885622 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:49.885607    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="49fb5c34-7a16-4094-b33c-2c511e1c1bd9" containerName="revoke-keys"
Nov 03 17:40:49.893457 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:49.891757    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:49.948239 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:49.948160    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-5f88f673-gw-openshift-default-dockercfg-q72k8\""
Nov 03 17:40:49.965500 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:49.964633    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-data\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:49.965500 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:49.964768    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-token\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:49.965500 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:49.964810    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-envoy\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:49.965500 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:49.964857    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-workload-socket\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:49.965500 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:49.964955    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-workload-certs\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:49.965500 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:49.965009    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-credential-socket\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:49.965500 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:49.965075    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istiod-ca-cert\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:49.965500 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:49.965111    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-podinfo\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:49.965500 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:49.965261    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2msdn\" (UniqueName: \"kubernetes.io/projected/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-kube-api-access-2msdn\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:50.005795 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.005764    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"]
Nov 03 17:40:50.023498 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.023467    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:50.066327 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.066271    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-token\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:50.066558 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.066336    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-envoy\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:50.066558 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.066391    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-workload-socket\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:50.066558 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.066481    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-workload-certs\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:50.066558 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.066517    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-credential-socket\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:50.066558 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.066556    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istiod-ca-cert\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:50.066943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.066590    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-podinfo\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:50.066943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.066724    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2msdn\" (UniqueName: \"kubernetes.io/projected/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-kube-api-access-2msdn\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:50.066943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.066780    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-data\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:50.066943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.066905    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-workload-socket\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:50.067164 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.066986    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-workload-certs\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:50.067164 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.067101    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-data\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:50.067281 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.067180    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-credential-socket\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:50.067501 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.067476    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istiod-ca-cert\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:50.069518 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.069493    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-envoy\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:50.069761 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.069744    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-podinfo\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:50.129810 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.129773    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-token\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:50.132157 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.132134    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2msdn\" (UniqueName: \"kubernetes.io/projected/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-kube-api-access-2msdn\") pod \"e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") " pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:50.216482 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.216442    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:50.233425 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.233392    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:50.666420 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:40:50.666382    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b11bcfe_5150_4eee_95e9_ee213aea1e8e.slice/crio-916cfa16a0b1c61159e82dbe57f8d596a28be895c6cfa3e79842c689b6d7a0df WatchSource:0}: Error finding container 916cfa16a0b1c61159e82dbe57f8d596a28be895c6cfa3e79842c689b6d7a0df: Status 404 returned error can't find the container with id 916cfa16a0b1c61159e82dbe57f8d596a28be895c6cfa3e79842c689b6d7a0df
Nov 03 17:40:50.669034 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.669002    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:40:50.669123 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.669072    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:40:50.669123 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.669100    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:40:50.669949 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:50.669919    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"]
Nov 03 17:40:51.007460 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:51.007415    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2" event={"ID":"6b11bcfe-5150-4eee-95e9-ee213aea1e8e","Type":"ContainerStarted","Data":"03b25e537b3bc4296111e37a811e5ddd0c7188a88a34cc744fe549af8516d7a0"}
Nov 03 17:40:51.007946 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:51.007530    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2" event={"ID":"6b11bcfe-5150-4eee-95e9-ee213aea1e8e","Type":"ContainerStarted","Data":"916cfa16a0b1c61159e82dbe57f8d596a28be895c6cfa3e79842c689b6d7a0df"}
Nov 03 17:40:51.027534 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:51.027494    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:51.133609 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:51.133544    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2" podStartSLOduration=2.133522664 podStartE2EDuration="2.133522664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:51.130819236 +0000 UTC m=+1391.928532017" watchObservedRunningTime="2026-11-03 17:40:51.133522664 +0000 UTC m=+1391.931235448"
Nov 03 17:40:51.197960 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:51.197928    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:51.233938 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:51.233898    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:51.239853 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:51.239817    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:51.383031 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:51.382951    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:51.504667 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:51.504627    6059 patch_prober.go:29] interesting pod/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:40:51.508451 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:51.504692    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq" podUID="3e53c74d-ceff-4b92-b579-35ef2b76de83" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:40:52.017483 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:52.017447    6059 generic.go:356] "Generic (PLEG): container finished" podID="7d3b16ed-f9c0-4c11-b7de-4f472c859170" containerID="34f240a2a5be507ab00360ed3768aa60e74a382a394fe9b62f5dcc0638588a85" exitCode=0
Nov 03 17:40:52.017917 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:52.017515    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7" event={"ID":"7d3b16ed-f9c0-4c11-b7de-4f472c859170","Type":"ContainerDied","Data":"34f240a2a5be507ab00360ed3768aa60e74a382a394fe9b62f5dcc0638588a85"}
Nov 03 17:40:52.017917 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:52.017784    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:52.018126 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:52.018108    6059 scope.go:122] "RemoveContainer" containerID="34f240a2a5be507ab00360ed3768aa60e74a382a394fe9b62f5dcc0638588a85"
Nov 03 17:40:52.019093 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:52.018944    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:40:52.019620 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:52.019589    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:40:52.019679 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:52.019661    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:40:52.019721 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:52.019694    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:40:53.028092 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.028051    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7" event={"ID":"7d3b16ed-f9c0-4c11-b7de-4f472c859170","Type":"ContainerStarted","Data":"044182d8a0b4d486379b91efbcce4bc4cbfa3077d05e45804fd86c7c9cbf4f37"}
Nov 03 17:40:53.117695 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.117655    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:53.300705 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.300612    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:53.427685 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.427653    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:40:53.427907 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.427883    6059 patch_prober.go:29] interesting pod/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 03 17:40:53.427958 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.427931    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7" podUID="7d3b16ed-f9c0-4c11-b7de-4f472c859170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 03 17:40:53.654673 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.654637    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:40:53.655933 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.655904    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:53.724577 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.724531    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-data\") pod \"3e53c74d-ceff-4b92-b579-35ef2b76de83\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") "
Nov 03 17:40:53.724759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.724641    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-envoy\") pod \"3e53c74d-ceff-4b92-b579-35ef2b76de83\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") "
Nov 03 17:40:53.724759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.724668    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e53c74d-ceff-4b92-b579-35ef2b76de83-kube-api-access-5bjqp\" (UniqueName: \"kubernetes.io/projected/3e53c74d-ceff-4b92-b579-35ef2b76de83-kube-api-access-5bjqp\") pod \"3e53c74d-ceff-4b92-b579-35ef2b76de83\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") "
Nov 03 17:40:53.724759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.724699    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-podinfo\") pod \"3e53c74d-ceff-4b92-b579-35ef2b76de83\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") "
Nov 03 17:40:53.724759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.724748    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e53c74d-ceff-4b92-b579-35ef2b76de83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e53c74d-ceff-4b92-b579-35ef2b76de83-istiod-ca-cert\") pod \"3e53c74d-ceff-4b92-b579-35ef2b76de83\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") "
Nov 03 17:40:53.725009 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.724827    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-workload-certs\") pod \"3e53c74d-ceff-4b92-b579-35ef2b76de83\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") "
Nov 03 17:40:53.725009 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.724855    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-token\" (UniqueName: \"kubernetes.io/projected/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-token\") pod \"3e53c74d-ceff-4b92-b579-35ef2b76de83\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") "
Nov 03 17:40:53.725009 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.724921    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-workload-socket\") pod \"3e53c74d-ceff-4b92-b579-35ef2b76de83\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") "
Nov 03 17:40:53.725009 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.724923    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-data" pod "3e53c74d-ceff-4b92-b579-35ef2b76de83" (UID: "3e53c74d-ceff-4b92-b579-35ef2b76de83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:53.725009 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.724981    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-credential-socket\") pod \"3e53c74d-ceff-4b92-b579-35ef2b76de83\" (UID: \"3e53c74d-ceff-4b92-b579-35ef2b76de83\") "
Nov 03 17:40:53.725264 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.725090    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-workload-certs" pod "3e53c74d-ceff-4b92-b579-35ef2b76de83" (UID: "3e53c74d-ceff-4b92-b579-35ef2b76de83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:53.725264 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.725233    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-workload-socket" pod "3e53c74d-ceff-4b92-b579-35ef2b76de83" (UID: "3e53c74d-ceff-4b92-b579-35ef2b76de83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:53.725470 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.725446    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-credential-socket" pod "3e53c74d-ceff-4b92-b579-35ef2b76de83" (UID: "3e53c74d-ceff-4b92-b579-35ef2b76de83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:53.725578 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.725556    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e53c74d-ceff-4b92-b579-35ef2b76de83-istiod-ca-cert" pod "3e53c74d-ceff-4b92-b579-35ef2b76de83" (UID: "3e53c74d-ceff-4b92-b579-35ef2b76de83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:40:53.725687 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.725666    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:53.725812 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.725689    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e53c74d-ceff-4b92-b579-35ef2b76de83-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:53.725812 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.725703    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:53.725812 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.725718    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:53.725812 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.725731    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:53.727569 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.727536    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-token" pod "3e53c74d-ceff-4b92-b579-35ef2b76de83" (UID: "3e53c74d-ceff-4b92-b579-35ef2b76de83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:40:53.727674 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.727563    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e53c74d-ceff-4b92-b579-35ef2b76de83-kube-api-access-5bjqp" pod "3e53c74d-ceff-4b92-b579-35ef2b76de83" (UID: "3e53c74d-ceff-4b92-b579-35ef2b76de83"). InnerVolumeSpecName "kube-api-access-5bjqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:40:53.727674 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.727632    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-podinfo" pod "3e53c74d-ceff-4b92-b579-35ef2b76de83" (UID: "3e53c74d-ceff-4b92-b579-35ef2b76de83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:40:53.727674 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.727641    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-envoy" pod "3e53c74d-ceff-4b92-b579-35ef2b76de83" (UID: "3e53c74d-ceff-4b92-b579-35ef2b76de83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:40:53.827147 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.827092    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:53.827147 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.827140    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bjqp\" (UniqueName: \"kubernetes.io/projected/3e53c74d-ceff-4b92-b579-35ef2b76de83-kube-api-access-5bjqp\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:53.827147 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.827153    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:53.827147 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.827161    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e53c74d-ceff-4b92-b579-35ef2b76de83-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:40:53.855342 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:53.855249    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:54.066203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:54.066154    6059 generic.go:356] "Generic (PLEG): container finished" podID="3e53c74d-ceff-4b92-b579-35ef2b76de83" containerID="2e86ef0087015f81d2b1179ff4297ab04a7b43b4625f6d2a4e0d5915c673c901" exitCode=0
Nov 03 17:40:54.067706 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:54.067672    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq" event={"ID":"3e53c74d-ceff-4b92-b579-35ef2b76de83","Type":"ContainerDied","Data":"2e86ef0087015f81d2b1179ff4297ab04a7b43b4625f6d2a4e0d5915c673c901"}
Nov 03 17:40:54.067798 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:54.067783    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq" event={"ID":"3e53c74d-ceff-4b92-b579-35ef2b76de83","Type":"ContainerDied","Data":"9f965a28f39a3902b6ea2dbf29e3205c09d1ce9bd544ee3598fa4643fb8bae69"}
Nov 03 17:40:54.067843 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:54.067692    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"
Nov 03 17:40:54.067843 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:54.067827    6059 scope.go:122] "RemoveContainer" containerID="2e86ef0087015f81d2b1179ff4297ab04a7b43b4625f6d2a4e0d5915c673c901"
Nov 03 17:40:54.082656 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:54.082634    6059 scope.go:122] "RemoveContainer" containerID="c29ac6f3f77f4fdf3a59636f7804eac7311724a5feb38d17d8b8248cc0e12fb6"
Nov 03 17:40:54.093370 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:54.093325    6059 scope.go:122] "RemoveContainer" containerID="2e86ef0087015f81d2b1179ff4297ab04a7b43b4625f6d2a4e0d5915c673c901"
Nov 03 17:40:54.094054 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:40:54.094030    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e86ef0087015f81d2b1179ff4297ab04a7b43b4625f6d2a4e0d5915c673c901\": container with ID starting with 2e86ef0087015f81d2b1179ff4297ab04a7b43b4625f6d2a4e0d5915c673c901 not found: ID does not exist" containerID="2e86ef0087015f81d2b1179ff4297ab04a7b43b4625f6d2a4e0d5915c673c901"
Nov 03 17:40:54.094165 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:54.094065    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e86ef0087015f81d2b1179ff4297ab04a7b43b4625f6d2a4e0d5915c673c901"} err="failed to get container status \"2e86ef0087015f81d2b1179ff4297ab04a7b43b4625f6d2a4e0d5915c673c901\": rpc error: code = NotFound desc = could not find container \"2e86ef0087015f81d2b1179ff4297ab04a7b43b4625f6d2a4e0d5915c673c901\": container with ID starting with 2e86ef0087015f81d2b1179ff4297ab04a7b43b4625f6d2a4e0d5915c673c901 not found: ID does not exist"
Nov 03 17:40:54.094165 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:54.094092    6059 scope.go:122] "RemoveContainer" containerID="c29ac6f3f77f4fdf3a59636f7804eac7311724a5feb38d17d8b8248cc0e12fb6"
Nov 03 17:40:54.094464 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:40:54.094443    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c29ac6f3f77f4fdf3a59636f7804eac7311724a5feb38d17d8b8248cc0e12fb6\": container with ID starting with c29ac6f3f77f4fdf3a59636f7804eac7311724a5feb38d17d8b8248cc0e12fb6 not found: ID does not exist" containerID="c29ac6f3f77f4fdf3a59636f7804eac7311724a5feb38d17d8b8248cc0e12fb6"
Nov 03 17:40:54.094521 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:54.094471    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c29ac6f3f77f4fdf3a59636f7804eac7311724a5feb38d17d8b8248cc0e12fb6"} err="failed to get container status \"c29ac6f3f77f4fdf3a59636f7804eac7311724a5feb38d17d8b8248cc0e12fb6\": rpc error: code = NotFound desc = could not find container \"c29ac6f3f77f4fdf3a59636f7804eac7311724a5feb38d17d8b8248cc0e12fb6\": container with ID starting with c29ac6f3f77f4fdf3a59636f7804eac7311724a5feb38d17d8b8248cc0e12fb6 not found: ID does not exist"
Nov 03 17:40:54.257573 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:54.257528    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"]
Nov 03 17:40:54.294572 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:54.294533    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1ed607ea-openshift-default-655cc5c447-z4glq"]
Nov 03 17:40:54.427257 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:54.427223    6059 patch_prober.go:29] interesting pod/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 03 17:40:54.427502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:54.427285    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7" podUID="7d3b16ed-f9c0-4c11-b7de-4f472c859170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 03 17:40:55.155281 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.155243    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:55.364781 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.364746    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e53c74d-ceff-4b92-b579-35ef2b76de83" path="/var/lib/kubelet/pods/3e53c74d-ceff-4b92-b579-35ef2b76de83/volumes"
Nov 03 17:40:55.427765 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.427677    6059 patch_prober.go:29] interesting pod/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 03 17:40:55.427765 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.427743    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7" podUID="7d3b16ed-f9c0-4c11-b7de-4f472c859170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 03 17:40:55.616622 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.616586    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"]
Nov 03 17:40:55.617253 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.617233    6059 cpu_manager.go:414] "containerMap: removing container" podUID="3e53c74d-ceff-4b92-b579-35ef2b76de83" containerName="istio-proxy"
Nov 03 17:40:55.617324 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.617257    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e53c74d-ceff-4b92-b579-35ef2b76de83" containerName="istio-proxy"
Nov 03 17:40:55.617324 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.617290    6059 cpu_manager.go:414] "containerMap: removing container" podUID="3e53c74d-ceff-4b92-b579-35ef2b76de83" containerName="istio-proxy"
Nov 03 17:40:55.617324 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.617297    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e53c74d-ceff-4b92-b579-35ef2b76de83" containerName="istio-proxy"
Nov 03 17:40:55.617506 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.617440    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e53c74d-ceff-4b92-b579-35ef2b76de83" containerName="istio-proxy"
Nov 03 17:40:55.621269 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.621247    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"
Nov 03 17:40:55.678861 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.678765    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 17:40:55.679032 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.678870    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 17:40:55.755691 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.755478    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-adbf463d8a89\""
Nov 03 17:40:55.755691 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.755647    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-2j857\""
Nov 03 17:40:55.760873 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.760840    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"]
Nov 03 17:40:55.778013 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.777974    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-provider-credentials\") pod \"payload-processing-external-model-8657f8dff8-j9qn4\" (UID: \"44e7441c-6611-4640-9642-f67d1f3c8172\") " pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"
Nov 03 17:40:55.778200 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.778031    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44e7441c-6611-4640-9642-f67d1f3c8172-routing-overlay-volume\") pod \"payload-processing-external-model-8657f8dff8-j9qn4\" (UID: \"44e7441c-6611-4640-9642-f67d1f3c8172\") " pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"
Nov 03 17:40:55.778200 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.778051    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82fjx\" (UniqueName: \"kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-kube-api-access-82fjx\") pod \"payload-processing-external-model-8657f8dff8-j9qn4\" (UID: \"44e7441c-6611-4640-9642-f67d1f3c8172\") " pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"
Nov 03 17:40:55.778200 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.778126    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44e7441c-6611-4640-9642-f67d1f3c8172-plugins-config-volume\") pod \"payload-processing-external-model-8657f8dff8-j9qn4\" (UID: \"44e7441c-6611-4640-9642-f67d1f3c8172\") " pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"
Nov 03 17:40:55.804547 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.804514    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 17:40:55.804727 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.804556    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-03ea5a1192b1\""
Nov 03 17:40:55.804727 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.804602    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 17:40:55.879179 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.879143    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-provider-credentials\") pod \"payload-processing-external-model-8657f8dff8-j9qn4\" (UID: \"44e7441c-6611-4640-9642-f67d1f3c8172\") " pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"
Nov 03 17:40:55.879405 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.879209    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44e7441c-6611-4640-9642-f67d1f3c8172-routing-overlay-volume\") pod \"payload-processing-external-model-8657f8dff8-j9qn4\" (UID: \"44e7441c-6611-4640-9642-f67d1f3c8172\") " pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"
Nov 03 17:40:55.879405 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.879330    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82fjx\" (UniqueName: \"kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-kube-api-access-82fjx\") pod \"payload-processing-external-model-8657f8dff8-j9qn4\" (UID: \"44e7441c-6611-4640-9642-f67d1f3c8172\") " pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"
Nov 03 17:40:55.879405 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.879390    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44e7441c-6611-4640-9642-f67d1f3c8172-plugins-config-volume\") pod \"payload-processing-external-model-8657f8dff8-j9qn4\" (UID: \"44e7441c-6611-4640-9642-f67d1f3c8172\") " pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"
Nov 03 17:40:55.879944 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.879919    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44e7441c-6611-4640-9642-f67d1f3c8172-routing-overlay-volume\") pod \"payload-processing-external-model-8657f8dff8-j9qn4\" (UID: \"44e7441c-6611-4640-9642-f67d1f3c8172\") " pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"
Nov 03 17:40:55.880168 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.880149    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44e7441c-6611-4640-9642-f67d1f3c8172-plugins-config-volume\") pod \"payload-processing-external-model-8657f8dff8-j9qn4\" (UID: \"44e7441c-6611-4640-9642-f67d1f3c8172\") " pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"
Nov 03 17:40:55.881818 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.881799    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-provider-credentials\") pod \"payload-processing-external-model-8657f8dff8-j9qn4\" (UID: \"44e7441c-6611-4640-9642-f67d1f3c8172\") " pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"
Nov 03 17:40:55.955348 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.955246    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82fjx\" (UniqueName: \"kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-kube-api-access-82fjx\") pod \"payload-processing-external-model-8657f8dff8-j9qn4\" (UID: \"44e7441c-6611-4640-9642-f67d1f3c8172\") " pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"
Nov 03 17:40:55.962855 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:55.962815    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"
Nov 03 17:40:56.427572 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:56.427532    6059 patch_prober.go:29] interesting pod/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 03 17:40:56.428021 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:56.427593    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7" podUID="7d3b16ed-f9c0-4c11-b7de-4f472c859170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 03 17:40:56.477996 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:40:56.477962    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e7441c_6611_4640_9642_f67d1f3c8172.slice/crio-97854cf1eff17c4ecfa100e1cd7b545bd85153c65695ddf7bd2f280ddbae9cc8 WatchSource:0}: Error finding container 97854cf1eff17c4ecfa100e1cd7b545bd85153c65695ddf7bd2f280ddbae9cc8: Status 404 returned error can't find the container with id 97854cf1eff17c4ecfa100e1cd7b545bd85153c65695ddf7bd2f280ddbae9cc8
Nov 03 17:40:56.481628 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:56.481599    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"]
Nov 03 17:40:57.106907 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:57.106868    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4" event={"ID":"44e7441c-6611-4640-9642-f67d1f3c8172","Type":"ContainerStarted","Data":"a2b0250db61dfffcb5a5f13bb0d0a1059e7eae71c38951e5ca9eecb5e88fac97"}
Nov 03 17:40:57.107075 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:57.106925    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4" event={"ID":"44e7441c-6611-4640-9642-f67d1f3c8172","Type":"ContainerStarted","Data":"97854cf1eff17c4ecfa100e1cd7b545bd85153c65695ddf7bd2f280ddbae9cc8"}
Nov 03 17:40:57.107075 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:57.106954    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"
Nov 03 17:40:57.229140 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:57.229089    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4" podStartSLOduration=2.229075232 podStartE2EDuration="2.229075232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:57.227313139 +0000 UTC m=+1398.025025922" watchObservedRunningTime="2026-11-03 17:40:57.229075232 +0000 UTC m=+1398.026788003"
Nov 03 17:40:57.427758 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:57.427674    6059 patch_prober.go:29] interesting pod/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 03 17:40:57.427758 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:57.427735    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7" podUID="7d3b16ed-f9c0-4c11-b7de-4f472c859170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 03 17:40:57.713985 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:57.713947    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:57.918798 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:57.918765    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:58.081628 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.081534    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn"]
Nov 03 17:40:58.082390 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.082330    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e53c74d-ceff-4b92-b579-35ef2b76de83" containerName="istio-proxy"
Nov 03 17:40:58.085861 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.085840    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn"
Nov 03 17:40:58.142447 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.141766    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-5f88f673\""
Nov 03 17:40:58.142447 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.142402    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-5f88f673-dockercfg-9f57f\""
Nov 03 17:40:58.177280 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.177237    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:40:58.181740 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.181702    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edbd8e63-4b01-4d65-abd5-af5bda2a1e24-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn\" (UID: \"edbd8e63-4b01-4d65-abd5-af5bda2a1e24\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn"
Nov 03 17:40:58.181973 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.181944    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf9d6\" (UniqueName: \"kubernetes.io/projected/edbd8e63-4b01-4d65-abd5-af5bda2a1e24-kube-api-access-nf9d6\") pod \"payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn\" (UID: \"edbd8e63-4b01-4d65-abd5-af5bda2a1e24\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn"
Nov 03 17:40:58.197483 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.197445    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn"]
Nov 03 17:40:58.291420 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.289041    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf9d6\" (UniqueName: \"kubernetes.io/projected/edbd8e63-4b01-4d65-abd5-af5bda2a1e24-kube-api-access-nf9d6\") pod \"payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn\" (UID: \"edbd8e63-4b01-4d65-abd5-af5bda2a1e24\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn"
Nov 03 17:40:58.291420 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.289189    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edbd8e63-4b01-4d65-abd5-af5bda2a1e24-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn\" (UID: \"edbd8e63-4b01-4d65-abd5-af5bda2a1e24\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn"
Nov 03 17:40:58.291420 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.290379    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edbd8e63-4b01-4d65-abd5-af5bda2a1e24-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn\" (UID: \"edbd8e63-4b01-4d65-abd5-af5bda2a1e24\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn"
Nov 03 17:40:58.323388 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.312844    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl"]
Nov 03 17:40:58.323388 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.321040    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl"
Nov 03 17:40:58.338856 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.338764    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl"]
Nov 03 17:40:58.359388 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.357850    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf9d6\" (UniqueName: \"kubernetes.io/projected/edbd8e63-4b01-4d65-abd5-af5bda2a1e24-kube-api-access-nf9d6\") pod \"payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn\" (UID: \"edbd8e63-4b01-4d65-abd5-af5bda2a1e24\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn"
Nov 03 17:40:58.401093 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.401053    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn"
Nov 03 17:40:58.491437 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.491397    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96150112-b203-45a9-928a-471c023eee31-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl\" (UID: \"96150112-b203-45a9-928a-471c023eee31\") " pod="openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl"
Nov 03 17:40:58.491437 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.491436    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2jff\" (UniqueName: \"kubernetes.io/projected/96150112-b203-45a9-928a-471c023eee31-kube-api-access-z2jff\") pod \"payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl\" (UID: \"96150112-b203-45a9-928a-471c023eee31\") " pod="openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl"
Nov 03 17:40:58.546827 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.546788    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:40:58.547882 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.547853    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:40:58.548968 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.548948    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:40:58.593160 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.593070    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96150112-b203-45a9-928a-471c023eee31-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl\" (UID: \"96150112-b203-45a9-928a-471c023eee31\") " pod="openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl"
Nov 03 17:40:58.593160 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.593114    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2jff\" (UniqueName: \"kubernetes.io/projected/96150112-b203-45a9-928a-471c023eee31-kube-api-access-z2jff\") pod \"payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl\" (UID: \"96150112-b203-45a9-928a-471c023eee31\") " pod="openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl"
Nov 03 17:40:58.593891 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.593870    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96150112-b203-45a9-928a-471c023eee31-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl\" (UID: \"96150112-b203-45a9-928a-471c023eee31\") " pod="openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl"
Nov 03 17:40:58.656504 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.656470    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2jff\" (UniqueName: \"kubernetes.io/projected/96150112-b203-45a9-928a-471c023eee31-kube-api-access-z2jff\") pod \"payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl\" (UID: \"96150112-b203-45a9-928a-471c023eee31\") " pod="openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl"
Nov 03 17:40:58.799462 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.794754    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn"]
Nov 03 17:40:58.950822 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:58.950792    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl"
Nov 03 17:40:59.124528 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:59.124421    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn" event={"ID":"edbd8e63-4b01-4d65-abd5-af5bda2a1e24","Type":"ContainerStarted","Data":"051ed64693ff67de9c5905e88d8321506c58f9588b601ca8a39e88f16c4ea7a0"}
Nov 03 17:40:59.124528 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:59.124498    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn" event={"ID":"edbd8e63-4b01-4d65-abd5-af5bda2a1e24","Type":"ContainerStarted","Data":"ac9b4e02c079743dcf645b246e7b5df2143049cb5fe43447fc0fc8d7515ee8e8"}
Nov 03 17:40:59.125087 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:59.125058    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn"
Nov 03 17:40:59.250944 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:59.250871    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn" podStartSLOduration=1.250850261 podStartE2EDuration="1.250850261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:59.247239773 +0000 UTC m=+1400.044952559" watchObservedRunningTime="2026-11-03 17:40:59.250850261 +0000 UTC m=+1400.048563043"
Nov 03 17:40:59.407788 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:40:59.398928    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl"]
Nov 03 17:41:00.148458 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:00.148412    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl" event={"ID":"96150112-b203-45a9-928a-471c023eee31","Type":"ContainerStarted","Data":"fe9f415485032d8d6a129fb652094a0070e358c63b9473a102af05deb62e57a8"}
Nov 03 17:41:00.149055 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:00.149026    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl" event={"ID":"96150112-b203-45a9-928a-471c023eee31","Type":"ContainerStarted","Data":"ddb154d2fa0955ef57131e392de360d538aff125e9aa3f348525d08a40b7e8dc"}
Nov 03 17:41:00.164382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:00.150201    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl"
Nov 03 17:41:00.295381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:00.294773    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl" podStartSLOduration=2.294755685 podStartE2EDuration="2.294755685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:00.292962751 +0000 UTC m=+1401.090675532" watchObservedRunningTime="2026-11-03 17:41:00.294755685 +0000 UTC m=+1401.092468467"
Nov 03 17:41:03.622784 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:03.622746    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:05.052685 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:05.052646    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44878b2a-d6mqr"]
Nov 03 17:41:05.104226 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:05.104189    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44878b2a-d6mqr"]
Nov 03 17:41:05.336260 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:05.336175    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9edabf3d-035f-42cd-8efd-c2150fd3921a" path="/var/lib/kubelet/pods/9edabf3d-035f-42cd-8efd-c2150fd3921a/volumes"
Nov 03 17:41:06.866987 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:41:06.866941    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:41:08.118140 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:08.118110    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"
Nov 03 17:41:08.348941 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:08.348909    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:10.149864 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:10.149834    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn"
Nov 03 17:41:12.166616 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:12.166582    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl"
Nov 03 17:41:13.056389 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:13.056328    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:13.369099 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:13.369009    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:16.813811 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:16.813778    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:18.059250 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.059206    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:18.619452 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.619413    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"]
Nov 03 17:41:18.625421 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.625395    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.670877 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.670843    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"]
Nov 03 17:41:18.675150 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.675130    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.722411 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.721014    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-t4h2g\""
Nov 03 17:41:18.726763 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.726727    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"]
Nov 03 17:41:18.739255 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.739216    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.739578 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.739542    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.739709 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.739626    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.739709 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.739700    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.739842 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.739724    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.739842 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.739765    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.739842 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.739805    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.739991 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.739842    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2tgf\" (UniqueName: \"kubernetes.io/projected/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-kube-api-access-g2tgf\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.739991 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.739876    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.739991 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.739911    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.739991 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.739942    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.740200 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.740007    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.740200 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.740061    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.740200 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.740109    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.740200 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.740187    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.740437 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.740220    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t6nt\" (UniqueName: \"kubernetes.io/projected/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-kube-api-access-6t6nt\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.740437 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.740252    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.740437 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.740277    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.744682 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.744645    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"]
Nov 03 17:41:18.841715 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.841681    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.841715 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.841718    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.841985 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.841747    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.841985 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.841860    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.841985 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.841901    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2tgf\" (UniqueName: \"kubernetes.io/projected/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-kube-api-access-g2tgf\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.841985 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.841934    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.841985 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.841956    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.841985 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.841974    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.842303 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.842030    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.842303 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.842081    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.842303 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.842125    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.842303 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.842196    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.842303 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.842201    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.842303 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.842214    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.842303 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.842231    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t6nt\" (UniqueName: \"kubernetes.io/projected/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-kube-api-access-6t6nt\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.842303 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.842263    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.842303 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.842287    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.842803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.842325    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.842803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.842446    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.842803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.842506    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.842803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.842574    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.842803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.842581    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.842803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.842602    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.842803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.842735    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.843198 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.842900    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.843198 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.842931    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.843198 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.842973    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.843198 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.843192    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.844666 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.844632    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.844810 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.844789    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.844932 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.844916    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.845299 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.845282    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.910240 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.910167    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.911319 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.911296    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.911483 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.911384    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t6nt\" (UniqueName: \"kubernetes.io/projected/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-kube-api-access-6t6nt\") pod \"maas-default-gateway-openshift-default-58c6585447-h98kq\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:18.911712 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.911691    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2tgf\" (UniqueName: \"kubernetes.io/projected/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-kube-api-access-g2tgf\") pod \"maas-default-gateway-openshift-default-58c6585447-tdr2c\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.942042 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.942001    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:18.991426 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:18.991044    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:19.375373 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:41:19.375320    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0c42da_8020_48d1_9e61_7dfab0d4a65b.slice/crio-e42a294c379042500651251ce4d59a24fdad6e7b0cefc387e4c725437f59e906 WatchSource:0}: Error finding container e42a294c379042500651251ce4d59a24fdad6e7b0cefc387e4c725437f59e906: Status 404 returned error can't find the container with id e42a294c379042500651251ce4d59a24fdad6e7b0cefc387e4c725437f59e906
Nov 03 17:41:19.378021 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:19.377985    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:41:19.378111 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:19.378059    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:41:19.378111 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:19.378089    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:41:19.379428 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:19.379403    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"]
Nov 03 17:41:19.429790 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:41:19.429755    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c7a4f1d_3c17_41f8_b8a2_b97d039f2fbc.slice/crio-0826ad0c2e5c4df44973bc4d890c830db0282836a50af15c6c3119e3f181efa5 WatchSource:0}: Error finding container 0826ad0c2e5c4df44973bc4d890c830db0282836a50af15c6c3119e3f181efa5: Status 404 returned error can't find the container with id 0826ad0c2e5c4df44973bc4d890c830db0282836a50af15c6c3119e3f181efa5
Nov 03 17:41:19.430797 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:19.430773    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"]
Nov 03 17:41:19.432769 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:19.432730    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:41:19.432857 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:19.432819    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:41:19.432916 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:19.432900    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:41:20.305661 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:20.305626    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq" event={"ID":"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc","Type":"ContainerStarted","Data":"79d854461eb1c0342cd39749425b53ac99d6c825a0e245f4d35eabfefb4ce24c"}
Nov 03 17:41:20.305850 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:20.305678    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq" event={"ID":"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc","Type":"ContainerStarted","Data":"0826ad0c2e5c4df44973bc4d890c830db0282836a50af15c6c3119e3f181efa5"}
Nov 03 17:41:20.307299 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:20.307274    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c" event={"ID":"4f0c42da-8020-48d1-9e61-7dfab0d4a65b","Type":"ContainerStarted","Data":"e8f193522683ef5e8fe2dca93fac72da4f903f4eac63caf3cdd51b4e3c62698f"}
Nov 03 17:41:20.307419 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:20.307314    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c" event={"ID":"4f0c42da-8020-48d1-9e61-7dfab0d4a65b","Type":"ContainerStarted","Data":"e42a294c379042500651251ce4d59a24fdad6e7b0cefc387e4c725437f59e906"}
Nov 03 17:41:20.505958 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:20.505903    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq" podStartSLOduration=2.5058860530000002 podStartE2EDuration="2.505886053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:20.501694806 +0000 UTC m=+1421.299407590" watchObservedRunningTime="2026-11-03 17:41:20.505886053 +0000 UTC m=+1421.303598834"
Nov 03 17:41:20.949380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:20.948775    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:20.969383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:20.964503    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-tdr2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 17:41:20.969383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:20.964574    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c" podUID="4f0c42da-8020-48d1-9e61-7dfab0d4a65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 17:41:20.992137 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:20.992095    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:20.993703 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:20.993674    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-h98kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 17:41:20.993829 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:20.993724    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq" podUID="1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 17:41:21.055381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:21.054916    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c" podStartSLOduration=3.054895516 podStartE2EDuration="3.054895516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:20.649646029 +0000 UTC m=+1421.447358810" watchObservedRunningTime="2026-11-03 17:41:21.054895516 +0000 UTC m=+1421.852608297"
Nov 03 17:41:21.084381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:21.083647    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3222aef-twttc"]
Nov 03 17:41:21.088388 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:21.088012    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3222aef-twttc"]
Nov 03 17:41:21.385499 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:21.385460    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe504ece-3403-4332-9029-12f5b7ca4671" path="/var/lib/kubelet/pods/fe504ece-3403-4332-9029-12f5b7ca4671/volumes"
Nov 03 17:41:21.951384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:21.945760    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-tdr2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 17:41:21.951384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:21.945824    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c" podUID="4f0c42da-8020-48d1-9e61-7dfab0d4a65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 17:41:21.992141 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:21.992102    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-h98kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 17:41:21.992301 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:21.992171    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq" podUID="1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 17:41:22.719933 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:22.719900    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:22.945384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:22.942821    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-tdr2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 17:41:22.945384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:22.942904    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c" podUID="4f0c42da-8020-48d1-9e61-7dfab0d4a65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 17:41:22.992195 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:22.992103    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-h98kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 17:41:22.992195 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:22.992172    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq" podUID="1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 17:41:23.943396 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:23.942805    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-tdr2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 17:41:23.943396 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:23.942888    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c" podUID="4f0c42da-8020-48d1-9e61-7dfab0d4a65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 17:41:23.995377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:23.992752    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-h98kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 17:41:23.995377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:23.992833    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq" podUID="1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 17:41:24.953382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:24.946522    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-tdr2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 17:41:24.953382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:24.946594    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c" podUID="4f0c42da-8020-48d1-9e61-7dfab0d4a65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 17:41:24.991923 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:24.991884    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-h98kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 17:41:24.992096 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:24.991958    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq" podUID="1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 17:41:25.942731 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:25.942684    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-tdr2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 17:41:25.943241 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:25.942751    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c" podUID="4f0c42da-8020-48d1-9e61-7dfab0d4a65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 17:41:25.991728 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:25.991693    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-h98kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 17:41:25.991896 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:25.991748    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq" podUID="1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 17:41:26.942475 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:26.942436    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-tdr2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 17:41:26.942680 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:26.942504    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c" podUID="4f0c42da-8020-48d1-9e61-7dfab0d4a65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 17:41:26.991655 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:26.991610    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-h98kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 17:41:26.992026 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:26.991684    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq" podUID="1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 17:41:27.438118 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:27.438083    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:27.942990 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:27.942938    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-tdr2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 17:41:27.943220 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:27.943023    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c" podUID="4f0c42da-8020-48d1-9e61-7dfab0d4a65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 17:41:27.991590 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:27.991554    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-h98kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 17:41:27.991761 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:27.991622    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq" podUID="1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 17:41:29.283436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:29.283395    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:29.284609 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:29.284577    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:29.285494 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:29.285470    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:41:29.293111 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:29.293079    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:29.293555 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:29.293530    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:29.294719 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:29.294694    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:41:31.817043 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:31.816999    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:32.035067 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:32.035030    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:32.272886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:32.272786    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:33.775484 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:33.775449    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"]
Nov 03 17:41:33.783891 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:33.783860    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:33.846878 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:33.846839    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:33.847156 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:33.847129    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32988d90-426a-4a7b-b034-a4685db935b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:33.847332 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:33.847310    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6pgw\" (UniqueName: \"kubernetes.io/projected/32988d90-426a-4a7b-b034-a4685db935b8-kube-api-access-q6pgw\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:33.847509 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:33.847489    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32988d90-426a-4a7b-b034-a4685db935b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:33.879780 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:33.847645    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:33.902853 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:33.902810    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:33.903176 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:33.903142    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:33.903936 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:33.903223    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:33.903936 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:33.903285    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32988d90-426a-4a7b-b034-a4685db935b8-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:33.916315 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:33.916278    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"]
Nov 03 17:41:34.004568 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.004527    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32988d90-426a-4a7b-b034-a4685db935b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:34.004568 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.004571    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:34.004788 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.004626    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:34.004788 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.004655    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:34.004788 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.004684    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:34.004788 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.004714    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32988d90-426a-4a7b-b034-a4685db935b8-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:34.004788 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.004782    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:34.004948 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.004801    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32988d90-426a-4a7b-b034-a4685db935b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:34.004948 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.004845    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6pgw\" (UniqueName: \"kubernetes.io/projected/32988d90-426a-4a7b-b034-a4685db935b8-kube-api-access-q6pgw\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:34.005421 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.005387    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:34.005840 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.005816    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:34.005951 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.005837    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:34.006205 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.006178    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:34.006291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.006184    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32988d90-426a-4a7b-b034-a4685db935b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:34.007523 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.007500    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:34.007766 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.007749    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32988d90-426a-4a7b-b034-a4685db935b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:34.074426 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.074174    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32988d90-426a-4a7b-b034-a4685db935b8-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:34.083908 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.083794    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6pgw\" (UniqueName: \"kubernetes.io/projected/32988d90-426a-4a7b-b034-a4685db935b8-kube-api-access-q6pgw\") pod \"maas-default-gateway-openshift-default-58c6585447-8kpl6\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:34.105935 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.105901    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:34.591998 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.588138    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"]
Nov 03 17:41:34.591998 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.589240    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:41:34.591998 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.589324    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:41:34.591998 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:34.589390    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:41:35.530382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:35.530324    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6" event={"ID":"32988d90-426a-4a7b-b034-a4685db935b8","Type":"ContainerStarted","Data":"fc2a5b83871e84ce9f9c7c75b9e50eb323ccc38e6f041a781aa94eeae78d98fd"}
Nov 03 17:41:35.531125 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:35.531094    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6" event={"ID":"32988d90-426a-4a7b-b034-a4685db935b8","Type":"ContainerStarted","Data":"2e80d98c8b00eda65ec88dcb26d039e24763827960b2765c6b0be1cccdbfc769"}
Nov 03 17:41:35.716592 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:35.715489    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6" podStartSLOduration=2.715466756 podStartE2EDuration="2.715466756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:35.711485255 +0000 UTC m=+1436.509198036" watchObservedRunningTime="2026-11-03 17:41:35.715466756 +0000 UTC m=+1436.513179539"
Nov 03 17:41:36.106648 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:36.106602    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:36.108279 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:36.108252    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8kpl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 17:41:36.108481 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:36.108303    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6" podUID="32988d90-426a-4a7b-b034-a4685db935b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 17:41:36.755887 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:36.755845    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:36.891393 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:41:36.891337    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:41:37.107456 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:37.107332    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8kpl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 17:41:37.107456 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:37.107420    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6" podUID="32988d90-426a-4a7b-b034-a4685db935b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 17:41:38.107212 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:38.107166    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8kpl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 17:41:38.107723 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:38.107230    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6" podUID="32988d90-426a-4a7b-b034-a4685db935b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 17:41:38.373679 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:38.373594    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:39.106730 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:39.106690    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8kpl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 17:41:39.106924 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:39.106764    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6" podUID="32988d90-426a-4a7b-b034-a4685db935b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 17:41:40.106619 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:40.106573    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8kpl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 17:41:40.107040 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:40.106636    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6" podUID="32988d90-426a-4a7b-b034-a4685db935b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 17:41:41.106674 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:41.106634    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8kpl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 17:41:41.107110 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:41.106694    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6" podUID="32988d90-426a-4a7b-b034-a4685db935b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 17:41:41.235564 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:41.235525    6059 scope.go:122] "RemoveContainer" containerID="a25b3ee0d142f2e267d68e42cbea5fc9050eec61a40709f089708435eef862bd"
Nov 03 17:41:41.248639 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:41.248612    6059 scope.go:122] "RemoveContainer" containerID="4f7bb50953c2fe3ee0585c54a4d24c7d2420c6b8b2f7416dead9d46826e02aa4"
Nov 03 17:41:41.259952 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:41.259930    6059 scope.go:122] "RemoveContainer" containerID="fc04c4dedefee596304da30f6c528ff3eda2226f5cad6c9dd176516b63b284c7"
Nov 03 17:41:41.270707 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:41.270677    6059 scope.go:122] "RemoveContainer" containerID="d62caa29bf94f9f6deb054a340b9d3e496a9f4a313909c511a93afb353765090"
Nov 03 17:41:41.794528 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:41.794494    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:42.107292 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:42.107205    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8kpl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 17:41:42.107292 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:42.107269    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6" podUID="32988d90-426a-4a7b-b034-a4685db935b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 17:41:43.106797 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:43.106749    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8kpl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 17:41:43.106998 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:43.106847    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6" podUID="32988d90-426a-4a7b-b034-a4685db935b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 17:41:44.679798 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:44.679763    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:44.681432 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:44.681352    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:44.682171 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:44.682151    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 17:41:46.526097 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:46.526063    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:46.726604 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:46.726556    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:46.922097 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:46.921999    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:48.055501 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:48.055466    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q"]
Nov 03 17:41:48.106708 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:48.106670    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df0ea99-kv54q"]
Nov 03 17:41:49.337542 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:49.337506    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="596510a6-5e78-40ea-8d68-11104a26aad0" path="/var/lib/kubelet/pods/596510a6-5e78-40ea-8d68-11104a26aad0/volumes"
Nov 03 17:41:51.242117 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:51.242084    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:52.059801 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:52.059758    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf"]
Nov 03 17:41:52.105923 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:52.105887    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7663366-8njmf"]
Nov 03 17:41:52.623845 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:52.623814    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/ai-tenant-e2e-extmod-b-8c9b0f_payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx_e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853/payload-processing/0.log"
Nov 03 17:41:53.346770 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:53.346733    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/models-as-a-service_payload-processing-external-model-8657f8dff8-j9qn4_44e7441c-6611-4640-9642-f67d1f3c8172/payload-processing/0.log"
Nov 03 17:41:53.360859 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:53.360820    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8319bee0-f2bf-488e-bd77-7caaf56be17a" path="/var/lib/kubelet/pods/8319bee0-f2bf-488e-bd77-7caaf56be17a/volumes"
Nov 03 17:41:55.644921 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:55.644883    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:55.845381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:55.829596    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:41:56.087350 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:41:56.087306    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:00.732663 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:00.732625    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:00.814828 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:00.814736    6059 generic.go:356] "Generic (PLEG): container finished" podID="6b11bcfe-5150-4eee-95e9-ee213aea1e8e" containerID="03b25e537b3bc4296111e37a811e5ddd0c7188a88a34cc744fe549af8516d7a0" exitCode=0
Nov 03 17:42:00.815007 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:00.814820    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2" event={"ID":"6b11bcfe-5150-4eee-95e9-ee213aea1e8e","Type":"ContainerDied","Data":"03b25e537b3bc4296111e37a811e5ddd0c7188a88a34cc744fe549af8516d7a0"}
Nov 03 17:42:00.815406 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:00.815378    6059 scope.go:122] "RemoveContainer" containerID="03b25e537b3bc4296111e37a811e5ddd0c7188a88a34cc744fe549af8516d7a0"
Nov 03 17:42:00.817428 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:00.817393    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:42:00.817566 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:00.817500    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:42:00.817566 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:00.817557    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:42:01.021378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.018336    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:01.494219 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.494185    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"]
Nov 03 17:42:01.498910 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.498886    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"
Nov 03 17:42:01.553870 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.553821    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"agc-cred-c899977d32bf\""
Nov 03 17:42:01.554101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.553880    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"payload-processing-external-model-e2e--58d254fa-dockercfg-xb9t5\""
Nov 03 17:42:01.554870 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.554844    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"kube-root-ca.crt\""
Nov 03 17:42:01.555672 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.555487    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-029ca0\""
Nov 03 17:42:01.560550 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.560526    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"]
Nov 03 17:42:01.607503 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.607469    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"openshift-service-ca.crt\""
Nov 03 17:42:01.607503 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.607487    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"routing-overlay\""
Nov 03 17:42:01.662736 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.662692    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e433f04f-2bfe-4844-baa8-82f127259f71-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj\" (UID: \"e433f04f-2bfe-4844-baa8-82f127259f71\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"
Nov 03 17:42:01.662935 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.662758    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e433f04f-2bfe-4844-baa8-82f127259f71-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj\" (UID: \"e433f04f-2bfe-4844-baa8-82f127259f71\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"
Nov 03 17:42:01.663028 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.663001    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj\" (UID: \"e433f04f-2bfe-4844-baa8-82f127259f71\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"
Nov 03 17:42:01.663081 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.663037    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28v4l\" (UniqueName: \"kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-kube-api-access-28v4l\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj\" (UID: \"e433f04f-2bfe-4844-baa8-82f127259f71\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"
Nov 03 17:42:01.764538 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.764432    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj\" (UID: \"e433f04f-2bfe-4844-baa8-82f127259f71\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"
Nov 03 17:42:01.764538 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.764470    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28v4l\" (UniqueName: \"kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-kube-api-access-28v4l\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj\" (UID: \"e433f04f-2bfe-4844-baa8-82f127259f71\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"
Nov 03 17:42:01.764538 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.764501    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e433f04f-2bfe-4844-baa8-82f127259f71-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj\" (UID: \"e433f04f-2bfe-4844-baa8-82f127259f71\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"
Nov 03 17:42:01.765185 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.764677    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e433f04f-2bfe-4844-baa8-82f127259f71-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj\" (UID: \"e433f04f-2bfe-4844-baa8-82f127259f71\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"
Nov 03 17:42:01.765260 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.765196    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e433f04f-2bfe-4844-baa8-82f127259f71-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj\" (UID: \"e433f04f-2bfe-4844-baa8-82f127259f71\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"
Nov 03 17:42:01.765448 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.765423    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e433f04f-2bfe-4844-baa8-82f127259f71-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj\" (UID: \"e433f04f-2bfe-4844-baa8-82f127259f71\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"
Nov 03 17:42:01.766993 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.766967    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj\" (UID: \"e433f04f-2bfe-4844-baa8-82f127259f71\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"
Nov 03 17:42:01.825089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.825047    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2" event={"ID":"6b11bcfe-5150-4eee-95e9-ee213aea1e8e","Type":"ContainerStarted","Data":"4408b2e1e836a0f5893d2e423dcef7e17919e11363fe9d987340558a97f27d8d"}
Nov 03 17:42:01.828150 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:01.828122    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28v4l\" (UniqueName: \"kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-kube-api-access-28v4l\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj\" (UID: \"e433f04f-2bfe-4844-baa8-82f127259f71\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"
Nov 03 17:42:02.113440 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:02.113315    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"
Nov 03 17:42:02.257207 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:02.255204    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:42:02.257207 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:02.256990    6059 patch_prober.go:29] interesting pod/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 17:42:02.257207 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:02.257046    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2" podUID="6b11bcfe-5150-4eee-95e9-ee213aea1e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 17:42:02.557403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:02.557337    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"]
Nov 03 17:42:02.903440 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:02.902877    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj" event={"ID":"e433f04f-2bfe-4844-baa8-82f127259f71","Type":"ContainerStarted","Data":"6a6560468f162787b06cb87bf00ca274adb56eedeaaa49e84f62009afc76acd6"}
Nov 03 17:42:02.903440 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:02.902937    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj" event={"ID":"e433f04f-2bfe-4844-baa8-82f127259f71","Type":"ContainerStarted","Data":"30e9dded0fd5d80912dee5e2a1e8e1ed9e3580b185a188e09d1b11c3cf48b7dd"}
Nov 03 17:42:02.903440 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:02.902989    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"
Nov 03 17:42:02.997159 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:02.997129    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-029ca0/agc-cred-c899977d32bf: secret "agc-cred-c899977d32bf" not found
Nov 03 17:42:02.997159 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:02.997155    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj: secret "agc-cred-c899977d32bf" not found
Nov 03 17:42:02.997470 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:02.997224    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-provider-credentials podName:e433f04f-2bfe-4844-baa8-82f127259f71 nodeName:}" failed. No retries permitted until 2026-11-03 17:42:03.497205435 +0000 UTC m=+1464.294918201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj" (UID: "e433f04f-2bfe-4844-baa8-82f127259f71") : secret "agc-cred-c899977d32bf" not found
Nov 03 17:42:03.049382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:03.037081    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj" podStartSLOduration=2.037063927 podStartE2EDuration="2.037063927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:42:03.035111892 +0000 UTC m=+1463.832824675" watchObservedRunningTime="2026-11-03 17:42:03.037063927 +0000 UTC m=+1463.834776720"
Nov 03 17:42:03.251380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:03.245790    6059 patch_prober.go:29] interesting pod/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 17:42:03.251380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:03.245857    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2" podUID="6b11bcfe-5150-4eee-95e9-ee213aea1e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 17:42:03.363414 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:03.362311    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:03.510443 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:03.505520    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-029ca0/agc-cred-c899977d32bf: secret "agc-cred-c899977d32bf" not found
Nov 03 17:42:03.510443 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:03.505555    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj: secret "agc-cred-c899977d32bf" not found
Nov 03 17:42:03.510443 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:03.505619    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-provider-credentials podName:e433f04f-2bfe-4844-baa8-82f127259f71 nodeName:}" failed. No retries permitted until 2026-11-03 17:42:04.505600075 +0000 UTC m=+1465.303312851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj" (UID: "e433f04f-2bfe-4844-baa8-82f127259f71") : secret "agc-cred-c899977d32bf" not found
Nov 03 17:42:03.573388 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:03.565547    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:04.069399 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:04.069317    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf"]
Nov 03 17:42:04.073272 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:04.073243    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d813936c-cm7mf"]
Nov 03 17:42:04.253391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:04.252841    6059 patch_prober.go:29] interesting pod/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 17:42:04.253391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:04.252909    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2" podUID="6b11bcfe-5150-4eee-95e9-ee213aea1e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 17:42:04.519826 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:04.519787    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-029ca0/agc-cred-c899977d32bf: secret "agc-cred-c899977d32bf" not found
Nov 03 17:42:04.519826 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:04.519818    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj: secret "agc-cred-c899977d32bf" not found
Nov 03 17:42:04.520050 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:04.519875    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-provider-credentials podName:e433f04f-2bfe-4844-baa8-82f127259f71 nodeName:}" failed. No retries permitted until 2026-11-03 17:42:06.519860122 +0000 UTC m=+1467.317572894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj" (UID: "e433f04f-2bfe-4844-baa8-82f127259f71") : secret "agc-cred-c899977d32bf" not found
Nov 03 17:42:04.758091 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:04.758052    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"]
Nov 03 17:42:04.951081 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:04.950965    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:05.028465 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:05.028428    6059 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-plugins-e2e-extmod-a-029ca0: configmap "payload-processing-external-model-plugins-e2e-extmod-a-029ca0" not found
Nov 03 17:42:05.028772 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:05.028516    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e433f04f-2bfe-4844-baa8-82f127259f71-plugins-config-volume podName:e433f04f-2bfe-4844-baa8-82f127259f71 nodeName:}" failed. No retries permitted until 2026-11-03 17:42:05.528496924 +0000 UTC m=+1466.326209697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e433f04f-2bfe-4844-baa8-82f127259f71-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj" (UID: "e433f04f-2bfe-4844-baa8-82f127259f71") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-029ca0" not found
Nov 03 17:42:05.236562 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:05.236032    6059 patch_prober.go:29] interesting pod/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 17:42:05.236562 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:05.236099    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2" podUID="6b11bcfe-5150-4eee-95e9-ee213aea1e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 17:42:05.386381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:05.386006    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95" path="/var/lib/kubelet/pods/76db6a7a-3db3-4c2f-9a0d-16e2ab3e5d95/volumes"
Nov 03 17:42:05.550236 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:05.549722    6059 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-plugins-e2e-extmod-a-029ca0: configmap "payload-processing-external-model-plugins-e2e-extmod-a-029ca0" not found
Nov 03 17:42:05.550236 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:05.549814    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e433f04f-2bfe-4844-baa8-82f127259f71-plugins-config-volume podName:e433f04f-2bfe-4844-baa8-82f127259f71 nodeName:}" failed. No retries permitted until 2026-11-03 17:42:06.549794124 +0000 UTC m=+1467.347506901 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e433f04f-2bfe-4844-baa8-82f127259f71-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj" (UID: "e433f04f-2bfe-4844-baa8-82f127259f71") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-029ca0" not found
Nov 03 17:42:05.672179 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:05.672131    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:06.238382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:06.237747    6059 patch_prober.go:29] interesting pod/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 17:42:06.238382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:06.237812    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2" podUID="6b11bcfe-5150-4eee-95e9-ee213aea1e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 17:42:06.572073 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:06.571992    6059 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-plugins-e2e-extmod-a-029ca0: configmap "payload-processing-external-model-plugins-e2e-extmod-a-029ca0" not found
Nov 03 17:42:06.572212 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:06.572075    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e433f04f-2bfe-4844-baa8-82f127259f71-plugins-config-volume podName:e433f04f-2bfe-4844-baa8-82f127259f71 nodeName:}" failed. No retries permitted until 2026-11-03 17:42:08.572056038 +0000 UTC m=+1469.369768809 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e433f04f-2bfe-4844-baa8-82f127259f71-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj" (UID: "e433f04f-2bfe-4844-baa8-82f127259f71") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-029ca0" not found
Nov 03 17:42:06.572212 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:06.572132    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-029ca0/agc-cred-c899977d32bf: secret "agc-cred-c899977d32bf" not found
Nov 03 17:42:06.572212 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:06.572151    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj: secret "agc-cred-c899977d32bf" not found
Nov 03 17:42:06.572212 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:06.572210    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-provider-credentials podName:e433f04f-2bfe-4844-baa8-82f127259f71 nodeName:}" failed. No retries permitted until 2026-11-03 17:42:10.57219863 +0000 UTC m=+1471.369911406 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj" (UID: "e433f04f-2bfe-4844-baa8-82f127259f71") : secret "agc-cred-c899977d32bf" not found
Nov 03 17:42:06.884799 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:06.884692    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:42:07.017881 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:07.017845    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:42:07.250060 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:07.250023    6059 patch_prober.go:29] interesting pod/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 17:42:07.250694 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:07.250656    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2" podUID="6b11bcfe-5150-4eee-95e9-ee213aea1e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 17:42:08.238690 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:08.238656    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:42:08.239913 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:08.239890    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:42:08.602424 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:08.602298    6059 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-plugins-e2e-extmod-a-029ca0: configmap "payload-processing-external-model-plugins-e2e-extmod-a-029ca0" not found
Nov 03 17:42:08.602424 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:08.602395    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e433f04f-2bfe-4844-baa8-82f127259f71-plugins-config-volume podName:e433f04f-2bfe-4844-baa8-82f127259f71 nodeName:}" failed. No retries permitted until 2026-11-03 17:42:12.602377766 +0000 UTC m=+1473.400090530 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e433f04f-2bfe-4844-baa8-82f127259f71-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj" (UID: "e433f04f-2bfe-4844-baa8-82f127259f71") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-029ca0" not found
Nov 03 17:42:09.224776 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:09.224736    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:09.396070 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:09.396033    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:10.372504 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:10.372467    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:10.633089 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:10.633004    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-029ca0/agc-cred-c899977d32bf: secret "agc-cred-c899977d32bf" not found
Nov 03 17:42:10.633089 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:10.633029    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj: secret "agc-cred-c899977d32bf" not found
Nov 03 17:42:10.633089 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:10.633080    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-provider-credentials podName:e433f04f-2bfe-4844-baa8-82f127259f71 nodeName:}" failed. No retries permitted until 2026-11-03 17:42:18.633064569 +0000 UTC m=+1479.430777341 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj" (UID: "e433f04f-2bfe-4844-baa8-82f127259f71") : secret "agc-cred-c899977d32bf" not found
Nov 03 17:42:10.929196 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:10.929084    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj" podUID="e433f04f-2bfe-4844-baa8-82f127259f71" containerName="payload-processing" containerID="cri-o://6a6560468f162787b06cb87bf00ca274adb56eedeaaa49e84f62009afc76acd6" gracePeriod=25
Nov 03 17:42:11.011423 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.011389    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"]
Nov 03 17:42:11.015834 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.015813    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"
Nov 03 17:42:11.126824 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.126756    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"]
Nov 03 17:42:11.148068 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.148034    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-routing-overlay-volume\") pod \"payload-processing-external-model-68478f785f-mnkbn\" (UID: \"09ca8b69-3b26-4d5f-97b7-ad0646e5556a\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"
Nov 03 17:42:11.148068 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.148069    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-provider-credentials\") pod \"payload-processing-external-model-68478f785f-mnkbn\" (UID: \"09ca8b69-3b26-4d5f-97b7-ad0646e5556a\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"
Nov 03 17:42:11.148291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.148152    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t8f4\" (UniqueName: \"kubernetes.io/projected/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-kube-api-access-2t8f4\") pod \"payload-processing-external-model-68478f785f-mnkbn\" (UID: \"09ca8b69-3b26-4d5f-97b7-ad0646e5556a\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"
Nov 03 17:42:11.148291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.148203    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-plugins-config-volume\") pod \"payload-processing-external-model-68478f785f-mnkbn\" (UID: \"09ca8b69-3b26-4d5f-97b7-ad0646e5556a\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"
Nov 03 17:42:11.238934 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.238908    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"
Nov 03 17:42:11.249836 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.249798    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-routing-overlay-volume\") pod \"payload-processing-external-model-68478f785f-mnkbn\" (UID: \"09ca8b69-3b26-4d5f-97b7-ad0646e5556a\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"
Nov 03 17:42:11.249836 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.249838    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-provider-credentials\") pod \"payload-processing-external-model-68478f785f-mnkbn\" (UID: \"09ca8b69-3b26-4d5f-97b7-ad0646e5556a\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"
Nov 03 17:42:11.250097 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.249876    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t8f4\" (UniqueName: \"kubernetes.io/projected/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-kube-api-access-2t8f4\") pod \"payload-processing-external-model-68478f785f-mnkbn\" (UID: \"09ca8b69-3b26-4d5f-97b7-ad0646e5556a\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"
Nov 03 17:42:11.250097 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.249905    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-plugins-config-volume\") pod \"payload-processing-external-model-68478f785f-mnkbn\" (UID: \"09ca8b69-3b26-4d5f-97b7-ad0646e5556a\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"
Nov 03 17:42:11.250572 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.250547    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-routing-overlay-volume\") pod \"payload-processing-external-model-68478f785f-mnkbn\" (UID: \"09ca8b69-3b26-4d5f-97b7-ad0646e5556a\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"
Nov 03 17:42:11.250706 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.250686    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-plugins-config-volume\") pod \"payload-processing-external-model-68478f785f-mnkbn\" (UID: \"09ca8b69-3b26-4d5f-97b7-ad0646e5556a\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"
Nov 03 17:42:11.252476 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.252449    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-provider-credentials\") pod \"payload-processing-external-model-68478f785f-mnkbn\" (UID: \"09ca8b69-3b26-4d5f-97b7-ad0646e5556a\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"
Nov 03 17:42:11.313308 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.313279    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t8f4\" (UniqueName: \"kubernetes.io/projected/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-kube-api-access-2t8f4\") pod \"payload-processing-external-model-68478f785f-mnkbn\" (UID: \"09ca8b69-3b26-4d5f-97b7-ad0646e5556a\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"
Nov 03 17:42:11.345770 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.345743    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"
Nov 03 17:42:11.351258 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.351235    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-provider-credentials\") pod \"e433f04f-2bfe-4844-baa8-82f127259f71\" (UID: \"e433f04f-2bfe-4844-baa8-82f127259f71\") "
Nov 03 17:42:11.351334 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.351312    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-kube-api-access-28v4l\" (UniqueName: \"kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-kube-api-access-28v4l\") pod \"e433f04f-2bfe-4844-baa8-82f127259f71\" (UID: \"e433f04f-2bfe-4844-baa8-82f127259f71\") "
Nov 03 17:42:11.351549 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.351533    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e433f04f-2bfe-4844-baa8-82f127259f71-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e433f04f-2bfe-4844-baa8-82f127259f71-routing-overlay-volume\") pod \"e433f04f-2bfe-4844-baa8-82f127259f71\" (UID: \"e433f04f-2bfe-4844-baa8-82f127259f71\") "
Nov 03 17:42:11.351683 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.351666    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e433f04f-2bfe-4844-baa8-82f127259f71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e433f04f-2bfe-4844-baa8-82f127259f71-plugins-config-volume\") pod \"e433f04f-2bfe-4844-baa8-82f127259f71\" (UID: \"e433f04f-2bfe-4844-baa8-82f127259f71\") "
Nov 03 17:42:11.351876 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.351828    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e433f04f-2bfe-4844-baa8-82f127259f71-routing-overlay-volume" pod "e433f04f-2bfe-4844-baa8-82f127259f71" (UID: "e433f04f-2bfe-4844-baa8-82f127259f71"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:42:11.352072 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.352048    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e433f04f-2bfe-4844-baa8-82f127259f71-plugins-config-volume" pod "e433f04f-2bfe-4844-baa8-82f127259f71" (UID: "e433f04f-2bfe-4844-baa8-82f127259f71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:42:11.352216 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.352193    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e433f04f-2bfe-4844-baa8-82f127259f71-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:42:11.352216 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.352216    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e433f04f-2bfe-4844-baa8-82f127259f71-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:42:11.353446 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.353428    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-provider-credentials" pod "e433f04f-2bfe-4844-baa8-82f127259f71" (UID: "e433f04f-2bfe-4844-baa8-82f127259f71"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:42:11.353762 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.353739    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-kube-api-access-28v4l" pod "e433f04f-2bfe-4844-baa8-82f127259f71" (UID: "e433f04f-2bfe-4844-baa8-82f127259f71"). InnerVolumeSpecName "kube-api-access-28v4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:42:11.454690 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.454480    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:42:11.454690 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.454524    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28v4l\" (UniqueName: \"kubernetes.io/projected/e433f04f-2bfe-4844-baa8-82f127259f71-kube-api-access-28v4l\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:42:11.729676 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:42:11.729642    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ca8b69_3b26_4d5f_97b7_ad0646e5556a.slice/crio-08f5088b0fd1eb4abadba133e403561fff658c6a98cf8e16a63eed4296a783d7 WatchSource:0}: Error finding container 08f5088b0fd1eb4abadba133e403561fff658c6a98cf8e16a63eed4296a783d7: Status 404 returned error can't find the container with id 08f5088b0fd1eb4abadba133e403561fff658c6a98cf8e16a63eed4296a783d7
Nov 03 17:42:11.734130 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:11.734101    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"]
Nov 03 17:42:12.005060 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:12.004960    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn" event={"ID":"09ca8b69-3b26-4d5f-97b7-ad0646e5556a","Type":"ContainerStarted","Data":"db04db97ee12ed84a6a1bfc0d1f26dfdd8e546253f86383e59e3fe52435c3e45"}
Nov 03 17:42:12.005238 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:12.005207    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn" event={"ID":"09ca8b69-3b26-4d5f-97b7-ad0646e5556a","Type":"ContainerStarted","Data":"08f5088b0fd1eb4abadba133e403561fff658c6a98cf8e16a63eed4296a783d7"}
Nov 03 17:42:12.005306 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:12.005253    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"
Nov 03 17:42:12.007412 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:12.007385    6059 generic.go:356] "Generic (PLEG): container finished" podID="e433f04f-2bfe-4844-baa8-82f127259f71" containerID="6a6560468f162787b06cb87bf00ca274adb56eedeaaa49e84f62009afc76acd6" exitCode=0
Nov 03 17:42:12.007550 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:12.007420    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"
Nov 03 17:42:12.007550 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:12.007446    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj" event={"ID":"e433f04f-2bfe-4844-baa8-82f127259f71","Type":"ContainerDied","Data":"6a6560468f162787b06cb87bf00ca274adb56eedeaaa49e84f62009afc76acd6"}
Nov 03 17:42:12.007550 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:12.007501    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj" event={"ID":"e433f04f-2bfe-4844-baa8-82f127259f71","Type":"ContainerDied","Data":"30e9dded0fd5d80912dee5e2a1e8e1ed9e3580b185a188e09d1b11c3cf48b7dd"}
Nov 03 17:42:12.007550 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:12.007533    6059 scope.go:122] "RemoveContainer" containerID="6a6560468f162787b06cb87bf00ca274adb56eedeaaa49e84f62009afc76acd6"
Nov 03 17:42:12.018479 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:12.018443    6059 scope.go:122] "RemoveContainer" containerID="6a6560468f162787b06cb87bf00ca274adb56eedeaaa49e84f62009afc76acd6"
Nov 03 17:42:12.018831 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:12.018809    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a6560468f162787b06cb87bf00ca274adb56eedeaaa49e84f62009afc76acd6\": container with ID starting with 6a6560468f162787b06cb87bf00ca274adb56eedeaaa49e84f62009afc76acd6 not found: ID does not exist" containerID="6a6560468f162787b06cb87bf00ca274adb56eedeaaa49e84f62009afc76acd6"
Nov 03 17:42:12.018908 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:12.018840    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a6560468f162787b06cb87bf00ca274adb56eedeaaa49e84f62009afc76acd6"} err="failed to get container status \"6a6560468f162787b06cb87bf00ca274adb56eedeaaa49e84f62009afc76acd6\": rpc error: code = NotFound desc = could not find container \"6a6560468f162787b06cb87bf00ca274adb56eedeaaa49e84f62009afc76acd6\": container with ID starting with 6a6560468f162787b06cb87bf00ca274adb56eedeaaa49e84f62009afc76acd6 not found: ID does not exist"
Nov 03 17:42:12.133569 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:12.132780    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn" podStartSLOduration=2.132758388 podStartE2EDuration="2.132758388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:42:12.125212428 +0000 UTC m=+1472.922925209" watchObservedRunningTime="2026-11-03 17:42:12.132758388 +0000 UTC m=+1472.930471171"
Nov 03 17:42:12.246963 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:12.246921    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"]
Nov 03 17:42:12.258216 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:12.258132    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-d8fd48klj"]
Nov 03 17:42:13.336829 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:13.336795    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e433f04f-2bfe-4844-baa8-82f127259f71" path="/var/lib/kubelet/pods/e433f04f-2bfe-4844-baa8-82f127259f71/volumes"
Nov 03 17:42:14.939530 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:14.939498    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:18.321043 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:18.320965    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:18.521329 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:18.521295    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:18.781069 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:18.781034    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:19.477820 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:19.477775    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:19.497093 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:19.497060    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"]
Nov 03 17:42:20.703552 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:20.703514    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:21.054577 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:21.054541    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk"]
Nov 03 17:42:21.105470 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:21.105421    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e36c42c5-hwnbk"]
Nov 03 17:42:21.336450 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:21.336338    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3457b5b-4c9b-457a-9275-8a10d710da0b" path="/var/lib/kubelet/pods/b3457b5b-4c9b-457a-9275-8a10d710da0b/volumes"
Nov 03 17:42:22.488769 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:22.488724    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:22.669020 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:22.668935    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:23.020568 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:23.020502    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn" podUID="09ca8b69-3b26-4d5f-97b7-ad0646e5556a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 17:42:24.525686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:24.525652    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:29.037328 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:29.037290    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:32.003510 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:32.003457    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl"]
Nov 03 17:42:32.126379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:32.126319    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl"]
Nov 03 17:42:32.133592 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:32.133553    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c367ba5-775zl"]
Nov 03 17:42:32.171002 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:32.170443    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn"]
Nov 03 17:42:32.509536 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:32.441051    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:32.645547 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:32.645509    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:33.018516 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:33.018473    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn" podUID="09ca8b69-3b26-4d5f-97b7-ad0646e5556a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 17:42:33.384079 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:33.383990    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181" path="/var/lib/kubelet/pods/84c4e55c-ceba-4fa0-9c41-1bbb4bfd4181/volumes"
Nov 03 17:42:34.518454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:34.518387    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w" podUID="6120f11d-14c8-4666-a201-469a5bca37de" containerName="main" containerID="cri-o://a380062a162c02aece834579a21a76edab1d371f891cce90f4d0f1657a7aa76d" gracePeriod=45
Nov 03 17:42:34.794860 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:34.794832    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:42:34.832681 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:34.832623    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:34.842728 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:34.842698    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6120f11d-14c8-4666-a201-469a5bca37de-kube-api-access-bs2ts\" (UniqueName: \"kubernetes.io/projected/6120f11d-14c8-4666-a201-469a5bca37de-kube-api-access-bs2ts\") pod \"6120f11d-14c8-4666-a201-469a5bca37de\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") "
Nov 03 17:42:34.842908 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:34.842792    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-home\" (UniqueName: \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-home\") pod \"6120f11d-14c8-4666-a201-469a5bca37de\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") "
Nov 03 17:42:34.843033 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:34.842958    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-dshm\") pod \"6120f11d-14c8-4666-a201-469a5bca37de\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") "
Nov 03 17:42:34.843033 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:34.843015    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-model-cache\") pod \"6120f11d-14c8-4666-a201-469a5bca37de\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") "
Nov 03 17:42:34.843145 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:34.843074    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-tmp-dir\") pod \"6120f11d-14c8-4666-a201-469a5bca37de\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") "
Nov 03 17:42:34.843145 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:34.843094    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-home" pod "6120f11d-14c8-4666-a201-469a5bca37de" (UID: "6120f11d-14c8-4666-a201-469a5bca37de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:42:34.843145 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:34.843125    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6120f11d-14c8-4666-a201-469a5bca37de-tls-certs\" (UniqueName: \"kubernetes.io/secret/6120f11d-14c8-4666-a201-469a5bca37de-tls-certs\") pod \"6120f11d-14c8-4666-a201-469a5bca37de\" (UID: \"6120f11d-14c8-4666-a201-469a5bca37de\") "
Nov 03 17:42:34.843345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:34.843324    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-tmp-dir" pod "6120f11d-14c8-4666-a201-469a5bca37de" (UID: "6120f11d-14c8-4666-a201-469a5bca37de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:42:34.843421 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:34.843341    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-model-cache" pod "6120f11d-14c8-4666-a201-469a5bca37de" (UID: "6120f11d-14c8-4666-a201-469a5bca37de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:42:34.843843 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:34.843818    6059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-home\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:42:34.843913 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:34.843848    6059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-model-cache\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:42:34.843913 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:34.843863    6059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-tmp-dir\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:42:34.845227 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:34.845192    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-dshm" pod "6120f11d-14c8-4666-a201-469a5bca37de" (UID: "6120f11d-14c8-4666-a201-469a5bca37de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:42:34.845335 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:34.845244    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6120f11d-14c8-4666-a201-469a5bca37de-tls-certs" pod "6120f11d-14c8-4666-a201-469a5bca37de" (UID: "6120f11d-14c8-4666-a201-469a5bca37de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:42:34.845335 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:34.845290    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6120f11d-14c8-4666-a201-469a5bca37de-kube-api-access-bs2ts" pod "6120f11d-14c8-4666-a201-469a5bca37de" (UID: "6120f11d-14c8-4666-a201-469a5bca37de"). InnerVolumeSpecName "kube-api-access-bs2ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:42:34.945885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:34.945844    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs2ts\" (UniqueName: \"kubernetes.io/projected/6120f11d-14c8-4666-a201-469a5bca37de-kube-api-access-bs2ts\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:42:34.945885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:34.945888    6059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6120f11d-14c8-4666-a201-469a5bca37de-dshm\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:42:34.946096 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:34.945904    6059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6120f11d-14c8-4666-a201-469a5bca37de-tls-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:42:35.294702 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:35.294664    6059 generic.go:356] "Generic (PLEG): container finished" podID="6120f11d-14c8-4666-a201-469a5bca37de" containerID="a380062a162c02aece834579a21a76edab1d371f891cce90f4d0f1657a7aa76d" exitCode=0
Nov 03 17:42:35.294887 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:35.294721    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"
Nov 03 17:42:35.294887 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:35.294752    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w" event={"ID":"6120f11d-14c8-4666-a201-469a5bca37de","Type":"ContainerDied","Data":"a380062a162c02aece834579a21a76edab1d371f891cce90f4d0f1657a7aa76d"}
Nov 03 17:42:35.294887 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:35.294790    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w" event={"ID":"6120f11d-14c8-4666-a201-469a5bca37de","Type":"ContainerDied","Data":"fdc34b98d99ddabbcd972d82ce372f36e900d7d65646484042a65199fb85e114"}
Nov 03 17:42:35.294887 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:35.294808    6059 scope.go:122] "RemoveContainer" containerID="a380062a162c02aece834579a21a76edab1d371f891cce90f4d0f1657a7aa76d"
Nov 03 17:42:35.306716 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:35.306690    6059 scope.go:122] "RemoveContainer" containerID="a380062a162c02aece834579a21a76edab1d371f891cce90f4d0f1657a7aa76d"
Nov 03 17:42:35.307041 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:35.307021    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a380062a162c02aece834579a21a76edab1d371f891cce90f4d0f1657a7aa76d\": container with ID starting with a380062a162c02aece834579a21a76edab1d371f891cce90f4d0f1657a7aa76d not found: ID does not exist" containerID="a380062a162c02aece834579a21a76edab1d371f891cce90f4d0f1657a7aa76d"
Nov 03 17:42:35.307089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:35.307052    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a380062a162c02aece834579a21a76edab1d371f891cce90f4d0f1657a7aa76d"} err="failed to get container status \"a380062a162c02aece834579a21a76edab1d371f891cce90f4d0f1657a7aa76d\": rpc error: code = NotFound desc = could not find container \"a380062a162c02aece834579a21a76edab1d371f891cce90f4d0f1657a7aa76d\": container with ID starting with a380062a162c02aece834579a21a76edab1d371f891cce90f4d0f1657a7aa76d not found: ID does not exist"
Nov 03 17:42:35.351429 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:35.351390    6059 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-03ea5a1192b1: secret "agc-cred-03ea5a1192b1" not found
Nov 03 17:42:35.351429 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:35.351428    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4: secret "agc-cred-03ea5a1192b1" not found
Nov 03 17:42:35.351721 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:35.351487    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-provider-credentials podName:44e7441c-6611-4640-9642-f67d1f3c8172 nodeName:}" failed. No retries permitted until 2026-11-03 17:42:35.851467103 +0000 UTC m=+1496.649179880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-provider-credentials") pod "payload-processing-external-model-8657f8dff8-j9qn4" (UID: "44e7441c-6611-4640-9642-f67d1f3c8172") : secret "agc-cred-03ea5a1192b1" not found
Nov 03 17:42:35.467886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:35.467845    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"]
Nov 03 17:42:35.473321 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:35.473285    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af9e3c/rate-test-model-af9e3c-kserve-68fd55f97-pjt6w"]
Nov 03 17:42:35.859390 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:35.859336    6059 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-03ea5a1192b1: secret "agc-cred-03ea5a1192b1" not found
Nov 03 17:42:35.859390 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:35.859387    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4: secret "agc-cred-03ea5a1192b1" not found
Nov 03 17:42:35.859828 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:35.859472    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-provider-credentials podName:44e7441c-6611-4640-9642-f67d1f3c8172 nodeName:}" failed. No retries permitted until 2026-11-03 17:42:36.859455247 +0000 UTC m=+1497.657168005 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-provider-credentials") pod "payload-processing-external-model-8657f8dff8-j9qn4" (UID: "44e7441c-6611-4640-9642-f67d1f3c8172") : secret "agc-cred-03ea5a1192b1" not found
Nov 03 17:42:36.863045 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:36.863006    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:42:36.872014 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:36.871926    6059 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-03ea5a1192b1: secret "agc-cred-03ea5a1192b1" not found
Nov 03 17:42:36.872014 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:36.871955    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4: secret "agc-cred-03ea5a1192b1" not found
Nov 03 17:42:36.872211 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:36.872026    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-provider-credentials podName:44e7441c-6611-4640-9642-f67d1f3c8172 nodeName:}" failed. No retries permitted until 2026-11-03 17:42:38.872008634 +0000 UTC m=+1499.669721398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-provider-credentials") pod "payload-processing-external-model-8657f8dff8-j9qn4" (UID: "44e7441c-6611-4640-9642-f67d1f3c8172") : secret "agc-cred-03ea5a1192b1" not found
Nov 03 17:42:37.004832 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:37.004757    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl" podUID="96150112-b203-45a9-928a-471c023eee31" containerName="payload-processing" containerID="cri-o://fe9f415485032d8d6a129fb652094a0070e358c63b9473a102af05deb62e57a8" gracePeriod=25
Nov 03 17:42:37.227900 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:37.227864    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn" podUID="edbd8e63-4b01-4d65-abd5-af5bda2a1e24" containerName="payload-pre-processing" containerID="cri-o://051ed64693ff67de9c5905e88d8321506c58f9588b601ca8a39e88f16c4ea7a0" gracePeriod=25
Nov 03 17:42:37.318610 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:37.318579    6059 generic.go:356] "Generic (PLEG): container finished" podID="96150112-b203-45a9-928a-471c023eee31" containerID="fe9f415485032d8d6a129fb652094a0070e358c63b9473a102af05deb62e57a8" exitCode=0
Nov 03 17:42:37.318779 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:37.318647    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl" event={"ID":"96150112-b203-45a9-928a-471c023eee31","Type":"ContainerDied","Data":"fe9f415485032d8d6a129fb652094a0070e358c63b9473a102af05deb62e57a8"}
Nov 03 17:42:37.337592 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:37.337560    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6120f11d-14c8-4666-a201-469a5bca37de" path="/var/lib/kubelet/pods/6120f11d-14c8-4666-a201-469a5bca37de/volumes"
Nov 03 17:42:37.376013 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:37.375923    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl"
Nov 03 17:42:37.479560 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:37.479521    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96150112-b203-45a9-928a-471c023eee31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96150112-b203-45a9-928a-471c023eee31-plugins-config-volume\") pod \"96150112-b203-45a9-928a-471c023eee31\" (UID: \"96150112-b203-45a9-928a-471c023eee31\") "
Nov 03 17:42:37.479738 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:37.479672    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96150112-b203-45a9-928a-471c023eee31-kube-api-access-z2jff\" (UniqueName: \"kubernetes.io/projected/96150112-b203-45a9-928a-471c023eee31-kube-api-access-z2jff\") pod \"96150112-b203-45a9-928a-471c023eee31\" (UID: \"96150112-b203-45a9-928a-471c023eee31\") "
Nov 03 17:42:37.480188 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:37.480160    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96150112-b203-45a9-928a-471c023eee31-plugins-config-volume" pod "96150112-b203-45a9-928a-471c023eee31" (UID: "96150112-b203-45a9-928a-471c023eee31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:42:37.484285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:37.482975    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96150112-b203-45a9-928a-471c023eee31-kube-api-access-z2jff" pod "96150112-b203-45a9-928a-471c023eee31" (UID: "96150112-b203-45a9-928a-471c023eee31"). InnerVolumeSpecName "kube-api-access-z2jff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:42:37.487530 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:37.487504    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn"
Nov 03 17:42:37.582861 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:37.581999    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edbd8e63-4b01-4d65-abd5-af5bda2a1e24-kube-api-access-nf9d6\" (UniqueName: \"kubernetes.io/projected/edbd8e63-4b01-4d65-abd5-af5bda2a1e24-kube-api-access-nf9d6\") pod \"edbd8e63-4b01-4d65-abd5-af5bda2a1e24\" (UID: \"edbd8e63-4b01-4d65-abd5-af5bda2a1e24\") "
Nov 03 17:42:37.582861 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:37.582148    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edbd8e63-4b01-4d65-abd5-af5bda2a1e24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edbd8e63-4b01-4d65-abd5-af5bda2a1e24-plugins-config-volume\") pod \"edbd8e63-4b01-4d65-abd5-af5bda2a1e24\" (UID: \"edbd8e63-4b01-4d65-abd5-af5bda2a1e24\") "
Nov 03 17:42:37.585648 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:37.585570    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96150112-b203-45a9-928a-471c023eee31-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:42:37.585648 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:37.585602    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2jff\" (UniqueName: \"kubernetes.io/projected/96150112-b203-45a9-928a-471c023eee31-kube-api-access-z2jff\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:42:37.586036 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:37.585970    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edbd8e63-4b01-4d65-abd5-af5bda2a1e24-kube-api-access-nf9d6" pod "edbd8e63-4b01-4d65-abd5-af5bda2a1e24" (UID: "edbd8e63-4b01-4d65-abd5-af5bda2a1e24"). InnerVolumeSpecName "kube-api-access-nf9d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:42:37.586036 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:37.585969    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edbd8e63-4b01-4d65-abd5-af5bda2a1e24-plugins-config-volume" pod "edbd8e63-4b01-4d65-abd5-af5bda2a1e24" (UID: "edbd8e63-4b01-4d65-abd5-af5bda2a1e24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:42:37.686877 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:37.686779    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edbd8e63-4b01-4d65-abd5-af5bda2a1e24-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:42:37.686877 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:37.686820    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf9d6\" (UniqueName: \"kubernetes.io/projected/edbd8e63-4b01-4d65-abd5-af5bda2a1e24-kube-api-access-nf9d6\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:42:38.329032 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:38.328947    6059 generic.go:356] "Generic (PLEG): container finished" podID="edbd8e63-4b01-4d65-abd5-af5bda2a1e24" containerID="051ed64693ff67de9c5905e88d8321506c58f9588b601ca8a39e88f16c4ea7a0" exitCode=0
Nov 03 17:42:38.329032 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:38.328998    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn"
Nov 03 17:42:38.329527 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:38.329023    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn" event={"ID":"edbd8e63-4b01-4d65-abd5-af5bda2a1e24","Type":"ContainerDied","Data":"051ed64693ff67de9c5905e88d8321506c58f9588b601ca8a39e88f16c4ea7a0"}
Nov 03 17:42:38.329527 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:38.329075    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn" event={"ID":"edbd8e63-4b01-4d65-abd5-af5bda2a1e24","Type":"ContainerDied","Data":"ac9b4e02c079743dcf645b246e7b5df2143049cb5fe43447fc0fc8d7515ee8e8"}
Nov 03 17:42:38.329527 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:38.329098    6059 scope.go:122] "RemoveContainer" containerID="051ed64693ff67de9c5905e88d8321506c58f9588b601ca8a39e88f16c4ea7a0"
Nov 03 17:42:38.330814 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:38.330791    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl"
Nov 03 17:42:38.330902 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:38.330810    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl" event={"ID":"96150112-b203-45a9-928a-471c023eee31","Type":"ContainerDied","Data":"ddb154d2fa0955ef57131e392de360d538aff125e9aa3f348525d08a40b7e8dc"}
Nov 03 17:42:38.342181 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:38.342161    6059 scope.go:122] "RemoveContainer" containerID="051ed64693ff67de9c5905e88d8321506c58f9588b601ca8a39e88f16c4ea7a0"
Nov 03 17:42:38.342575 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:38.342542    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"051ed64693ff67de9c5905e88d8321506c58f9588b601ca8a39e88f16c4ea7a0\": container with ID starting with 051ed64693ff67de9c5905e88d8321506c58f9588b601ca8a39e88f16c4ea7a0 not found: ID does not exist" containerID="051ed64693ff67de9c5905e88d8321506c58f9588b601ca8a39e88f16c4ea7a0"
Nov 03 17:42:38.342670 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:38.342586    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051ed64693ff67de9c5905e88d8321506c58f9588b601ca8a39e88f16c4ea7a0"} err="failed to get container status \"051ed64693ff67de9c5905e88d8321506c58f9588b601ca8a39e88f16c4ea7a0\": rpc error: code = NotFound desc = could not find container \"051ed64693ff67de9c5905e88d8321506c58f9588b601ca8a39e88f16c4ea7a0\": container with ID starting with 051ed64693ff67de9c5905e88d8321506c58f9588b601ca8a39e88f16c4ea7a0 not found: ID does not exist"
Nov 03 17:42:38.342670 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:38.342606    6059 scope.go:122] "RemoveContainer" containerID="fe9f415485032d8d6a129fb652094a0070e358c63b9473a102af05deb62e57a8"
Nov 03 17:42:38.501270 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:38.501236    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn"]
Nov 03 17:42:38.553251 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:38.553210    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5f88f673-5f86f69b99-8klfn"]
Nov 03 17:42:38.620618 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:38.620534    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl"]
Nov 03 17:42:38.624440 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:38.624410    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5f88f673-64558f5799-9ldjl"]
Nov 03 17:42:38.731183 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:38.731147    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:38.905817 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:38.905732    6059 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-03ea5a1192b1: secret "agc-cred-03ea5a1192b1" not found
Nov 03 17:42:38.905817 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:38.905770    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4: secret "agc-cred-03ea5a1192b1" not found
Nov 03 17:42:38.906002 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:38.905864    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-provider-credentials podName:44e7441c-6611-4640-9642-f67d1f3c8172 nodeName:}" failed. No retries permitted until 2026-11-03 17:42:42.905843132 +0000 UTC m=+1503.703555894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-provider-credentials") pod "payload-processing-external-model-8657f8dff8-j9qn4" (UID: "44e7441c-6611-4640-9642-f67d1f3c8172") : secret "agc-cred-03ea5a1192b1" not found
Nov 03 17:42:39.340562 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:39.340531    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96150112-b203-45a9-928a-471c023eee31" path="/var/lib/kubelet/pods/96150112-b203-45a9-928a-471c023eee31/volumes"
Nov 03 17:42:39.341147 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:39.341120    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edbd8e63-4b01-4d65-abd5-af5bda2a1e24" path="/var/lib/kubelet/pods/edbd8e63-4b01-4d65-abd5-af5bda2a1e24/volumes"
Nov 03 17:42:39.559343 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:39.559306    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:39.587909 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:39.587876    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 17:42:39.595193 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:39.595108    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 17:42:39.606716 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:39.606688    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:42:39.612120 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:39.612098    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:42:40.189664 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:40.189628    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"]
Nov 03 17:42:41.109217 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:41.109180    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:41.323531 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:41.323489    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:41.331966 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:41.331939    6059 scope.go:122] "RemoveContainer" containerID="2043b288d95f82bb7c7dd4dcdaf2ba638f633b4a94a2f5aab0b33c550a9ced58"
Nov 03 17:42:41.342510 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:41.342484    6059 scope.go:122] "RemoveContainer" containerID="f5a0c60adf16a7e62153b2819e30f21a465bc11fd3377651f9a0d35f89ddb95c"
Nov 03 17:42:41.352896 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:41.352874    6059 scope.go:122] "RemoveContainer" containerID="13551372c0c7bfa6c25d13166e6b266fcafe4b9fb60b9097cffa346d6c4f7548"
Nov 03 17:42:41.511788 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:41.511753    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:42.012465 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:42.012431    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:42.970688 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:42.970649    6059 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-03ea5a1192b1: secret "agc-cred-03ea5a1192b1" not found
Nov 03 17:42:42.970688 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:42.970683    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4: secret "agc-cred-03ea5a1192b1" not found
Nov 03 17:42:42.971160 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:42.970740    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-provider-credentials podName:44e7441c-6611-4640-9642-f67d1f3c8172 nodeName:}" failed. No retries permitted until 2026-11-03 17:42:50.970725031 +0000 UTC m=+1511.768437826 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-provider-credentials") pod "payload-processing-external-model-8657f8dff8-j9qn4" (UID: "44e7441c-6611-4640-9642-f67d1f3c8172") : secret "agc-cred-03ea5a1192b1" not found
Nov 03 17:42:43.018046 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.017999    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn" podUID="09ca8b69-3b26-4d5f-97b7-ad0646e5556a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 17:42:43.054308 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.054276    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c6a27d-kd5zr"]
Nov 03 17:42:43.089482 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.089442    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76c6a27d-kd5zr"]
Nov 03 17:42:43.336962 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.336884    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="766f8ea9-89c1-459e-be88-dcdf8411abdd" path="/var/lib/kubelet/pods/766f8ea9-89c1-459e-be88-dcdf8411abdd/volumes"
Nov 03 17:42:43.384759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.384721    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"]
Nov 03 17:42:43.386066 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.386027    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn" podUID="09ca8b69-3b26-4d5f-97b7-ad0646e5556a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 17:42:43.420396 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.420343    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"]
Nov 03 17:42:43.420987 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.420972    6059 cpu_manager.go:414] "containerMap: removing container" podUID="edbd8e63-4b01-4d65-abd5-af5bda2a1e24" containerName="payload-pre-processing"
Nov 03 17:42:43.421037 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.420989    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edbd8e63-4b01-4d65-abd5-af5bda2a1e24" containerName="payload-pre-processing"
Nov 03 17:42:43.421037 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.421016    6059 cpu_manager.go:414] "containerMap: removing container" podUID="e433f04f-2bfe-4844-baa8-82f127259f71" containerName="payload-processing"
Nov 03 17:42:43.421037 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.421022    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e433f04f-2bfe-4844-baa8-82f127259f71" containerName="payload-processing"
Nov 03 17:42:43.421160 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.421041    6059 cpu_manager.go:414] "containerMap: removing container" podUID="6120f11d-14c8-4666-a201-469a5bca37de" containerName="main"
Nov 03 17:42:43.421160 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.421047    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6120f11d-14c8-4666-a201-469a5bca37de" containerName="main"
Nov 03 17:42:43.421160 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.421054    6059 cpu_manager.go:414] "containerMap: removing container" podUID="96150112-b203-45a9-928a-471c023eee31" containerName="payload-processing"
Nov 03 17:42:43.421160 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.421060    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96150112-b203-45a9-928a-471c023eee31" containerName="payload-processing"
Nov 03 17:42:43.421160 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.421153    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6120f11d-14c8-4666-a201-469a5bca37de" containerName="main"
Nov 03 17:42:43.421160 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.421162    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="96150112-b203-45a9-928a-471c023eee31" containerName="payload-processing"
Nov 03 17:42:43.421349 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.421172    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="edbd8e63-4b01-4d65-abd5-af5bda2a1e24" containerName="payload-pre-processing"
Nov 03 17:42:43.421349 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.421181    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e433f04f-2bfe-4844-baa8-82f127259f71" containerName="payload-processing"
Nov 03 17:42:43.426601 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.426578    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"
Nov 03 17:42:43.481021 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.480980    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-d070390c9017\""
Nov 03 17:42:43.481176 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.481042    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c4482bd19ba8\""
Nov 03 17:42:43.485900 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.485867    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"]
Nov 03 17:42:43.579166 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.579125    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ebefb030-567e-47f1-bb77-f62b222d5280-routing-overlay-volume\") pod \"payload-processing-external-model-86cbdddf86-2k5kw\" (UID: \"ebefb030-567e-47f1-bb77-f62b222d5280\") " pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"
Nov 03 17:42:43.579349 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.579180    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebefb030-567e-47f1-bb77-f62b222d5280-plugins-config-volume\") pod \"payload-processing-external-model-86cbdddf86-2k5kw\" (UID: \"ebefb030-567e-47f1-bb77-f62b222d5280\") " pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"
Nov 03 17:42:43.579349 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.579274    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82qkm\" (UniqueName: \"kubernetes.io/projected/ebefb030-567e-47f1-bb77-f62b222d5280-kube-api-access-82qkm\") pod \"payload-processing-external-model-86cbdddf86-2k5kw\" (UID: \"ebefb030-567e-47f1-bb77-f62b222d5280\") " pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"
Nov 03 17:42:43.579349 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.579325    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ebefb030-567e-47f1-bb77-f62b222d5280-provider-credentials\") pod \"payload-processing-external-model-86cbdddf86-2k5kw\" (UID: \"ebefb030-567e-47f1-bb77-f62b222d5280\") " pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"
Nov 03 17:42:43.681337 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.681228    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ebefb030-567e-47f1-bb77-f62b222d5280-routing-overlay-volume\") pod \"payload-processing-external-model-86cbdddf86-2k5kw\" (UID: \"ebefb030-567e-47f1-bb77-f62b222d5280\") " pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"
Nov 03 17:42:43.681337 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.681306    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebefb030-567e-47f1-bb77-f62b222d5280-plugins-config-volume\") pod \"payload-processing-external-model-86cbdddf86-2k5kw\" (UID: \"ebefb030-567e-47f1-bb77-f62b222d5280\") " pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"
Nov 03 17:42:43.681618 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.681406    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82qkm\" (UniqueName: \"kubernetes.io/projected/ebefb030-567e-47f1-bb77-f62b222d5280-kube-api-access-82qkm\") pod \"payload-processing-external-model-86cbdddf86-2k5kw\" (UID: \"ebefb030-567e-47f1-bb77-f62b222d5280\") " pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"
Nov 03 17:42:43.681618 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.681479    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ebefb030-567e-47f1-bb77-f62b222d5280-provider-credentials\") pod \"payload-processing-external-model-86cbdddf86-2k5kw\" (UID: \"ebefb030-567e-47f1-bb77-f62b222d5280\") " pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"
Nov 03 17:42:43.682067 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.682044    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ebefb030-567e-47f1-bb77-f62b222d5280-routing-overlay-volume\") pod \"payload-processing-external-model-86cbdddf86-2k5kw\" (UID: \"ebefb030-567e-47f1-bb77-f62b222d5280\") " pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"
Nov 03 17:42:43.682160 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.682147    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebefb030-567e-47f1-bb77-f62b222d5280-plugins-config-volume\") pod \"payload-processing-external-model-86cbdddf86-2k5kw\" (UID: \"ebefb030-567e-47f1-bb77-f62b222d5280\") " pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"
Nov 03 17:42:43.684021 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.683999    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ebefb030-567e-47f1-bb77-f62b222d5280-provider-credentials\") pod \"payload-processing-external-model-86cbdddf86-2k5kw\" (UID: \"ebefb030-567e-47f1-bb77-f62b222d5280\") " pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"
Nov 03 17:42:43.745786 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:43.745747    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82qkm\" (UniqueName: \"kubernetes.io/projected/ebefb030-567e-47f1-bb77-f62b222d5280-kube-api-access-82qkm\") pod \"payload-processing-external-model-86cbdddf86-2k5kw\" (UID: \"ebefb030-567e-47f1-bb77-f62b222d5280\") " pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"
Nov 03 17:42:44.041365 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:44.041330    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"
Nov 03 17:42:44.445272 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:42:44.444341    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebefb030_567e_47f1_bb77_f62b222d5280.slice/crio-902fab72ac2ae16614b81f2d06ccdf86891b955f70f20e57e414553a741590ac WatchSource:0}: Error finding container 902fab72ac2ae16614b81f2d06ccdf86891b955f70f20e57e414553a741590ac: Status 404 returned error can't find the container with id 902fab72ac2ae16614b81f2d06ccdf86891b955f70f20e57e414553a741590ac
Nov 03 17:42:44.452575 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:44.452517    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"]
Nov 03 17:42:44.460084 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:44.460053    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:45.190753 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:45.190699    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx" podUID="e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853" containerName="payload-processing" containerID="cri-o://43ab9a3dcb4dcc1a64aec263f036f2666300e0a1966e6e06866ca2fc63544e0f" gracePeriod=25
Nov 03 17:42:45.424867 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:45.424827    6059 generic.go:356] "Generic (PLEG): container finished" podID="e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853" containerID="43ab9a3dcb4dcc1a64aec263f036f2666300e0a1966e6e06866ca2fc63544e0f" exitCode=0
Nov 03 17:42:45.425070 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:45.424897    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx" event={"ID":"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853","Type":"ContainerDied","Data":"43ab9a3dcb4dcc1a64aec263f036f2666300e0a1966e6e06866ca2fc63544e0f"}
Nov 03 17:42:45.426955 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:45.426924    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw" event={"ID":"ebefb030-567e-47f1-bb77-f62b222d5280","Type":"ContainerStarted","Data":"7d193d49070c96a5bd0dcee8f5d0f38ca04f9df6f4d34a6fb30ee60cb36b3bf6"}
Nov 03 17:42:45.427077 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:45.426983    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw" event={"ID":"ebefb030-567e-47f1-bb77-f62b222d5280","Type":"ContainerStarted","Data":"902fab72ac2ae16614b81f2d06ccdf86891b955f70f20e57e414553a741590ac"}
Nov 03 17:42:45.427186 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:45.427167    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"
Nov 03 17:42:45.468381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:45.468327    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"
Nov 03 17:42:45.550219 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:45.550151    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw" podStartSLOduration=2.550134635 podStartE2EDuration="2.550134635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:42:45.546628889 +0000 UTC m=+1506.344341670" watchObservedRunningTime="2026-11-03 17:42:45.550134635 +0000 UTC m=+1506.347847416"
Nov 03 17:42:45.609435 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:45.609400    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-provider-credentials\") pod \"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853\" (UID: \"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853\") "
Nov 03 17:42:45.609605 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:45.609495    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-kube-api-access-xw5hq\" (UniqueName: \"kubernetes.io/projected/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-kube-api-access-xw5hq\") pod \"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853\" (UID: \"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853\") "
Nov 03 17:42:45.609643 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:45.609615    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-plugins-config-volume\") pod \"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853\" (UID: \"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853\") "
Nov 03 17:42:45.609735 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:45.609720    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-routing-overlay-volume\") pod \"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853\" (UID: \"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853\") "
Nov 03 17:42:45.610083 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:45.610054    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-plugins-config-volume" pod "e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853" (UID: "e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:42:45.610083 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:45.610061    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-routing-overlay-volume" pod "e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853" (UID: "e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:42:45.610296 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:45.610281    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:42:45.610334 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:45.610299    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:42:45.611618 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:45.611601    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-kube-api-access-xw5hq" pod "e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853" (UID: "e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853"). InnerVolumeSpecName "kube-api-access-xw5hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:42:45.611707 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:45.611647    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-provider-credentials" pod "e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853" (UID: "e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:42:45.712224 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:45.712185    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:42:45.712224 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:45.712223    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw5hq\" (UniqueName: \"kubernetes.io/projected/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853-kube-api-access-xw5hq\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:42:46.435966 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:46.435934    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"
Nov 03 17:42:46.436502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:46.435966    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx" event={"ID":"e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853","Type":"ContainerDied","Data":"1809a4355de510954a2ecc7880107883e99bcdec4636f80d7ab6930208d7569d"}
Nov 03 17:42:46.436502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:46.436033    6059 scope.go:122] "RemoveContainer" containerID="43ab9a3dcb4dcc1a64aec263f036f2666300e0a1966e6e06866ca2fc63544e0f"
Nov 03 17:42:46.608243 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:46.608201    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"]
Nov 03 17:42:46.659817 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:46.659777    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f25xdx"]
Nov 03 17:42:47.336391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:47.336345    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853" path="/var/lib/kubelet/pods/e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853/volumes"
Nov 03 17:42:48.385442 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:48.385395    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn" podUID="09ca8b69-3b26-4d5f-97b7-ad0646e5556a" containerName="payload-processing" containerID="cri-o://db04db97ee12ed84a6a1bfc0d1f26dfdd8e546253f86383e59e3fe52435c3e45" gracePeriod=25
Nov 03 17:42:48.661308 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:48.661280    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"
Nov 03 17:42:48.825736 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:48.825672    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:48.848865 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:48.848826    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-kube-api-access-2t8f4\" (UniqueName: \"kubernetes.io/projected/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-kube-api-access-2t8f4\") pod \"09ca8b69-3b26-4d5f-97b7-ad0646e5556a\" (UID: \"09ca8b69-3b26-4d5f-97b7-ad0646e5556a\") "
Nov 03 17:42:48.848865 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:48.848872    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-plugins-config-volume\") pod \"09ca8b69-3b26-4d5f-97b7-ad0646e5556a\" (UID: \"09ca8b69-3b26-4d5f-97b7-ad0646e5556a\") "
Nov 03 17:42:48.849082 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:48.848910    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-routing-overlay-volume\") pod \"09ca8b69-3b26-4d5f-97b7-ad0646e5556a\" (UID: \"09ca8b69-3b26-4d5f-97b7-ad0646e5556a\") "
Nov 03 17:42:48.849130 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:48.849104    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-provider-credentials\") pod \"09ca8b69-3b26-4d5f-97b7-ad0646e5556a\" (UID: \"09ca8b69-3b26-4d5f-97b7-ad0646e5556a\") "
Nov 03 17:42:48.849333 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:48.849302    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-routing-overlay-volume" pod "09ca8b69-3b26-4d5f-97b7-ad0646e5556a" (UID: "09ca8b69-3b26-4d5f-97b7-ad0646e5556a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:42:48.849447 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:48.849341    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-plugins-config-volume" pod "09ca8b69-3b26-4d5f-97b7-ad0646e5556a" (UID: "09ca8b69-3b26-4d5f-97b7-ad0646e5556a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:42:48.849709 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:48.849692    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:42:48.849749 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:48.849713    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:42:48.851151 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:48.851123    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-kube-api-access-2t8f4" pod "09ca8b69-3b26-4d5f-97b7-ad0646e5556a" (UID: "09ca8b69-3b26-4d5f-97b7-ad0646e5556a"). InnerVolumeSpecName "kube-api-access-2t8f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:42:48.851257 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:48.851190    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-provider-credentials" pod "09ca8b69-3b26-4d5f-97b7-ad0646e5556a" (UID: "09ca8b69-3b26-4d5f-97b7-ad0646e5556a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:42:48.951297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:48.951211    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t8f4\" (UniqueName: \"kubernetes.io/projected/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-kube-api-access-2t8f4\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:42:48.951297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:48.951246    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09ca8b69-3b26-4d5f-97b7-ad0646e5556a-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:42:49.054381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:49.054331    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp"]
Nov 03 17:42:49.057885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:49.057850    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3e6d015-bs6kp"]
Nov 03 17:42:49.336545 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:49.336507    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec7cf9d3-8eb7-4398-a248-11712584d152" path="/var/lib/kubelet/pods/ec7cf9d3-8eb7-4398-a248-11712584d152/volumes"
Nov 03 17:42:49.464853 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:49.464819    6059 generic.go:356] "Generic (PLEG): container finished" podID="09ca8b69-3b26-4d5f-97b7-ad0646e5556a" containerID="db04db97ee12ed84a6a1bfc0d1f26dfdd8e546253f86383e59e3fe52435c3e45" exitCode=0
Nov 03 17:42:49.465312 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:49.464860    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"
Nov 03 17:42:49.465312 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:49.464901    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn" event={"ID":"09ca8b69-3b26-4d5f-97b7-ad0646e5556a","Type":"ContainerDied","Data":"db04db97ee12ed84a6a1bfc0d1f26dfdd8e546253f86383e59e3fe52435c3e45"}
Nov 03 17:42:49.465312 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:49.464962    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68478f785f-mnkbn" event={"ID":"09ca8b69-3b26-4d5f-97b7-ad0646e5556a","Type":"ContainerDied","Data":"08f5088b0fd1eb4abadba133e403561fff658c6a98cf8e16a63eed4296a783d7"}
Nov 03 17:42:49.465312 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:49.464988    6059 scope.go:122] "RemoveContainer" containerID="db04db97ee12ed84a6a1bfc0d1f26dfdd8e546253f86383e59e3fe52435c3e45"
Nov 03 17:42:49.475824 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:49.475800    6059 scope.go:122] "RemoveContainer" containerID="db04db97ee12ed84a6a1bfc0d1f26dfdd8e546253f86383e59e3fe52435c3e45"
Nov 03 17:42:49.476187 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:49.476160    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db04db97ee12ed84a6a1bfc0d1f26dfdd8e546253f86383e59e3fe52435c3e45\": container with ID starting with db04db97ee12ed84a6a1bfc0d1f26dfdd8e546253f86383e59e3fe52435c3e45 not found: ID does not exist" containerID="db04db97ee12ed84a6a1bfc0d1f26dfdd8e546253f86383e59e3fe52435c3e45"
Nov 03 17:42:49.476292 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:49.476195    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db04db97ee12ed84a6a1bfc0d1f26dfdd8e546253f86383e59e3fe52435c3e45"} err="failed to get container status \"db04db97ee12ed84a6a1bfc0d1f26dfdd8e546253f86383e59e3fe52435c3e45\": rpc error: code = NotFound desc = could not find container \"db04db97ee12ed84a6a1bfc0d1f26dfdd8e546253f86383e59e3fe52435c3e45\": container with ID starting with db04db97ee12ed84a6a1bfc0d1f26dfdd8e546253f86383e59e3fe52435c3e45 not found: ID does not exist"
Nov 03 17:42:49.637476 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:49.637390    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"]
Nov 03 17:42:49.641911 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:49.641882    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-68478f785f-mnkbn"]
Nov 03 17:42:50.982159 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:50.982125    6059 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-03ea5a1192b1: secret "agc-cred-03ea5a1192b1" not found
Nov 03 17:42:50.982159 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:50.982156    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4: secret "agc-cred-03ea5a1192b1" not found
Nov 03 17:42:50.982647 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:42:50.982213    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-provider-credentials podName:44e7441c-6611-4640-9642-f67d1f3c8172 nodeName:}" failed. No retries permitted until 2026-11-03 17:43:06.982197998 +0000 UTC m=+1527.779910757 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-provider-credentials") pod "payload-processing-external-model-8657f8dff8-j9qn4" (UID: "44e7441c-6611-4640-9642-f67d1f3c8172") : secret "agc-cred-03ea5a1192b1" not found
Nov 03 17:42:51.337324 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:51.337243    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09ca8b69-3b26-4d5f-97b7-ad0646e5556a" path="/var/lib/kubelet/pods/09ca8b69-3b26-4d5f-97b7-ad0646e5556a/volumes"
Nov 03 17:42:53.464381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:53.463337    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:56.439618 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:56.439590    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"
Nov 03 17:42:56.617232 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:56.617201    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"]
Nov 03 17:42:57.638300 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:57.638257    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7481-51fc92"]
Nov 03 17:42:57.638889 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:57.638874    6059 cpu_manager.go:414] "containerMap: removing container" podUID="e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853" containerName="payload-processing"
Nov 03 17:42:57.638938 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:57.638890    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853" containerName="payload-processing"
Nov 03 17:42:57.638938 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:57.638907    6059 cpu_manager.go:414] "containerMap: removing container" podUID="09ca8b69-3b26-4d5f-97b7-ad0646e5556a" containerName="payload-processing"
Nov 03 17:42:57.638938 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:57.638912    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ca8b69-3b26-4d5f-97b7-ad0646e5556a" containerName="payload-processing"
Nov 03 17:42:57.639040 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:57.639027    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4a7b2c4-a543-4cfd-bb16-9cd0b03e0853" containerName="payload-processing"
Nov 03 17:42:57.639040 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:57.639037    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ca8b69-3b26-4d5f-97b7-ad0646e5556a" containerName="payload-processing"
Nov 03 17:42:57.642497 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:57.642481    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-51fc92"
Nov 03 17:42:57.701109 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:57.701074    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-51fc92"]
Nov 03 17:42:57.747463 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:57.747422    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hjccf\""
Nov 03 17:42:57.748176 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:57.748151    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx87f\" (UniqueName: \"kubernetes.io/projected/b9fa62ee-206b-4b9f-951e-15dd710d75b0-kube-api-access-vx87f\") pod \"test-curl-7481-51fc92\" (UID: \"b9fa62ee-206b-4b9f-951e-15dd710d75b0\") " pod="opendatahub/test-curl-7481-51fc92"
Nov 03 17:42:57.849955 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:57.849919    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx87f\" (UniqueName: \"kubernetes.io/projected/b9fa62ee-206b-4b9f-951e-15dd710d75b0-kube-api-access-vx87f\") pod \"test-curl-7481-51fc92\" (UID: \"b9fa62ee-206b-4b9f-951e-15dd710d75b0\") " pod="opendatahub/test-curl-7481-51fc92"
Nov 03 17:42:57.889749 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:57.889657    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:57.913150 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:57.913119    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx87f\" (UniqueName: \"kubernetes.io/projected/b9fa62ee-206b-4b9f-951e-15dd710d75b0-kube-api-access-vx87f\") pod \"test-curl-7481-51fc92\" (UID: \"b9fa62ee-206b-4b9f-951e-15dd710d75b0\") " pod="opendatahub/test-curl-7481-51fc92"
Nov 03 17:42:57.955057 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:57.955021    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-51fc92"
Nov 03 17:42:58.365801 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:42:58.365765    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9fa62ee_206b_4b9f_951e_15dd710d75b0.slice/crio-8a4aecca0e7039fde6fc56dc9c824708f6e33cd9303bc98f9107f3a5d80c6acd WatchSource:0}: Error finding container 8a4aecca0e7039fde6fc56dc9c824708f6e33cd9303bc98f9107f3a5d80c6acd: Status 404 returned error can't find the container with id 8a4aecca0e7039fde6fc56dc9c824708f6e33cd9303bc98f9107f3a5d80c6acd
Nov 03 17:42:58.368717 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:58.368692    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-51fc92"]
Nov 03 17:42:58.555322 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:58.555281    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-51fc92" event={"ID":"b9fa62ee-206b-4b9f-951e-15dd710d75b0","Type":"ContainerStarted","Data":"9f3b5a51eacbc386d9ed470b35eb96218d52b6003b394533f716afd2131e17aa"}
Nov 03 17:42:58.555531 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:58.555336    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-51fc92" event={"ID":"b9fa62ee-206b-4b9f-951e-15dd710d75b0","Type":"ContainerStarted","Data":"8a4aecca0e7039fde6fc56dc9c824708f6e33cd9303bc98f9107f3a5d80c6acd"}
Nov 03 17:42:58.675334 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:58.675209    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7481-51fc92" podStartSLOduration=1.675192323 podStartE2EDuration="1.675192323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:42:58.67217875 +0000 UTC m=+1519.469891532" watchObservedRunningTime="2026-11-03 17:42:58.675192323 +0000 UTC m=+1519.472905103"
Nov 03 17:42:59.053044 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:59.053015    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a87a9d-th4lr"]
Nov 03 17:42:59.056593 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:59.056564    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a87a9d-th4lr"]
Nov 03 17:42:59.337040 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:59.336963    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edd16912-5916-4520-a958-943c07fd6626" path="/var/lib/kubelet/pods/edd16912-5916-4520-a958-943c07fd6626/volumes"
Nov 03 17:42:59.359531 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:59.359498    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:59.616484 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:59.616401    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:42:59.790067 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:42:59.790029    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:00.044374 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:00.044318    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"]
Nov 03 17:43:00.044619 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:00.044592    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2" podUID="6b11bcfe-5150-4eee-95e9-ee213aea1e8e" containerName="istio-proxy" containerID="cri-o://4408b2e1e836a0f5893d2e423dcef7e17919e11363fe9d987340558a97f27d8d" gracePeriod=30
Nov 03 17:43:00.064072 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:00.064025    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:00.239716 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:00.239667    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7481-51fc92"]
Nov 03 17:43:00.243069 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:00.243033    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7481-51fc92"]
Nov 03 17:43:00.570979 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:00.570934    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7481-51fc92" podUID="b9fa62ee-206b-4b9f-951e-15dd710d75b0" containerName="test-curl-7481-51fc92" containerID="cri-o://9f3b5a51eacbc386d9ed470b35eb96218d52b6003b394533f716afd2131e17aa" gracePeriod=2
Nov 03 17:43:00.624428 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:43:00.624383    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7481-51fc92\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="b9fa62ee-206b-4b9f-951e-15dd710d75b0" pod="opendatahub/test-curl-7481-51fc92"
Nov 03 17:43:00.719971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:00.719934    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:01.128216 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.128180    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5"]
Nov 03 17:43:01.129042 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.129016    6059 cpu_manager.go:414] "containerMap: removing container" podUID="b9fa62ee-206b-4b9f-951e-15dd710d75b0" containerName="test-curl-7481-51fc92"
Nov 03 17:43:01.129163 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.129043    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9fa62ee-206b-4b9f-951e-15dd710d75b0" containerName="test-curl-7481-51fc92"
Nov 03 17:43:01.129252 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.129237    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9fa62ee-206b-4b9f-951e-15dd710d75b0" containerName="test-curl-7481-51fc92"
Nov 03 17:43:01.133003 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.132980    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5"
Nov 03 17:43:01.187775 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.187735    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:43:01.209847 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.209793    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqf8m\" (UniqueName: \"kubernetes.io/projected/fea7fda0-d82f-4f09-8c32-cec21b718990-kube-api-access-sqf8m\") pod \"maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5\" (UID: \"fea7fda0-d82f-4f09-8c32-cec21b718990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5"
Nov 03 17:43:01.209847 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.209850    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea7fda0-d82f-4f09-8c32-cec21b718990-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5\" (UID: \"fea7fda0-d82f-4f09-8c32-cec21b718990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5"
Nov 03 17:43:01.210119 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.210101    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea7fda0-d82f-4f09-8c32-cec21b718990-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5\" (UID: \"fea7fda0-d82f-4f09-8c32-cec21b718990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5"
Nov 03 17:43:01.239711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.239651    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:43:01.239941 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.239798    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:43:01.239941 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.239848    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:43:01.244383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.244341    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5"]
Nov 03 17:43:01.304335 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:43:01.304294    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7481-51fc92\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="b9fa62ee-206b-4b9f-951e-15dd710d75b0" pod="opendatahub/test-curl-7481-51fc92"
Nov 03 17:43:01.311112 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.311072    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea7fda0-d82f-4f09-8c32-cec21b718990-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5\" (UID: \"fea7fda0-d82f-4f09-8c32-cec21b718990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5"
Nov 03 17:43:01.311974 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.311943    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqf8m\" (UniqueName: \"kubernetes.io/projected/fea7fda0-d82f-4f09-8c32-cec21b718990-kube-api-access-sqf8m\") pod \"maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5\" (UID: \"fea7fda0-d82f-4f09-8c32-cec21b718990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5"
Nov 03 17:43:01.312171 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.312139    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea7fda0-d82f-4f09-8c32-cec21b718990-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5\" (UID: \"fea7fda0-d82f-4f09-8c32-cec21b718990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5"
Nov 03 17:43:01.312297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.311946    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea7fda0-d82f-4f09-8c32-cec21b718990-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5\" (UID: \"fea7fda0-d82f-4f09-8c32-cec21b718990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5"
Nov 03 17:43:01.315225 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.315201    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea7fda0-d82f-4f09-8c32-cec21b718990-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5\" (UID: \"fea7fda0-d82f-4f09-8c32-cec21b718990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5"
Nov 03 17:43:01.346659 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.346627    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"]
Nov 03 17:43:01.352580 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.352549    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.395128 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.395029    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqf8m\" (UniqueName: \"kubernetes.io/projected/fea7fda0-d82f-4f09-8c32-cec21b718990-kube-api-access-sqf8m\") pod \"maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5\" (UID: \"fea7fda0-d82f-4f09-8c32-cec21b718990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5"
Nov 03 17:43:01.408806 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.408773    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0b55a36f-openshift-default-dockercfg-5cnxq\""
Nov 03 17:43:01.413918 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.413882    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-podinfo\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.414197 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.414165    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-data\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.414378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.414343    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-credential-socket\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.414490 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.414475    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istiod-ca-cert\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.414576 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.414545    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"]
Nov 03 17:43:01.414690 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.414657    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-workload-certs\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.415002 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.414980    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-token\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.415145 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.415127    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhcwp\" (UniqueName: \"kubernetes.io/projected/bcf48b79-ec61-4698-9c03-74dd7ff368a1-kube-api-access-qhcwp\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.415338 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.415314    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-workload-socket\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.415481 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.415442    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-envoy\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.447310 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.447274    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5"
Nov 03 17:43:01.517750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.517708    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-credential-socket\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.517885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.517777    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istiod-ca-cert\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.517885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.517829    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-workload-certs\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.518005 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.517986    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-token\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.518053 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.518038    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhcwp\" (UniqueName: \"kubernetes.io/projected/bcf48b79-ec61-4698-9c03-74dd7ff368a1-kube-api-access-qhcwp\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.518111 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.518098    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-workload-socket\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.518196 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.518182    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-envoy\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.518268 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.518255    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-podinfo\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.518301 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.518290    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-data\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.518806 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.518778    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-workload-certs\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.518929 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.518830    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-data\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.520335 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.520308    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-workload-socket\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.520335 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.520327    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-credential-socket\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.521404 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.521328    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-envoy\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.523215 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.521906    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istiod-ca-cert\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.523215 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.522935    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-podinfo\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.559279 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.559248    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:01.595826 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.595789    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhcwp\" (UniqueName: \"kubernetes.io/projected/bcf48b79-ec61-4698-9c03-74dd7ff368a1-kube-api-access-qhcwp\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.596903 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.596875    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-token\") pod \"e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") " pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.624306 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.621184    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4" podUID="44e7441c-6611-4640-9642-f67d1f3c8172" containerName="payload-processing" containerID="cri-o://a2b0250db61dfffcb5a5f13bb0d0a1059e7eae71c38951e5ca9eecb5e88fac97" gracePeriod=25
Nov 03 17:43:01.672425 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.672394    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:01.797145 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.797107    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:01.848479 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.848446    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5"]
Nov 03 17:43:01.848666 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:43:01.848635    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea7fda0_d82f_4f09_8c32_cec21b718990.slice/crio-a4cf9f9e5fd135685e814f49f3f822371af2b0c1536a73da4fb0a6fc88cc741e WatchSource:0}: Error finding container a4cf9f9e5fd135685e814f49f3f822371af2b0c1536a73da4fb0a6fc88cc741e: Status 404 returned error can't find the container with id a4cf9f9e5fd135685e814f49f3f822371af2b0c1536a73da4fb0a6fc88cc741e
Nov 03 17:43:01.921855 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:01.921824    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"
Nov 03 17:43:02.030495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.030460    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-provider-credentials\" (UniqueName: \"kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-provider-credentials\") pod \"44e7441c-6611-4640-9642-f67d1f3c8172\" (UID: \"44e7441c-6611-4640-9642-f67d1f3c8172\") "
Nov 03 17:43:02.030495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.030497    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44e7441c-6611-4640-9642-f67d1f3c8172-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44e7441c-6611-4640-9642-f67d1f3c8172-plugins-config-volume\") pod \"44e7441c-6611-4640-9642-f67d1f3c8172\" (UID: \"44e7441c-6611-4640-9642-f67d1f3c8172\") "
Nov 03 17:43:02.030708 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.030651    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44e7441c-6611-4640-9642-f67d1f3c8172-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44e7441c-6611-4640-9642-f67d1f3c8172-routing-overlay-volume\") pod \"44e7441c-6611-4640-9642-f67d1f3c8172\" (UID: \"44e7441c-6611-4640-9642-f67d1f3c8172\") "
Nov 03 17:43:02.030708 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.030703    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-kube-api-access-82fjx\" (UniqueName: \"kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-kube-api-access-82fjx\") pod \"44e7441c-6611-4640-9642-f67d1f3c8172\" (UID: \"44e7441c-6611-4640-9642-f67d1f3c8172\") "
Nov 03 17:43:02.031040 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.031007    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44e7441c-6611-4640-9642-f67d1f3c8172-plugins-config-volume" pod "44e7441c-6611-4640-9642-f67d1f3c8172" (UID: "44e7441c-6611-4640-9642-f67d1f3c8172"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:43:02.031040 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.031025    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44e7441c-6611-4640-9642-f67d1f3c8172-routing-overlay-volume" pod "44e7441c-6611-4640-9642-f67d1f3c8172" (UID: "44e7441c-6611-4640-9642-f67d1f3c8172"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:43:02.031308 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.031289    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44e7441c-6611-4640-9642-f67d1f3c8172-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:43:02.031398 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.031309    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44e7441c-6611-4640-9642-f67d1f3c8172-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:43:02.032831 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.032805    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-provider-credentials" pod "44e7441c-6611-4640-9642-f67d1f3c8172" (UID: "44e7441c-6611-4640-9642-f67d1f3c8172"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:43:02.032948 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.032902    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-kube-api-access-82fjx" pod "44e7441c-6611-4640-9642-f67d1f3c8172" (UID: "44e7441c-6611-4640-9642-f67d1f3c8172"). InnerVolumeSpecName "kube-api-access-82fjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:43:02.059491 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:43:02.059449    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf48b79_ec61_4698_9c03_74dd7ff368a1.slice/crio-a11143ffff6a319795296f8736e794c5e2b99d78b0cb2c4bfb3dec2cf5630512 WatchSource:0}: Error finding container a11143ffff6a319795296f8736e794c5e2b99d78b0cb2c4bfb3dec2cf5630512: Status 404 returned error can't find the container with id a11143ffff6a319795296f8736e794c5e2b99d78b0cb2c4bfb3dec2cf5630512
Nov 03 17:43:02.063105 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.063074    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"]
Nov 03 17:43:02.063227 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.063140    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:43:02.063227 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.063195    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:43:02.063227 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.063226    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:43:02.132268 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.132235    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82fjx\" (UniqueName: \"kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-kube-api-access-82fjx\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:43:02.132268 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.132269    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44e7441c-6611-4640-9642-f67d1f3c8172-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:43:02.595944 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.595914    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq" event={"ID":"bcf48b79-ec61-4698-9c03-74dd7ff368a1","Type":"ContainerStarted","Data":"1a3f721a9863b2f8422a71b8322d68ae1244ea1ac51c4c2d5455b235796c1fff"}
Nov 03 17:43:02.596094 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.595967    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq" event={"ID":"bcf48b79-ec61-4698-9c03-74dd7ff368a1","Type":"ContainerStarted","Data":"a11143ffff6a319795296f8736e794c5e2b99d78b0cb2c4bfb3dec2cf5630512"}
Nov 03 17:43:02.597345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.597317    6059 generic.go:356] "Generic (PLEG): container finished" podID="44e7441c-6611-4640-9642-f67d1f3c8172" containerID="a2b0250db61dfffcb5a5f13bb0d0a1059e7eae71c38951e5ca9eecb5e88fac97" exitCode=0
Nov 03 17:43:02.597492 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.597394    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"
Nov 03 17:43:02.597492 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.597401    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4" event={"ID":"44e7441c-6611-4640-9642-f67d1f3c8172","Type":"ContainerDied","Data":"a2b0250db61dfffcb5a5f13bb0d0a1059e7eae71c38951e5ca9eecb5e88fac97"}
Nov 03 17:43:02.597492 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.597453    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4" event={"ID":"44e7441c-6611-4640-9642-f67d1f3c8172","Type":"ContainerDied","Data":"97854cf1eff17c4ecfa100e1cd7b545bd85153c65695ddf7bd2f280ddbae9cc8"}
Nov 03 17:43:02.597492 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.597476    6059 scope.go:122] "RemoveContainer" containerID="a2b0250db61dfffcb5a5f13bb0d0a1059e7eae71c38951e5ca9eecb5e88fac97"
Nov 03 17:43:02.599214 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.599191    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5" event={"ID":"fea7fda0-d82f-4f09-8c32-cec21b718990","Type":"ContainerStarted","Data":"183c6e29633e9715feb25b87f542a115f615c63c2ba6f8fe999007baeaf3eddf"}
Nov 03 17:43:02.599297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.599234    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5" event={"ID":"fea7fda0-d82f-4f09-8c32-cec21b718990","Type":"ContainerStarted","Data":"a4cf9f9e5fd135685e814f49f3f822371af2b0c1536a73da4fb0a6fc88cc741e"}
Nov 03 17:43:02.642417 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.642335    6059 scope.go:122] "RemoveContainer" containerID="a2b0250db61dfffcb5a5f13bb0d0a1059e7eae71c38951e5ca9eecb5e88fac97"
Nov 03 17:43:02.642828 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:43:02.642801    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2b0250db61dfffcb5a5f13bb0d0a1059e7eae71c38951e5ca9eecb5e88fac97\": container with ID starting with a2b0250db61dfffcb5a5f13bb0d0a1059e7eae71c38951e5ca9eecb5e88fac97 not found: ID does not exist" containerID="a2b0250db61dfffcb5a5f13bb0d0a1059e7eae71c38951e5ca9eecb5e88fac97"
Nov 03 17:43:02.642912 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.642839    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2b0250db61dfffcb5a5f13bb0d0a1059e7eae71c38951e5ca9eecb5e88fac97"} err="failed to get container status \"a2b0250db61dfffcb5a5f13bb0d0a1059e7eae71c38951e5ca9eecb5e88fac97\": rpc error: code = NotFound desc = could not find container \"a2b0250db61dfffcb5a5f13bb0d0a1059e7eae71c38951e5ca9eecb5e88fac97\": container with ID starting with a2b0250db61dfffcb5a5f13bb0d0a1059e7eae71c38951e5ca9eecb5e88fac97 not found: ID does not exist"
Nov 03 17:43:02.673282 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.673248    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:02.720984 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.720925    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq" podStartSLOduration=1.720907356 podStartE2EDuration="1.720907356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:43:02.71946676 +0000 UTC m=+1523.517179535" watchObservedRunningTime="2026-11-03 17:43:02.720907356 +0000 UTC m=+1523.518620138"
Nov 03 17:43:02.793152 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.793125    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-51fc92"
Nov 03 17:43:02.852092 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.852011    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"]
Nov 03 17:43:02.856072 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.856043    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-8657f8dff8-j9qn4"]
Nov 03 17:43:02.947247 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.947210    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9fa62ee-206b-4b9f-951e-15dd710d75b0-kube-api-access-vx87f\" (UniqueName: \"kubernetes.io/projected/b9fa62ee-206b-4b9f-951e-15dd710d75b0-kube-api-access-vx87f\") pod \"b9fa62ee-206b-4b9f-951e-15dd710d75b0\" (UID: \"b9fa62ee-206b-4b9f-951e-15dd710d75b0\") "
Nov 03 17:43:02.949545 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.949513    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9fa62ee-206b-4b9f-951e-15dd710d75b0-kube-api-access-vx87f" pod "b9fa62ee-206b-4b9f-951e-15dd710d75b0" (UID: "b9fa62ee-206b-4b9f-951e-15dd710d75b0"). InnerVolumeSpecName "kube-api-access-vx87f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:43:02.970168 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:02.970115    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5" podStartSLOduration=1.970099404 podStartE2EDuration="1.970099404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:43:02.966898718 +0000 UTC m=+1523.764611500" watchObservedRunningTime="2026-11-03 17:43:02.970099404 +0000 UTC m=+1523.767812184"
Nov 03 17:43:03.049176 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:03.049142    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx87f\" (UniqueName: \"kubernetes.io/projected/b9fa62ee-206b-4b9f-951e-15dd710d75b0-kube-api-access-vx87f\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:43:03.337246 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:03.337213    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44e7441c-6611-4640-9642-f67d1f3c8172" path="/var/lib/kubelet/pods/44e7441c-6611-4640-9642-f67d1f3c8172/volumes"
Nov 03 17:43:03.337663 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:03.337648    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9fa62ee-206b-4b9f-951e-15dd710d75b0" path="/var/lib/kubelet/pods/b9fa62ee-206b-4b9f-951e-15dd710d75b0/volumes"
Nov 03 17:43:03.609645 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:03.609550    6059 generic.go:356] "Generic (PLEG): container finished" podID="fea7fda0-d82f-4f09-8c32-cec21b718990" containerID="183c6e29633e9715feb25b87f542a115f615c63c2ba6f8fe999007baeaf3eddf" exitCode=0
Nov 03 17:43:03.609824 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:03.609635    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5" event={"ID":"fea7fda0-d82f-4f09-8c32-cec21b718990","Type":"ContainerDied","Data":"183c6e29633e9715feb25b87f542a115f615c63c2ba6f8fe999007baeaf3eddf"}
Nov 03 17:43:03.610984 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:03.610963    6059 generic.go:356] "Generic (PLEG): container finished" podID="b9fa62ee-206b-4b9f-951e-15dd710d75b0" containerID="9f3b5a51eacbc386d9ed470b35eb96218d52b6003b394533f716afd2131e17aa" exitCode=137
Nov 03 17:43:03.611091 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:03.611016    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-51fc92"
Nov 03 17:43:03.611091 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:03.611057    6059 scope.go:122] "RemoveContainer" containerID="9f3b5a51eacbc386d9ed470b35eb96218d52b6003b394533f716afd2131e17aa"
Nov 03 17:43:03.621036 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:03.621012    6059 scope.go:122] "RemoveContainer" containerID="9f3b5a51eacbc386d9ed470b35eb96218d52b6003b394533f716afd2131e17aa"
Nov 03 17:43:03.621387 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:43:03.621351    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f3b5a51eacbc386d9ed470b35eb96218d52b6003b394533f716afd2131e17aa\": container with ID starting with 9f3b5a51eacbc386d9ed470b35eb96218d52b6003b394533f716afd2131e17aa not found: ID does not exist" containerID="9f3b5a51eacbc386d9ed470b35eb96218d52b6003b394533f716afd2131e17aa"
Nov 03 17:43:03.621443 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:03.621397    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3b5a51eacbc386d9ed470b35eb96218d52b6003b394533f716afd2131e17aa"} err="failed to get container status \"9f3b5a51eacbc386d9ed470b35eb96218d52b6003b394533f716afd2131e17aa\": rpc error: code = NotFound desc = could not find container \"9f3b5a51eacbc386d9ed470b35eb96218d52b6003b394533f716afd2131e17aa\": container with ID starting with 9f3b5a51eacbc386d9ed470b35eb96218d52b6003b394533f716afd2131e17aa not found: ID does not exist"
Nov 03 17:43:03.679702 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:03.679671    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:04.619591 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:04.619557    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:04.620622 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:04.620599    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:43:04.764101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:04.764051    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:04.781273 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:04.781244    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5"
Nov 03 17:43:04.872555 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:04.872456    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fea7fda0-d82f-4f09-8c32-cec21b718990-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea7fda0-d82f-4f09-8c32-cec21b718990-maas-api-xks-ca\") pod \"fea7fda0-d82f-4f09-8c32-cec21b718990\" (UID: \"fea7fda0-d82f-4f09-8c32-cec21b718990\") "
Nov 03 17:43:04.872555 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:04.872511    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea7fda0-d82f-4f09-8c32-cec21b718990-kube-api-access-sqf8m\" (UniqueName: \"kubernetes.io/projected/fea7fda0-d82f-4f09-8c32-cec21b718990-kube-api-access-sqf8m\") pod \"fea7fda0-d82f-4f09-8c32-cec21b718990\" (UID: \"fea7fda0-d82f-4f09-8c32-cec21b718990\") "
Nov 03 17:43:04.873252 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:04.873228    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fea7fda0-d82f-4f09-8c32-cec21b718990-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea7fda0-d82f-4f09-8c32-cec21b718990-maas-api-openshift-ca\") pod \"fea7fda0-d82f-4f09-8c32-cec21b718990\" (UID: \"fea7fda0-d82f-4f09-8c32-cec21b718990\") "
Nov 03 17:43:04.873822 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:04.873793    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fea7fda0-d82f-4f09-8c32-cec21b718990-maas-api-openshift-ca" pod "fea7fda0-d82f-4f09-8c32-cec21b718990" (UID: "fea7fda0-d82f-4f09-8c32-cec21b718990"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:43:04.874195 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:04.874161    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea7fda0-d82f-4f09-8c32-cec21b718990-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:43:04.877008 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:04.876966    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea7fda0-d82f-4f09-8c32-cec21b718990-kube-api-access-sqf8m" pod "fea7fda0-d82f-4f09-8c32-cec21b718990" (UID: "fea7fda0-d82f-4f09-8c32-cec21b718990"). InnerVolumeSpecName "kube-api-access-sqf8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:43:04.877532 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:04.877505    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fea7fda0-d82f-4f09-8c32-cec21b718990-maas-api-xks-ca" pod "fea7fda0-d82f-4f09-8c32-cec21b718990" (UID: "fea7fda0-d82f-4f09-8c32-cec21b718990"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:43:04.955233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:04.955194    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:04.975863 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:04.975824    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea7fda0-d82f-4f09-8c32-cec21b718990-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:43:04.975863 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:04.975860    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqf8m\" (UniqueName: \"kubernetes.io/projected/fea7fda0-d82f-4f09-8c32-cec21b718990-kube-api-access-sqf8m\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:43:05.136122 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.136036    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:05.302714 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.302687    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:43:05.380475 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.380441    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-envoy\") pod \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") "
Nov 03 17:43:05.380652 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.380511    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-credential-socket\") pod \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") "
Nov 03 17:43:05.380652 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.380556    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-workload-certs\") pod \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") "
Nov 03 17:43:05.380652 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.380639    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-podinfo\") pod \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") "
Nov 03 17:43:05.380768 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.380673    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-token\" (UniqueName: \"kubernetes.io/projected/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-token\") pod \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") "
Nov 03 17:43:05.380768 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.380759    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-kube-api-access-2msdn\" (UniqueName: \"kubernetes.io/projected/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-kube-api-access-2msdn\") pod \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") "
Nov 03 17:43:05.380865 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.380768    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-credential-socket" pod "6b11bcfe-5150-4eee-95e9-ee213aea1e8e" (UID: "6b11bcfe-5150-4eee-95e9-ee213aea1e8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:43:05.380865 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.380814    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istiod-ca-cert\") pod \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") "
Nov 03 17:43:05.380973 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.380890    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-data\") pod \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") "
Nov 03 17:43:05.380973 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.380921    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-workload-socket\") pod \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\" (UID: \"6b11bcfe-5150-4eee-95e9-ee213aea1e8e\") "
Nov 03 17:43:05.380973 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.380942    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-workload-certs" pod "6b11bcfe-5150-4eee-95e9-ee213aea1e8e" (UID: "6b11bcfe-5150-4eee-95e9-ee213aea1e8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:43:05.381165 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.381136    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-data" pod "6b11bcfe-5150-4eee-95e9-ee213aea1e8e" (UID: "6b11bcfe-5150-4eee-95e9-ee213aea1e8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:43:05.381242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.381205    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istiod-ca-cert" pod "6b11bcfe-5150-4eee-95e9-ee213aea1e8e" (UID: "6b11bcfe-5150-4eee-95e9-ee213aea1e8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:43:05.381300 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.381213    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-workload-socket" pod "6b11bcfe-5150-4eee-95e9-ee213aea1e8e" (UID: "6b11bcfe-5150-4eee-95e9-ee213aea1e8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:43:05.381552 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.381532    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:43:05.381672 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.381561    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:43:05.381672 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.381584    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:43:05.381672 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.381602    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:43:05.381672 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.381616    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:43:05.382980 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.382953    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-podinfo" pod "6b11bcfe-5150-4eee-95e9-ee213aea1e8e" (UID: "6b11bcfe-5150-4eee-95e9-ee213aea1e8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:43:05.382980 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.382965    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-token" pod "6b11bcfe-5150-4eee-95e9-ee213aea1e8e" (UID: "6b11bcfe-5150-4eee-95e9-ee213aea1e8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:43:05.382980 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.382975    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-kube-api-access-2msdn" pod "6b11bcfe-5150-4eee-95e9-ee213aea1e8e" (UID: "6b11bcfe-5150-4eee-95e9-ee213aea1e8e"). InnerVolumeSpecName "kube-api-access-2msdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:43:05.383196 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.383182    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-envoy" pod "6b11bcfe-5150-4eee-95e9-ee213aea1e8e" (UID: "6b11bcfe-5150-4eee-95e9-ee213aea1e8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:43:05.483121 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.483084    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2msdn\" (UniqueName: \"kubernetes.io/projected/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-kube-api-access-2msdn\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:43:05.483121 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.483120    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:43:05.483121 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.483130    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:43:05.483422 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.483142    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b11bcfe-5150-4eee-95e9-ee213aea1e8e-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:43:05.629371 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.629326    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5"
Nov 03 17:43:05.629796 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.629322    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5" event={"ID":"fea7fda0-d82f-4f09-8c32-cec21b718990","Type":"ContainerDied","Data":"a4cf9f9e5fd135685e814f49f3f822371af2b0c1536a73da4fb0a6fc88cc741e"}
Nov 03 17:43:05.629796 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.629477    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4cf9f9e5fd135685e814f49f3f822371af2b0c1536a73da4fb0a6fc88cc741e"
Nov 03 17:43:05.631655 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.631624    6059 generic.go:356] "Generic (PLEG): container finished" podID="6b11bcfe-5150-4eee-95e9-ee213aea1e8e" containerID="4408b2e1e836a0f5893d2e423dcef7e17919e11363fe9d987340558a97f27d8d" exitCode=0
Nov 03 17:43:05.631803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.631724    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"
Nov 03 17:43:05.631803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.631747    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2" event={"ID":"6b11bcfe-5150-4eee-95e9-ee213aea1e8e","Type":"ContainerDied","Data":"4408b2e1e836a0f5893d2e423dcef7e17919e11363fe9d987340558a97f27d8d"}
Nov 03 17:43:05.631803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.631798    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2" event={"ID":"6b11bcfe-5150-4eee-95e9-ee213aea1e8e","Type":"ContainerDied","Data":"916cfa16a0b1c61159e82dbe57f8d596a28be895c6cfa3e79842c689b6d7a0df"}
Nov 03 17:43:05.631966 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.631830    6059 scope.go:122] "RemoveContainer" containerID="4408b2e1e836a0f5893d2e423dcef7e17919e11363fe9d987340558a97f27d8d"
Nov 03 17:43:05.643684 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.643653    6059 scope.go:122] "RemoveContainer" containerID="03b25e537b3bc4296111e37a811e5ddd0c7188a88a34cc744fe549af8516d7a0"
Nov 03 17:43:05.654795 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.654770    6059 scope.go:122] "RemoveContainer" containerID="4408b2e1e836a0f5893d2e423dcef7e17919e11363fe9d987340558a97f27d8d"
Nov 03 17:43:05.655141 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:43:05.655115    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4408b2e1e836a0f5893d2e423dcef7e17919e11363fe9d987340558a97f27d8d\": container with ID starting with 4408b2e1e836a0f5893d2e423dcef7e17919e11363fe9d987340558a97f27d8d not found: ID does not exist" containerID="4408b2e1e836a0f5893d2e423dcef7e17919e11363fe9d987340558a97f27d8d"
Nov 03 17:43:05.655268 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.655152    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4408b2e1e836a0f5893d2e423dcef7e17919e11363fe9d987340558a97f27d8d"} err="failed to get container status \"4408b2e1e836a0f5893d2e423dcef7e17919e11363fe9d987340558a97f27d8d\": rpc error: code = NotFound desc = could not find container \"4408b2e1e836a0f5893d2e423dcef7e17919e11363fe9d987340558a97f27d8d\": container with ID starting with 4408b2e1e836a0f5893d2e423dcef7e17919e11363fe9d987340558a97f27d8d not found: ID does not exist"
Nov 03 17:43:05.655268 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.655172    6059 scope.go:122] "RemoveContainer" containerID="03b25e537b3bc4296111e37a811e5ddd0c7188a88a34cc744fe549af8516d7a0"
Nov 03 17:43:05.655453 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:43:05.655434    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03b25e537b3bc4296111e37a811e5ddd0c7188a88a34cc744fe549af8516d7a0\": container with ID starting with 03b25e537b3bc4296111e37a811e5ddd0c7188a88a34cc744fe549af8516d7a0 not found: ID does not exist" containerID="03b25e537b3bc4296111e37a811e5ddd0c7188a88a34cc744fe549af8516d7a0"
Nov 03 17:43:05.655509 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.655458    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b25e537b3bc4296111e37a811e5ddd0c7188a88a34cc744fe549af8516d7a0"} err="failed to get container status \"03b25e537b3bc4296111e37a811e5ddd0c7188a88a34cc744fe549af8516d7a0\": rpc error: code = NotFound desc = could not find container \"03b25e537b3bc4296111e37a811e5ddd0c7188a88a34cc744fe549af8516d7a0\": container with ID starting with 03b25e537b3bc4296111e37a811e5ddd0c7188a88a34cc744fe549af8516d7a0 not found: ID does not exist"
Nov 03 17:43:05.859972 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.859932    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"]
Nov 03 17:43:05.863793 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:05.863763    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-5f88f673-gw-openshift-default-54cc969f57-v62t2"]
Nov 03 17:43:06.942647 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:43:06.942606    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:43:07.361815 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:07.357098    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b11bcfe-5150-4eee-95e9-ee213aea1e8e" path="/var/lib/kubelet/pods/6b11bcfe-5150-4eee-95e9-ee213aea1e8e/volumes"
Nov 03 17:43:07.939558 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:07.939522    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:08.163405 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.134260    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:08.403382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.398437    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g"]
Nov 03 17:43:08.403382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.402439    6059 cpu_manager.go:414] "containerMap: removing container" podUID="6b11bcfe-5150-4eee-95e9-ee213aea1e8e" containerName="istio-proxy"
Nov 03 17:43:08.403382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.402468    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b11bcfe-5150-4eee-95e9-ee213aea1e8e" containerName="istio-proxy"
Nov 03 17:43:08.403382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.402530    6059 cpu_manager.go:414] "containerMap: removing container" podUID="fea7fda0-d82f-4f09-8c32-cec21b718990" containerName="revoke-keys"
Nov 03 17:43:08.403382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.402540    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea7fda0-d82f-4f09-8c32-cec21b718990" containerName="revoke-keys"
Nov 03 17:43:08.403382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.402588    6059 cpu_manager.go:414] "containerMap: removing container" podUID="44e7441c-6611-4640-9642-f67d1f3c8172" containerName="payload-processing"
Nov 03 17:43:08.403382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.402596    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44e7441c-6611-4640-9642-f67d1f3c8172" containerName="payload-processing"
Nov 03 17:43:08.403382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.402756    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="44e7441c-6611-4640-9642-f67d1f3c8172" containerName="payload-processing"
Nov 03 17:43:08.403382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.402773    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b11bcfe-5150-4eee-95e9-ee213aea1e8e" containerName="istio-proxy"
Nov 03 17:43:08.403382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.402792    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea7fda0-d82f-4f09-8c32-cec21b718990" containerName="revoke-keys"
Nov 03 17:43:08.413389 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.408829    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g"
Nov 03 17:43:08.486562 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.486509    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g"]
Nov 03 17:43:08.515393 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.507320    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0b55a36f\""
Nov 03 17:43:08.520422 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.510768    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0b55a36f-dockercfg-t7f79\""
Nov 03 17:43:08.526916 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.526875    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhts6\" (UniqueName: \"kubernetes.io/projected/71e9707b-d130-45a1-ba0d-e7536a4c7dbe-kube-api-access-jhts6\") pod \"payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g\" (UID: \"71e9707b-d130-45a1-ba0d-e7536a4c7dbe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g"
Nov 03 17:43:08.528638 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.527193    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71e9707b-d130-45a1-ba0d-e7536a4c7dbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g\" (UID: \"71e9707b-d130-45a1-ba0d-e7536a4c7dbe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g"
Nov 03 17:43:08.528638 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.528285    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f"]
Nov 03 17:43:08.534387 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.528906    6059 cpu_manager.go:414] "containerMap: removing container" podUID="6b11bcfe-5150-4eee-95e9-ee213aea1e8e" containerName="istio-proxy"
Nov 03 17:43:08.534387 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.528922    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b11bcfe-5150-4eee-95e9-ee213aea1e8e" containerName="istio-proxy"
Nov 03 17:43:08.534387 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.529031    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b11bcfe-5150-4eee-95e9-ee213aea1e8e" containerName="istio-proxy"
Nov 03 17:43:08.563453 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.562705    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f"
Nov 03 17:43:08.643400 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.641888    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71e9707b-d130-45a1-ba0d-e7536a4c7dbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g\" (UID: \"71e9707b-d130-45a1-ba0d-e7536a4c7dbe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g"
Nov 03 17:43:08.643400 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.641942    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9ffz\" (UniqueName: \"kubernetes.io/projected/898b2096-6730-4be9-a3a4-c18f1dd3f936-kube-api-access-l9ffz\") pod \"payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f\" (UID: \"898b2096-6730-4be9-a3a4-c18f1dd3f936\") " pod="openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f"
Nov 03 17:43:08.643400 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.642020    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhts6\" (UniqueName: \"kubernetes.io/projected/71e9707b-d130-45a1-ba0d-e7536a4c7dbe-kube-api-access-jhts6\") pod \"payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g\" (UID: \"71e9707b-d130-45a1-ba0d-e7536a4c7dbe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g"
Nov 03 17:43:08.643400 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.642052    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898b2096-6730-4be9-a3a4-c18f1dd3f936-plugins-config-volume\") pod \"payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f\" (UID: \"898b2096-6730-4be9-a3a4-c18f1dd3f936\") " pod="openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f"
Nov 03 17:43:08.722379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.721169    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71e9707b-d130-45a1-ba0d-e7536a4c7dbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g\" (UID: \"71e9707b-d130-45a1-ba0d-e7536a4c7dbe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g"
Nov 03 17:43:08.808419 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.800819    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898b2096-6730-4be9-a3a4-c18f1dd3f936-plugins-config-volume\") pod \"payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f\" (UID: \"898b2096-6730-4be9-a3a4-c18f1dd3f936\") " pod="openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f"
Nov 03 17:43:08.903384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.815589    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f"]
Nov 03 17:43:08.903883 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.836346    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898b2096-6730-4be9-a3a4-c18f1dd3f936-plugins-config-volume\") pod \"payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f\" (UID: \"898b2096-6730-4be9-a3a4-c18f1dd3f936\") " pod="openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f"
Nov 03 17:43:08.907266 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.907231    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhts6\" (UniqueName: \"kubernetes.io/projected/71e9707b-d130-45a1-ba0d-e7536a4c7dbe-kube-api-access-jhts6\") pod \"payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g\" (UID: \"71e9707b-d130-45a1-ba0d-e7536a4c7dbe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g"
Nov 03 17:43:08.921132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:08.921088    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9ffz\" (UniqueName: \"kubernetes.io/projected/898b2096-6730-4be9-a3a4-c18f1dd3f936-kube-api-access-l9ffz\") pod \"payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f\" (UID: \"898b2096-6730-4be9-a3a4-c18f1dd3f936\") " pod="openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f"
Nov 03 17:43:09.002283 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:09.002195    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9ffz\" (UniqueName: \"kubernetes.io/projected/898b2096-6730-4be9-a3a4-c18f1dd3f936-kube-api-access-l9ffz\") pod \"payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f\" (UID: \"898b2096-6730-4be9-a3a4-c18f1dd3f936\") " pod="openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f"
Nov 03 17:43:09.121593 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:09.121550    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g"
Nov 03 17:43:09.169462 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:09.165624    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f"
Nov 03 17:43:10.012555 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:10.012466    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f"]
Nov 03 17:43:10.210745 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:10.208856    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g"]
Nov 03 17:43:10.885434 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:10.868970    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:11.008378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:11.004420    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f" event={"ID":"898b2096-6730-4be9-a3a4-c18f1dd3f936","Type":"ContainerStarted","Data":"a5e40e6353b956d3427fa0f91e5b36921c3c100049b6f110c882101247bf8246"}
Nov 03 17:43:11.008378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:11.004498    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f" event={"ID":"898b2096-6730-4be9-a3a4-c18f1dd3f936","Type":"ContainerStarted","Data":"901ee6bf4ee4929d305604697fd6ea71a71793f269f8d4194a3ac6327bcf5a2f"}
Nov 03 17:43:11.008378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:11.005123    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f"
Nov 03 17:43:11.023407 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:11.022629    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g" event={"ID":"71e9707b-d130-45a1-ba0d-e7536a4c7dbe","Type":"ContainerStarted","Data":"d34d815d38e1ad86e21db1f9b90a49b8d80603747c8efb70188eabbcc039a98c"}
Nov 03 17:43:11.023407 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:11.022701    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g" event={"ID":"71e9707b-d130-45a1-ba0d-e7536a4c7dbe","Type":"ContainerStarted","Data":"17b9a96ed2eb2e9284c455b4cad0d693ab393c7faeffaac2a18d367544fc16c2"}
Nov 03 17:43:11.023650 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:11.023562    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g"
Nov 03 17:43:11.264980 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:11.264917    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g" podStartSLOduration=3.264902386 podStartE2EDuration="3.264902386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:43:11.26227497 +0000 UTC m=+1532.059987750" watchObservedRunningTime="2026-11-03 17:43:11.264902386 +0000 UTC m=+1532.062615220"
Nov 03 17:43:11.265637 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:11.265609    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f" podStartSLOduration=3.265603137 podStartE2EDuration="3.265603137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:43:11.141197522 +0000 UTC m=+1531.938910304" watchObservedRunningTime="2026-11-03 17:43:11.265603137 +0000 UTC m=+1532.063315957"
Nov 03 17:43:14.059372 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:14.059324    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk"]
Nov 03 17:43:14.432840 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:14.432753    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:14.613568 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:14.613530    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:15.804206 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:15.804158    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:17.056927 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:17.056891    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj"]
Nov 03 17:43:17.065524 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:17.065469    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17d4d4c7-mgwfj"]
Nov 03 17:43:17.337113 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:17.336977    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bc18e10-5b81-4b42-8e65-16f29e49b2c5" path="/var/lib/kubelet/pods/6bc18e10-5b81-4b42-8e65-16f29e49b2c5/volumes"
Nov 03 17:43:17.971472 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:17.971438    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"]
Nov 03 17:43:17.976759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:17.976738    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:18.057061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.057024    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-tmp-dir\") pod \"sub-test-model-af9e3c-kserve-65d9f465d7-p82jt\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:18.057474 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.057067    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-home\") pod \"sub-test-model-af9e3c-kserve-65d9f465d7-p82jt\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:18.057474 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.057217    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-model-cache\") pod \"sub-test-model-af9e3c-kserve-65d9f465d7-p82jt\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:18.057474 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.057254    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-dshm\") pod \"sub-test-model-af9e3c-kserve-65d9f465d7-p82jt\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:18.057474 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.057288    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/708810ff-e592-4d83-a6a3-1f2c313499a0-tls-certs\") pod \"sub-test-model-af9e3c-kserve-65d9f465d7-p82jt\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:18.057474 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.057409    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tngcz\" (UniqueName: \"kubernetes.io/projected/708810ff-e592-4d83-a6a3-1f2c313499a0-kube-api-access-tngcz\") pod \"sub-test-model-af9e3c-kserve-65d9f465d7-p82jt\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:18.083496 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.083461    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-af9e3c\"/\"kube-root-ca.crt\""
Nov 03 17:43:18.083496 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.083491    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-af9e3c\"/\"openshift-service-ca.crt\""
Nov 03 17:43:18.083750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.083614    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-af9e3c\"/\"default-dockercfg-bl29d\""
Nov 03 17:43:18.084043 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.084025    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-af9e3c\"/\"sub-test-model-af9e3c-kserve-self-signed-certs\""
Nov 03 17:43:18.089771 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.089740    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"]
Nov 03 17:43:18.092937 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.092910    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:18.159455 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.159419    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-tmp-dir\") pod \"sub-test-model-af9e3c-kserve-65d9f465d7-p82jt\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:18.159671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.159465    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-home\") pod \"sub-test-model-af9e3c-kserve-65d9f465d7-p82jt\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:18.159671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.159533    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-model-cache\") pod \"sub-test-model-af9e3c-kserve-65d9f465d7-p82jt\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:18.159671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.159551    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-dshm\") pod \"sub-test-model-af9e3c-kserve-65d9f465d7-p82jt\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:18.159671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.159577    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/708810ff-e592-4d83-a6a3-1f2c313499a0-tls-certs\") pod \"sub-test-model-af9e3c-kserve-65d9f465d7-p82jt\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:18.159671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.159640    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tngcz\" (UniqueName: \"kubernetes.io/projected/708810ff-e592-4d83-a6a3-1f2c313499a0-kube-api-access-tngcz\") pod \"sub-test-model-af9e3c-kserve-65d9f465d7-p82jt\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:18.159968 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.159915    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-tmp-dir\") pod \"sub-test-model-af9e3c-kserve-65d9f465d7-p82jt\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:18.160094 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.159978    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-home\") pod \"sub-test-model-af9e3c-kserve-65d9f465d7-p82jt\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:18.160094 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.159991    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-model-cache\") pod \"sub-test-model-af9e3c-kserve-65d9f465d7-p82jt\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:18.162432 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.162402    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-dshm\") pod \"sub-test-model-af9e3c-kserve-65d9f465d7-p82jt\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:18.162662 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.162637    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/708810ff-e592-4d83-a6a3-1f2c313499a0-tls-certs\") pod \"sub-test-model-af9e3c-kserve-65d9f465d7-p82jt\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:18.222615 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.222545    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tngcz\" (UniqueName: \"kubernetes.io/projected/708810ff-e592-4d83-a6a3-1f2c313499a0-kube-api-access-tngcz\") pod \"sub-test-model-af9e3c-kserve-65d9f465d7-p82jt\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:18.290637 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.290598    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:18.679930 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:18.679907    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"]
Nov 03 17:43:19.060324 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:19.060270    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk" podUID="f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7" containerName="payload-processing" containerID="cri-o://6f60ddef030c81cfcf5e5981e7e23938180e225fe28aab91b3fff83b6c286e3b" gracePeriod=25
Nov 03 17:43:19.088621 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:19.088581    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt" event={"ID":"708810ff-e592-4d83-a6a3-1f2c313499a0","Type":"ContainerStarted","Data":"874c51cdc04ee6c22b1588135c49c533183b3bb17c4730bb87bd7fae9ec70167"}
Nov 03 17:43:19.088814 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:19.088634    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt" event={"ID":"708810ff-e592-4d83-a6a3-1f2c313499a0","Type":"ContainerStarted","Data":"6c3461bc3dc044501e1c5d7fbdf2bb28f8e57898cba9f7e8fa942913fbc00ac7"}
Nov 03 17:43:19.213502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:19.213425    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt" podStartSLOduration=2.213405663 podStartE2EDuration="2.213405663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:43:19.209558155 +0000 UTC m=+1540.007270961" watchObservedRunningTime="2026-11-03 17:43:19.213405663 +0000 UTC m=+1540.011118445"
Nov 03 17:43:19.290860 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:19.290823    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:20.522426 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:20.522384    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:21.445304 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:21.445266    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:22.034030 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:22.034002    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f"
Nov 03 17:43:23.042531 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:23.042492    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g"
Nov 03 17:43:23.199866 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:23.199804    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk" podUID="f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 17:43:25.050438 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:25.050389    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:26.818916 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:26.818883    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"]
Nov 03 17:43:26.823147 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:26.823127    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:26.901279 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:26.901233    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-workload-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:26.901479 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:26.901292    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtxx2\" (UniqueName: \"kubernetes.io/projected/24835092-2c35-423e-990a-1314dde40f2a-kube-api-access-vtxx2\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:26.901479 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:26.901408    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-workload-certs\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:26.901479 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:26.901446    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-credential-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:26.901600 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:26.901550    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-istio-envoy\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:26.901652 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:26.901602    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24835092-2c35-423e-990a-1314dde40f2a-istio-podinfo\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:26.901694 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:26.901686    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24835092-2c35-423e-990a-1314dde40f2a-istio-token\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:26.901743 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:26.901729    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-istio-data\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:26.901786 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:26.901757    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24835092-2c35-423e-990a-1314dde40f2a-istiod-ca-cert\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:26.948543 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:26.948498    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"]
Nov 03 17:43:27.002693 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.002645    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-workload-certs\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:27.002693 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.002696    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-credential-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:27.002975 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.002750    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-istio-envoy\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:27.002975 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.002806    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24835092-2c35-423e-990a-1314dde40f2a-istio-podinfo\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:27.002975 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.002849    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24835092-2c35-423e-990a-1314dde40f2a-istio-token\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:27.002975 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.002922    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-istio-data\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:27.002975 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.002951    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24835092-2c35-423e-990a-1314dde40f2a-istiod-ca-cert\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:27.003237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.003091    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-workload-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:27.003237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.003150    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtxx2\" (UniqueName: \"kubernetes.io/projected/24835092-2c35-423e-990a-1314dde40f2a-kube-api-access-vtxx2\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:27.003237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.003162    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-workload-certs\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:27.003237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.003208    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-credential-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:27.003464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.003342    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-istio-data\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:27.003557 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.003537    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-workload-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:27.003978 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.003954    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24835092-2c35-423e-990a-1314dde40f2a-istiod-ca-cert\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:27.005460 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.005438    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24835092-2c35-423e-990a-1314dde40f2a-istio-podinfo\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:27.005460 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.005452    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-istio-envoy\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:27.066975 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.066944    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24835092-2c35-423e-990a-1314dde40f2a-istio-token\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:27.067129 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.067094    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtxx2\" (UniqueName: \"kubernetes.io/projected/24835092-2c35-423e-990a-1314dde40f2a-kube-api-access-vtxx2\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:27.138312 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.138215    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:27.569161 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:43:27.569131    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24835092_2c35_423e_990a_1314dde40f2a.slice/crio-00f6be4bea14dba1a12c20350c32e4638ba7a7427c9612c77da9f13423597c96 WatchSource:0}: Error finding container 00f6be4bea14dba1a12c20350c32e4638ba7a7427c9612c77da9f13423597c96: Status 404 returned error can't find the container with id 00f6be4bea14dba1a12c20350c32e4638ba7a7427c9612c77da9f13423597c96
Nov 03 17:43:27.572044 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.572002    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:43:27.572204 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.572092    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:43:27.572204 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.572134    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:43:27.572600 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:27.572577    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"]
Nov 03 17:43:28.163545 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:28.163507    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68" event={"ID":"24835092-2c35-423e-990a-1314dde40f2a","Type":"ContainerStarted","Data":"81744c8b9c10b3e9122d4856c22239fe14901e3cab5b6ec591a445520d97138e"}
Nov 03 17:43:28.163943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:28.163571    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68" event={"ID":"24835092-2c35-423e-990a-1314dde40f2a","Type":"ContainerStarted","Data":"00f6be4bea14dba1a12c20350c32e4638ba7a7427c9612c77da9f13423597c96"}
Nov 03 17:43:28.293420 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:28.292928    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:28.302951 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:28.302872    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68" podStartSLOduration=2.302848334 podStartE2EDuration="2.302848334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:43:28.294595782 +0000 UTC m=+1549.092308565" watchObservedRunningTime="2026-11-03 17:43:28.302848334 +0000 UTC m=+1549.100561117"
Nov 03 17:43:28.315395 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:28.315342    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:29.150382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:29.149777    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:29.154569 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:29.154531    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 17:43:29.154747 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:29.154592    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68" podUID="24835092-2c35-423e-990a-1314dde40f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 17:43:29.182412 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:29.182382    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:43:29.614776 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:29.614739    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:30.158861 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:30.158815    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 17:43:30.159084 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:30.158886    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68" podUID="24835092-2c35-423e-990a-1314dde40f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 17:43:30.159869 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:30.159838    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln"]
Nov 03 17:43:30.202212 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:30.202178    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d1f969b-flgln"]
Nov 03 17:43:31.140843 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:31.140801    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 17:43:31.141120 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:31.141091    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68" podUID="24835092-2c35-423e-990a-1314dde40f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 17:43:31.362900 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:31.362864    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="436e4e0c-365c-42e6-a646-424b167a7686" path="/var/lib/kubelet/pods/436e4e0c-365c-42e6-a646-424b167a7686/volumes"
Nov 03 17:43:32.150876 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:32.150832    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 17:43:32.151962 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:32.151929    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68" podUID="24835092-2c35-423e-990a-1314dde40f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 17:43:33.154825 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:33.154771    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 17:43:33.155393 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:33.154841    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68" podUID="24835092-2c35-423e-990a-1314dde40f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 17:43:33.199168 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:33.199128    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk" podUID="f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 17:43:34.139263 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:34.139227    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 17:43:34.139603 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:34.139291    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68" podUID="24835092-2c35-423e-990a-1314dde40f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 17:43:34.314572 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:34.314533    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:35.143536 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:35.143502    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:35.144010 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:35.143990    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:35.144767 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:35.144748    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:43:36.862524 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:43:36.862478    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:43:37.399070 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:37.399032    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:38.930226 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:38.930178    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:39.169768 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:39.169716    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:39.280308 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:39.280269    6059 generic.go:356] "Generic (PLEG): container finished" podID="f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7" containerID="6f60ddef030c81cfcf5e5981e7e23938180e225fe28aab91b3fff83b6c286e3b" exitCode=0
Nov 03 17:43:39.280549 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:39.280486    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk" event={"ID":"f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7","Type":"ContainerDied","Data":"6f60ddef030c81cfcf5e5981e7e23938180e225fe28aab91b3fff83b6c286e3b"}
Nov 03 17:43:39.369602 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:39.369572    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk"
Nov 03 17:43:39.506206 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:39.505483    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7-plugins-config-volume\") pod \"f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7\" (UID: \"f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7\") "
Nov 03 17:43:39.506206 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:39.505648    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7-kube-api-access-qfxpw\" (UniqueName: \"kubernetes.io/projected/f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7-kube-api-access-qfxpw\") pod \"f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7\" (UID: \"f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7\") "
Nov 03 17:43:39.506206 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:39.505880    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7-plugins-config-volume" pod "f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7" (UID: "f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:43:39.506206 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:39.506162    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:43:39.507903 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:39.507867    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7-kube-api-access-qfxpw" pod "f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7" (UID: "f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7"). InnerVolumeSpecName "kube-api-access-qfxpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:43:39.607283 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:39.607234    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfxpw\" (UniqueName: \"kubernetes.io/projected/f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7-kube-api-access-qfxpw\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:43:40.290319 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:40.290286    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk"
Nov 03 17:43:40.290743 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:40.290327    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk" event={"ID":"f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7","Type":"ContainerDied","Data":"5aca8d0936308d0e15f2eae16cd42c25723de8c79256c5c4139bb07650a7e558"}
Nov 03 17:43:40.290743 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:40.290450    6059 scope.go:122] "RemoveContainer" containerID="6f60ddef030c81cfcf5e5981e7e23938180e225fe28aab91b3fff83b6c286e3b"
Nov 03 17:43:40.470492 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:40.470456    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk"]
Nov 03 17:43:40.522653 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:40.522613    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-vgvqk"]
Nov 03 17:43:41.337038 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:41.336994    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7" path="/var/lib/kubelet/pods/f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7/volumes"
Nov 03 17:43:41.427138 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:41.427108    6059 scope.go:122] "RemoveContainer" containerID="cb2a892a4bc364f4ed976b362872798182c0fcc266532347d93b6b19d133a5c6"
Nov 03 17:43:41.438243 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:41.438218    6059 scope.go:122] "RemoveContainer" containerID="144c663c6dc49a7945b4106f39a617a49e82263a9ae8c3b97f47e79a9b60c847"
Nov 03 17:43:41.448225 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:41.448203    6059 scope.go:122] "RemoveContainer" containerID="7919b8d4e26d095ac840596d9fefa1deba5bcee40965f3219edd26ab52076ffb"
Nov 03 17:43:41.458259 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:41.458224    6059 scope.go:122] "RemoveContainer" containerID="c7a8ff48df11ea25cec701febf092047dc8d21740b12801089c52f071a52468e"
Nov 03 17:43:41.468348 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:41.468328    6059 scope.go:122] "RemoveContainer" containerID="9eaa35bfda27baba1cc2c713469405a3d9a81c5e8591daeb8c63d257e0f5e852"
Nov 03 17:43:41.478268 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:41.478245    6059 scope.go:122] "RemoveContainer" containerID="89237ad86fc9faa9c3e4f99ad64b052e14caf0fa44f2de3018fa0227f73be61d"
Nov 03 17:43:43.386057 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:43.386019    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:43.603290 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:43.603254    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:43.827202 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:43.827162    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:48.406833 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:48.406793    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:49.065655 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:49.065617    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c"]
Nov 03 17:43:49.116881 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:49.116847    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4bf8108-d4j9c"]
Nov 03 17:43:49.336681 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:49.336590    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a6a7bae-214b-4470-af84-7d5301433e6d" path="/var/lib/kubelet/pods/7a6a7bae-214b-4470-af84-7d5301433e6d/volumes"
Nov 03 17:43:52.443340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:52.443304    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa5e2fd-pjg2p"]
Nov 03 17:43:52.443968 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:52.443951    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7" containerName="payload-processing"
Nov 03 17:43:52.444014 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:52.443969    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7" containerName="payload-processing"
Nov 03 17:43:52.444109 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:52.444099    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f85fa0a3-8fad-44ae-8ce4-1ab4461b38a7" containerName="payload-processing"
Nov 03 17:43:52.449676 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:52.449654    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa5e2fd-pjg2p"
Nov 03 17:43:52.505768 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:52.505722    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvxcb\" (UniqueName: \"kubernetes.io/projected/4789e44b-2cec-46b9-95fe-68fab84f0cfb-kube-api-access-vvxcb\") pod \"maas-api-revoke-sub-1fa5e2fd-pjg2p\" (UID: \"4789e44b-2cec-46b9-95fe-68fab84f0cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa5e2fd-pjg2p"
Nov 03 17:43:52.505947 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:52.505780    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4789e44b-2cec-46b9-95fe-68fab84f0cfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fa5e2fd-pjg2p\" (UID: \"4789e44b-2cec-46b9-95fe-68fab84f0cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa5e2fd-pjg2p"
Nov 03 17:43:52.506056 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:52.506038    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4789e44b-2cec-46b9-95fe-68fab84f0cfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fa5e2fd-pjg2p\" (UID: \"4789e44b-2cec-46b9-95fe-68fab84f0cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa5e2fd-pjg2p"
Nov 03 17:43:52.508986 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:52.508964    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:43:52.558533 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:52.558492    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:43:52.558533 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:52.558493    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:43:52.558789 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:52.558493    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:43:52.562362 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:52.562324    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa5e2fd-pjg2p"]
Nov 03 17:43:52.607850 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:52.607810    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4789e44b-2cec-46b9-95fe-68fab84f0cfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fa5e2fd-pjg2p\" (UID: \"4789e44b-2cec-46b9-95fe-68fab84f0cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa5e2fd-pjg2p"
Nov 03 17:43:52.608034 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:52.607869    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvxcb\" (UniqueName: \"kubernetes.io/projected/4789e44b-2cec-46b9-95fe-68fab84f0cfb-kube-api-access-vvxcb\") pod \"maas-api-revoke-sub-1fa5e2fd-pjg2p\" (UID: \"4789e44b-2cec-46b9-95fe-68fab84f0cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa5e2fd-pjg2p"
Nov 03 17:43:52.608034 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:52.607908    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4789e44b-2cec-46b9-95fe-68fab84f0cfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fa5e2fd-pjg2p\" (UID: \"4789e44b-2cec-46b9-95fe-68fab84f0cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa5e2fd-pjg2p"
Nov 03 17:43:52.608628 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:52.608597    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4789e44b-2cec-46b9-95fe-68fab84f0cfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fa5e2fd-pjg2p\" (UID: \"4789e44b-2cec-46b9-95fe-68fab84f0cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa5e2fd-pjg2p"
Nov 03 17:43:52.610396 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:52.610370    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4789e44b-2cec-46b9-95fe-68fab84f0cfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fa5e2fd-pjg2p\" (UID: \"4789e44b-2cec-46b9-95fe-68fab84f0cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa5e2fd-pjg2p"
Nov 03 17:43:52.671914 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:52.671875    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvxcb\" (UniqueName: \"kubernetes.io/projected/4789e44b-2cec-46b9-95fe-68fab84f0cfb-kube-api-access-vvxcb\") pod \"maas-api-revoke-sub-1fa5e2fd-pjg2p\" (UID: \"4789e44b-2cec-46b9-95fe-68fab84f0cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa5e2fd-pjg2p"
Nov 03 17:43:52.764123 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:52.764077    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa5e2fd-pjg2p"
Nov 03 17:43:53.134561 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:53.134467    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:53.147347 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:43:53.147288    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4789e44b_2cec_46b9_95fe_68fab84f0cfb.slice/crio-4d0a0417a21102a80c5b1fa74a9894b518e92009455efaad11c5a328fa3b8e4b WatchSource:0}: Error finding container 4d0a0417a21102a80c5b1fa74a9894b518e92009455efaad11c5a328fa3b8e4b: Status 404 returned error can't find the container with id 4d0a0417a21102a80c5b1fa74a9894b518e92009455efaad11c5a328fa3b8e4b
Nov 03 17:43:53.150612 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:53.150461    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa5e2fd-pjg2p"]
Nov 03 17:43:53.398069 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:53.397978    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa5e2fd-pjg2p" event={"ID":"4789e44b-2cec-46b9-95fe-68fab84f0cfb","Type":"ContainerStarted","Data":"4d0a0417a21102a80c5b1fa74a9894b518e92009455efaad11c5a328fa3b8e4b"}
Nov 03 17:43:53.452750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:53.452715    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:43:54.407457 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:54.407418    6059 generic.go:356] "Generic (PLEG): container finished" podID="4789e44b-2cec-46b9-95fe-68fab84f0cfb" containerID="0525af719d6fce866705e8e7a281407d02fb324b0b25ff07a1ddf7d0202cac0c" exitCode=0
Nov 03 17:43:54.407604 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:54.407473    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa5e2fd-pjg2p" event={"ID":"4789e44b-2cec-46b9-95fe-68fab84f0cfb","Type":"ContainerDied","Data":"0525af719d6fce866705e8e7a281407d02fb324b0b25ff07a1ddf7d0202cac0c"}
Nov 03 17:43:55.579645 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:55.579617    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa5e2fd-pjg2p"
Nov 03 17:43:55.645598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:55.645550    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4789e44b-2cec-46b9-95fe-68fab84f0cfb-kube-api-access-vvxcb\" (UniqueName: \"kubernetes.io/projected/4789e44b-2cec-46b9-95fe-68fab84f0cfb-kube-api-access-vvxcb\") pod \"4789e44b-2cec-46b9-95fe-68fab84f0cfb\" (UID: \"4789e44b-2cec-46b9-95fe-68fab84f0cfb\") "
Nov 03 17:43:55.645813 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:55.645681    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4789e44b-2cec-46b9-95fe-68fab84f0cfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4789e44b-2cec-46b9-95fe-68fab84f0cfb-maas-api-openshift-ca\") pod \"4789e44b-2cec-46b9-95fe-68fab84f0cfb\" (UID: \"4789e44b-2cec-46b9-95fe-68fab84f0cfb\") "
Nov 03 17:43:55.645938 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:55.645921    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4789e44b-2cec-46b9-95fe-68fab84f0cfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4789e44b-2cec-46b9-95fe-68fab84f0cfb-maas-api-xks-ca\") pod \"4789e44b-2cec-46b9-95fe-68fab84f0cfb\" (UID: \"4789e44b-2cec-46b9-95fe-68fab84f0cfb\") "
Nov 03 17:43:55.646068 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:55.646036    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4789e44b-2cec-46b9-95fe-68fab84f0cfb-maas-api-openshift-ca" pod "4789e44b-2cec-46b9-95fe-68fab84f0cfb" (UID: "4789e44b-2cec-46b9-95fe-68fab84f0cfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:43:55.646446 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:55.646428    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4789e44b-2cec-46b9-95fe-68fab84f0cfb-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:43:55.647727 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:55.647694    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4789e44b-2cec-46b9-95fe-68fab84f0cfb-kube-api-access-vvxcb" pod "4789e44b-2cec-46b9-95fe-68fab84f0cfb" (UID: "4789e44b-2cec-46b9-95fe-68fab84f0cfb"). InnerVolumeSpecName "kube-api-access-vvxcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:43:55.647835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:55.647787    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4789e44b-2cec-46b9-95fe-68fab84f0cfb-maas-api-xks-ca" pod "4789e44b-2cec-46b9-95fe-68fab84f0cfb" (UID: "4789e44b-2cec-46b9-95fe-68fab84f0cfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:43:55.748143 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:55.748101    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4789e44b-2cec-46b9-95fe-68fab84f0cfb-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:43:55.748143 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:55.748137    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvxcb\" (UniqueName: \"kubernetes.io/projected/4789e44b-2cec-46b9-95fe-68fab84f0cfb-kube-api-access-vvxcb\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:43:56.428440 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:56.428410    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa5e2fd-pjg2p"
Nov 03 17:43:56.428440 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:56.428416    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa5e2fd-pjg2p" event={"ID":"4789e44b-2cec-46b9-95fe-68fab84f0cfb","Type":"ContainerDied","Data":"4d0a0417a21102a80c5b1fa74a9894b518e92009455efaad11c5a328fa3b8e4b"}
Nov 03 17:43:56.428731 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:56.428469    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d0a0417a21102a80c5b1fa74a9894b518e92009455efaad11c5a328fa3b8e4b"
Nov 03 17:43:57.642692 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:43:57.642657    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:00.064587 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:00.064553    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ab418c-prc9k"]
Nov 03 17:44:00.116366 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:00.116313    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ab418c-prc9k"]
Nov 03 17:44:01.336477 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:01.336438    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdeae13a-f70e-4e1d-8202-d6558ce9b81a" path="/var/lib/kubelet/pods/fdeae13a-f70e-4e1d-8202-d6558ce9b81a/volumes"
Nov 03 17:44:02.256471 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:02.256426    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:05.062921 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:05.062881    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw"]
Nov 03 17:44:05.112703 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:05.112662    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd19cff0-rmtdw"]
Nov 03 17:44:05.361385 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:05.361293    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2234b5bb-0e99-44f3-b050-8f190df93d6e" path="/var/lib/kubelet/pods/2234b5bb-0e99-44f3-b050-8f190df93d6e/volumes"
Nov 03 17:44:06.880495 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:44:06.880456    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:44:07.061902 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:07.061863    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:11.675644 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:11.675047    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:12.551035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:12.550998    6059 generic.go:356] "Generic (PLEG): container finished" podID="bcf48b79-ec61-4698-9c03-74dd7ff368a1" containerID="1a3f721a9863b2f8422a71b8322d68ae1244ea1ac51c4c2d5455b235796c1fff" exitCode=0
Nov 03 17:44:12.551223 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:12.551069    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq" event={"ID":"bcf48b79-ec61-4698-9c03-74dd7ff368a1","Type":"ContainerDied","Data":"1a3f721a9863b2f8422a71b8322d68ae1244ea1ac51c4c2d5455b235796c1fff"}
Nov 03 17:44:12.551543 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:12.551528    6059 scope.go:122] "RemoveContainer" containerID="1a3f721a9863b2f8422a71b8322d68ae1244ea1ac51c4c2d5455b235796c1fff"
Nov 03 17:44:12.553092 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:12.553066    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:44:12.553154 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:12.553132    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:44:12.553190 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:12.553165    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:44:13.562458 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:13.562419    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq" event={"ID":"bcf48b79-ec61-4698-9c03-74dd7ff368a1","Type":"ContainerStarted","Data":"078807d2690b5a86b55e34a577414bcfed122a1b77650e82a9f186359a269fc2"}
Nov 03 17:44:13.672935 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:13.672898    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:44:13.673268 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:13.673243    6059 patch_prober.go:29] interesting pod/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 17:44:13.673417 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:13.673308    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq" podUID="bcf48b79-ec61-4698-9c03-74dd7ff368a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 17:44:14.673247 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:14.673212    6059 patch_prober.go:29] interesting pod/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 17:44:14.673717 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:14.673269    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq" podUID="bcf48b79-ec61-4698-9c03-74dd7ff368a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 17:44:15.672980 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:15.672939    6059 patch_prober.go:29] interesting pod/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 17:44:15.673168 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:15.673007    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq" podUID="bcf48b79-ec61-4698-9c03-74dd7ff368a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 17:44:16.247961 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:16.247923    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:16.673824 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:16.673733    6059 patch_prober.go:29] interesting pod/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 17:44:16.673824 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:16.673813    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq" podUID="bcf48b79-ec61-4698-9c03-74dd7ff368a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 17:44:17.673800 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:17.673762    6059 patch_prober.go:29] interesting pod/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 17:44:17.674233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:17.673835    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq" podUID="bcf48b79-ec61-4698-9c03-74dd7ff368a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 17:44:18.673229 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:18.673193    6059 patch_prober.go:29] interesting pod/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 17:44:18.673445 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:18.673266    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq" podUID="bcf48b79-ec61-4698-9c03-74dd7ff368a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 17:44:19.619788 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:19.619754    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:44:19.678135 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:19.678099    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:44:20.053058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:20.053017    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl"]
Nov 03 17:44:20.056339 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:20.056311    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80282998-zdfcl"]
Nov 03 17:44:20.615387 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:20.615329    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:44:21.026039 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:21.026007    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:21.337385 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:21.337274    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ff1efbd-eeae-4b26-9cb8-160edf44d4b3" path="/var/lib/kubelet/pods/3ff1efbd-eeae-4b26-9cb8-160edf44d4b3/volumes"
Nov 03 17:44:25.881447 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:25.881415    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:27.973140 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:27.973109    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz"]
Nov 03 17:44:27.973726 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:27.973709    6059 cpu_manager.go:414] "containerMap: removing container" podUID="4789e44b-2cec-46b9-95fe-68fab84f0cfb" containerName="revoke-keys"
Nov 03 17:44:27.973784 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:27.973737    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4789e44b-2cec-46b9-95fe-68fab84f0cfb" containerName="revoke-keys"
Nov 03 17:44:27.973916 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:27.973905    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4789e44b-2cec-46b9-95fe-68fab84f0cfb" containerName="revoke-keys"
Nov 03 17:44:27.977619 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:27.977598    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz"
Nov 03 17:44:28.024985 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:28.024949    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a42fd008-dd2d-43e4-b3b9-01459025a3f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz\" (UID: \"a42fd008-dd2d-43e4-b3b9-01459025a3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz"
Nov 03 17:44:28.024985 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:28.024993    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a42fd008-dd2d-43e4-b3b9-01459025a3f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz\" (UID: \"a42fd008-dd2d-43e4-b3b9-01459025a3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz"
Nov 03 17:44:28.025236 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:28.025150    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgv46\" (UniqueName: \"kubernetes.io/projected/a42fd008-dd2d-43e4-b3b9-01459025a3f8-kube-api-access-qgv46\") pod \"maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz\" (UID: \"a42fd008-dd2d-43e4-b3b9-01459025a3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz"
Nov 03 17:44:28.036265 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:28.036228    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz"]
Nov 03 17:44:28.083319 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:28.083198    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:44:28.083319 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:28.083258    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:44:28.083319 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:28.083258    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:44:28.083616 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:28.083368    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:44:28.126900 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:28.126856    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a42fd008-dd2d-43e4-b3b9-01459025a3f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz\" (UID: \"a42fd008-dd2d-43e4-b3b9-01459025a3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz"
Nov 03 17:44:28.126900 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:28.126899    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a42fd008-dd2d-43e4-b3b9-01459025a3f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz\" (UID: \"a42fd008-dd2d-43e4-b3b9-01459025a3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz"
Nov 03 17:44:28.127169 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:28.126960    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgv46\" (UniqueName: \"kubernetes.io/projected/a42fd008-dd2d-43e4-b3b9-01459025a3f8-kube-api-access-qgv46\") pod \"maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz\" (UID: \"a42fd008-dd2d-43e4-b3b9-01459025a3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz"
Nov 03 17:44:28.127617 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:28.127590    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a42fd008-dd2d-43e4-b3b9-01459025a3f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz\" (UID: \"a42fd008-dd2d-43e4-b3b9-01459025a3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz"
Nov 03 17:44:28.129262 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:28.129237    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a42fd008-dd2d-43e4-b3b9-01459025a3f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz\" (UID: \"a42fd008-dd2d-43e4-b3b9-01459025a3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz"
Nov 03 17:44:28.195064 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:28.195026    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgv46\" (UniqueName: \"kubernetes.io/projected/a42fd008-dd2d-43e4-b3b9-01459025a3f8-kube-api-access-qgv46\") pod \"maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz\" (UID: \"a42fd008-dd2d-43e4-b3b9-01459025a3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz"
Nov 03 17:44:28.291656 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:28.291567    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz"
Nov 03 17:44:28.677015 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:44:28.676975    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda42fd008_dd2d_43e4_b3b9_01459025a3f8.slice/crio-6568085d76fecebaa41c14d7fb0588f53da967a8720755f834890a40969ea197 WatchSource:0}: Error finding container 6568085d76fecebaa41c14d7fb0588f53da967a8720755f834890a40969ea197: Status 404 returned error can't find the container with id 6568085d76fecebaa41c14d7fb0588f53da967a8720755f834890a40969ea197
Nov 03 17:44:28.680531 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:28.680507    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz"]
Nov 03 17:44:29.066565 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:29.066527    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf"]
Nov 03 17:44:29.117644 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:29.117603    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-511b856d-mmbmf"]
Nov 03 17:44:29.348842 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:29.348805    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7972cf5-71cb-436b-bc1c-754a24d025fc" path="/var/lib/kubelet/pods/d7972cf5-71cb-436b-bc1c-754a24d025fc/volumes"
Nov 03 17:44:29.684556 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:29.684467    6059 generic.go:356] "Generic (PLEG): container finished" podID="a42fd008-dd2d-43e4-b3b9-01459025a3f8" containerID="806f25f785ba4c98c7f914fc385fee5612d7cba55bc765363229fd99d58883ff" exitCode=0
Nov 03 17:44:29.684556 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:29.684510    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz" event={"ID":"a42fd008-dd2d-43e4-b3b9-01459025a3f8","Type":"ContainerDied","Data":"806f25f785ba4c98c7f914fc385fee5612d7cba55bc765363229fd99d58883ff"}
Nov 03 17:44:29.684773 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:29.684565    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz" event={"ID":"a42fd008-dd2d-43e4-b3b9-01459025a3f8","Type":"ContainerStarted","Data":"6568085d76fecebaa41c14d7fb0588f53da967a8720755f834890a40969ea197"}
Nov 03 17:44:30.838753 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:30.838720    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:30.871166 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:30.871131    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz"
Nov 03 17:44:30.967112 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:30.967077    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a42fd008-dd2d-43e4-b3b9-01459025a3f8-kube-api-access-qgv46\" (UniqueName: \"kubernetes.io/projected/a42fd008-dd2d-43e4-b3b9-01459025a3f8-kube-api-access-qgv46\") pod \"a42fd008-dd2d-43e4-b3b9-01459025a3f8\" (UID: \"a42fd008-dd2d-43e4-b3b9-01459025a3f8\") "
Nov 03 17:44:30.967306 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:30.967161    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a42fd008-dd2d-43e4-b3b9-01459025a3f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a42fd008-dd2d-43e4-b3b9-01459025a3f8-maas-api-xks-ca\") pod \"a42fd008-dd2d-43e4-b3b9-01459025a3f8\" (UID: \"a42fd008-dd2d-43e4-b3b9-01459025a3f8\") "
Nov 03 17:44:30.967367 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:30.967338    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a42fd008-dd2d-43e4-b3b9-01459025a3f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a42fd008-dd2d-43e4-b3b9-01459025a3f8-maas-api-openshift-ca\") pod \"a42fd008-dd2d-43e4-b3b9-01459025a3f8\" (UID: \"a42fd008-dd2d-43e4-b3b9-01459025a3f8\") "
Nov 03 17:44:30.967730 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:30.967699    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a42fd008-dd2d-43e4-b3b9-01459025a3f8-maas-api-openshift-ca" pod "a42fd008-dd2d-43e4-b3b9-01459025a3f8" (UID: "a42fd008-dd2d-43e4-b3b9-01459025a3f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:44:30.967880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:30.967863    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a42fd008-dd2d-43e4-b3b9-01459025a3f8-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:44:30.969320 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:30.969298    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a42fd008-dd2d-43e4-b3b9-01459025a3f8-kube-api-access-qgv46" pod "a42fd008-dd2d-43e4-b3b9-01459025a3f8" (UID: "a42fd008-dd2d-43e4-b3b9-01459025a3f8"). InnerVolumeSpecName "kube-api-access-qgv46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:44:30.969443 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:30.969373    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a42fd008-dd2d-43e4-b3b9-01459025a3f8-maas-api-xks-ca" pod "a42fd008-dd2d-43e4-b3b9-01459025a3f8" (UID: "a42fd008-dd2d-43e4-b3b9-01459025a3f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:44:31.069350 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:31.069313    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgv46\" (UniqueName: \"kubernetes.io/projected/a42fd008-dd2d-43e4-b3b9-01459025a3f8-kube-api-access-qgv46\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:44:31.069542 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:31.069349    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a42fd008-dd2d-43e4-b3b9-01459025a3f8-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:44:31.700510 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:31.700433    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz"
Nov 03 17:44:31.700510 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:31.700446    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz" event={"ID":"a42fd008-dd2d-43e4-b3b9-01459025a3f8","Type":"ContainerDied","Data":"6568085d76fecebaa41c14d7fb0588f53da967a8720755f834890a40969ea197"}
Nov 03 17:44:31.700510 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:31.700497    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6568085d76fecebaa41c14d7fb0588f53da967a8720755f834890a40969ea197"
Nov 03 17:44:34.778832 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:34.778797    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:36.304673 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:36.304637    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:36.537514 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:36.537477    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:36.718969 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:36.718934    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:36.862379 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:44:36.862317    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:44:36.889589 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:36.889556    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5"]
Nov 03 17:44:36.893724 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:36.893687    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:36.941256 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:36.941220    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-49c123d1-fn6s5"]
Nov 03 17:44:37.337694 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:37.337656    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fea7fda0-d82f-4f09-8c32-cec21b718990" path="/var/lib/kubelet/pods/fea7fda0-d82f-4f09-8c32-cec21b718990/volumes"
Nov 03 17:44:38.058735 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:38.058700    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2"]
Nov 03 17:44:38.109892 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:38.109853    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c742ec8-bk7w2"]
Nov 03 17:44:39.338416 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:39.338380    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90049890-ff52-4463-b6cf-e87f76ee3190" path="/var/lib/kubelet/pods/90049890-ff52-4463-b6cf-e87f76ee3190/volumes"
Nov 03 17:44:39.403972 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:39.403937    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f"]
Nov 03 17:44:39.428199 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:39.428160    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:39.505648 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:39.505609    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g"]
Nov 03 17:44:39.781471 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:39.781435    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:39.967907 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:39.967868    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:41.167712 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:41.167679    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:41.568597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:41.568562    6059 scope.go:122] "RemoveContainer" containerID="7910e8bdd85bbebd54b67d2e38601450f2d4fe9a218711bad72cf299180ff0d5"
Nov 03 17:44:41.579449 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:41.579427    6059 scope.go:122] "RemoveContainer" containerID="37fa0140d27200bb3ea0a69b9c4cb1c37aea0a57ec2968c4be8637c016a34167"
Nov 03 17:44:41.589328 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:41.589306    6059 scope.go:122] "RemoveContainer" containerID="c7b15f24dcd8245ab547509af27bf8c5cc2dc01f93cc8aef4d10ec5d88a71586"
Nov 03 17:44:41.599026 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:41.599001    6059 scope.go:122] "RemoveContainer" containerID="c2bc45c1c5d7f9a1f9ebbc293142be7d2b2d6c887090a97e85d2d427a4ef738f"
Nov 03 17:44:41.667990 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:41.667953    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:43.052716 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:43.052678    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3dec6e8-nppxc"]
Nov 03 17:44:43.103844 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:43.103801    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3dec6e8-nppxc"]
Nov 03 17:44:43.337381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:43.337277    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="924ff06e-596a-432d-962a-bb7cb7f64d7c" path="/var/lib/kubelet/pods/924ff06e-596a-432d-962a-bb7cb7f64d7c/volumes"
Nov 03 17:44:44.404272 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.404211    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f" podUID="898b2096-6730-4be9-a3a4-c18f1dd3f936" containerName="payload-processing" containerID="cri-o://a5e40e6353b956d3427fa0f91e5b36921c3c100049b6f110c882101247bf8246" gracePeriod=25
Nov 03 17:44:44.506920 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.506877    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g" podUID="71e9707b-d130-45a1-ba0d-e7536a4c7dbe" containerName="payload-pre-processing" containerID="cri-o://d34d815d38e1ad86e21db1f9b90a49b8d80603747c8efb70188eabbcc039a98c" gracePeriod=25
Nov 03 17:44:44.684565 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.684533    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f"
Nov 03 17:44:44.774557 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.773437    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/898b2096-6730-4be9-a3a4-c18f1dd3f936-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898b2096-6730-4be9-a3a4-c18f1dd3f936-plugins-config-volume\") pod \"898b2096-6730-4be9-a3a4-c18f1dd3f936\" (UID: \"898b2096-6730-4be9-a3a4-c18f1dd3f936\") "
Nov 03 17:44:44.774557 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.773663    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/898b2096-6730-4be9-a3a4-c18f1dd3f936-kube-api-access-l9ffz\" (UniqueName: \"kubernetes.io/projected/898b2096-6730-4be9-a3a4-c18f1dd3f936-kube-api-access-l9ffz\") pod \"898b2096-6730-4be9-a3a4-c18f1dd3f936\" (UID: \"898b2096-6730-4be9-a3a4-c18f1dd3f936\") "
Nov 03 17:44:44.775582 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.775537    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/898b2096-6730-4be9-a3a4-c18f1dd3f936-plugins-config-volume" pod "898b2096-6730-4be9-a3a4-c18f1dd3f936" (UID: "898b2096-6730-4be9-a3a4-c18f1dd3f936"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:44:44.780118 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.780072    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898b2096-6730-4be9-a3a4-c18f1dd3f936-kube-api-access-l9ffz" pod "898b2096-6730-4be9-a3a4-c18f1dd3f936" (UID: "898b2096-6730-4be9-a3a4-c18f1dd3f936"). InnerVolumeSpecName "kube-api-access-l9ffz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:44:44.807484 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.807454    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g"
Nov 03 17:44:44.853240 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.853204    6059 generic.go:356] "Generic (PLEG): container finished" podID="71e9707b-d130-45a1-ba0d-e7536a4c7dbe" containerID="d34d815d38e1ad86e21db1f9b90a49b8d80603747c8efb70188eabbcc039a98c" exitCode=0
Nov 03 17:44:44.853240 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.853238    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g"
Nov 03 17:44:44.853488 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.853282    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g" event={"ID":"71e9707b-d130-45a1-ba0d-e7536a4c7dbe","Type":"ContainerDied","Data":"d34d815d38e1ad86e21db1f9b90a49b8d80603747c8efb70188eabbcc039a98c"}
Nov 03 17:44:44.853488 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.853342    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g" event={"ID":"71e9707b-d130-45a1-ba0d-e7536a4c7dbe","Type":"ContainerDied","Data":"17b9a96ed2eb2e9284c455b4cad0d693ab393c7faeffaac2a18d367544fc16c2"}
Nov 03 17:44:44.853488 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.853390    6059 scope.go:122] "RemoveContainer" containerID="d34d815d38e1ad86e21db1f9b90a49b8d80603747c8efb70188eabbcc039a98c"
Nov 03 17:44:44.855032 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.855003    6059 generic.go:356] "Generic (PLEG): container finished" podID="898b2096-6730-4be9-a3a4-c18f1dd3f936" containerID="a5e40e6353b956d3427fa0f91e5b36921c3c100049b6f110c882101247bf8246" exitCode=0
Nov 03 17:44:44.855169 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.855047    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f"
Nov 03 17:44:44.855169 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.855094    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f" event={"ID":"898b2096-6730-4be9-a3a4-c18f1dd3f936","Type":"ContainerDied","Data":"a5e40e6353b956d3427fa0f91e5b36921c3c100049b6f110c882101247bf8246"}
Nov 03 17:44:44.855169 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.855146    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f" event={"ID":"898b2096-6730-4be9-a3a4-c18f1dd3f936","Type":"ContainerDied","Data":"901ee6bf4ee4929d305604697fd6ea71a71793f269f8d4194a3ac6327bcf5a2f"}
Nov 03 17:44:44.865111 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.865093    6059 scope.go:122] "RemoveContainer" containerID="d34d815d38e1ad86e21db1f9b90a49b8d80603747c8efb70188eabbcc039a98c"
Nov 03 17:44:44.865424 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:44:44.865407    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d34d815d38e1ad86e21db1f9b90a49b8d80603747c8efb70188eabbcc039a98c\": container with ID starting with d34d815d38e1ad86e21db1f9b90a49b8d80603747c8efb70188eabbcc039a98c not found: ID does not exist" containerID="d34d815d38e1ad86e21db1f9b90a49b8d80603747c8efb70188eabbcc039a98c"
Nov 03 17:44:44.865480 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.865431    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d34d815d38e1ad86e21db1f9b90a49b8d80603747c8efb70188eabbcc039a98c"} err="failed to get container status \"d34d815d38e1ad86e21db1f9b90a49b8d80603747c8efb70188eabbcc039a98c\": rpc error: code = NotFound desc = could not find container \"d34d815d38e1ad86e21db1f9b90a49b8d80603747c8efb70188eabbcc039a98c\": container with ID starting with d34d815d38e1ad86e21db1f9b90a49b8d80603747c8efb70188eabbcc039a98c not found: ID does not exist"
Nov 03 17:44:44.865480 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.865449    6059 scope.go:122] "RemoveContainer" containerID="a5e40e6353b956d3427fa0f91e5b36921c3c100049b6f110c882101247bf8246"
Nov 03 17:44:44.875030 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.875001    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71e9707b-d130-45a1-ba0d-e7536a4c7dbe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71e9707b-d130-45a1-ba0d-e7536a4c7dbe-plugins-config-volume\") pod \"71e9707b-d130-45a1-ba0d-e7536a4c7dbe\" (UID: \"71e9707b-d130-45a1-ba0d-e7536a4c7dbe\") "
Nov 03 17:44:44.875223 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.875165    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71e9707b-d130-45a1-ba0d-e7536a4c7dbe-kube-api-access-jhts6\" (UniqueName: \"kubernetes.io/projected/71e9707b-d130-45a1-ba0d-e7536a4c7dbe-kube-api-access-jhts6\") pod \"71e9707b-d130-45a1-ba0d-e7536a4c7dbe\" (UID: \"71e9707b-d130-45a1-ba0d-e7536a4c7dbe\") "
Nov 03 17:44:44.875301 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.875279    6059 scope.go:122] "RemoveContainer" containerID="a5e40e6353b956d3427fa0f91e5b36921c3c100049b6f110c882101247bf8246"
Nov 03 17:44:44.875501 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.875475    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e9707b-d130-45a1-ba0d-e7536a4c7dbe-plugins-config-volume" pod "71e9707b-d130-45a1-ba0d-e7536a4c7dbe" (UID: "71e9707b-d130-45a1-ba0d-e7536a4c7dbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:44:44.875697 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:44:44.875667    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5e40e6353b956d3427fa0f91e5b36921c3c100049b6f110c882101247bf8246\": container with ID starting with a5e40e6353b956d3427fa0f91e5b36921c3c100049b6f110c882101247bf8246 not found: ID does not exist" containerID="a5e40e6353b956d3427fa0f91e5b36921c3c100049b6f110c882101247bf8246"
Nov 03 17:44:44.875794 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.875706    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5e40e6353b956d3427fa0f91e5b36921c3c100049b6f110c882101247bf8246"} err="failed to get container status \"a5e40e6353b956d3427fa0f91e5b36921c3c100049b6f110c882101247bf8246\": rpc error: code = NotFound desc = could not find container \"a5e40e6353b956d3427fa0f91e5b36921c3c100049b6f110c882101247bf8246\": container with ID starting with a5e40e6353b956d3427fa0f91e5b36921c3c100049b6f110c882101247bf8246 not found: ID does not exist"
Nov 03 17:44:44.875794 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.875769    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898b2096-6730-4be9-a3a4-c18f1dd3f936-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:44:44.875794 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.875786    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71e9707b-d130-45a1-ba0d-e7536a4c7dbe-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:44:44.875794 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.875795    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9ffz\" (UniqueName: \"kubernetes.io/projected/898b2096-6730-4be9-a3a4-c18f1dd3f936-kube-api-access-l9ffz\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:44:44.877252 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.877235    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e9707b-d130-45a1-ba0d-e7536a4c7dbe-kube-api-access-jhts6" pod "71e9707b-d130-45a1-ba0d-e7536a4c7dbe" (UID: "71e9707b-d130-45a1-ba0d-e7536a4c7dbe"). InnerVolumeSpecName "kube-api-access-jhts6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:44:44.977034 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:44.976983    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhts6\" (UniqueName: \"kubernetes.io/projected/71e9707b-d130-45a1-ba0d-e7536a4c7dbe-kube-api-access-jhts6\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:44:45.045085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:45.045046    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f"]
Nov 03 17:44:45.048841 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:45.048806    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0b55a36f-5d7d7cbd5f-qfk5f"]
Nov 03 17:44:45.063539 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:45.063508    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:45.238104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:45.238011    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:45.283254 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:45.283217    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g"]
Nov 03 17:44:45.287804 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:45.287771    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0b55a36f-7888d7c4b7-b4k5g"]
Nov 03 17:44:45.337973 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:45.337933    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71e9707b-d130-45a1-ba0d-e7536a4c7dbe" path="/var/lib/kubelet/pods/71e9707b-d130-45a1-ba0d-e7536a4c7dbe/volumes"
Nov 03 17:44:45.338384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:45.338345    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="898b2096-6730-4be9-a3a4-c18f1dd3f936" path="/var/lib/kubelet/pods/898b2096-6730-4be9-a3a4-c18f1dd3f936/volumes"
Nov 03 17:44:46.535526 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:46.535495    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:47.488268 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:47.488174    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:47.681744 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:47.681711    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:47.875688 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:47.875588    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r"]
Nov 03 17:44:47.876621 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:47.876593    6059 cpu_manager.go:414] "containerMap: removing container" podUID="898b2096-6730-4be9-a3a4-c18f1dd3f936" containerName="payload-processing"
Nov 03 17:44:47.876621 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:47.876620    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="898b2096-6730-4be9-a3a4-c18f1dd3f936" containerName="payload-processing"
Nov 03 17:44:47.876621 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:47.876634    6059 cpu_manager.go:414] "containerMap: removing container" podUID="a42fd008-dd2d-43e4-b3b9-01459025a3f8" containerName="revoke-keys"
Nov 03 17:44:47.876874 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:47.876642    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a42fd008-dd2d-43e4-b3b9-01459025a3f8" containerName="revoke-keys"
Nov 03 17:44:47.876874 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:47.876684    6059 cpu_manager.go:414] "containerMap: removing container" podUID="71e9707b-d130-45a1-ba0d-e7536a4c7dbe" containerName="payload-pre-processing"
Nov 03 17:44:47.876874 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:47.876694    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71e9707b-d130-45a1-ba0d-e7536a4c7dbe" containerName="payload-pre-processing"
Nov 03 17:44:47.877044 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:47.877000    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="898b2096-6730-4be9-a3a4-c18f1dd3f936" containerName="payload-processing"
Nov 03 17:44:47.877044 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:47.877019    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a42fd008-dd2d-43e4-b3b9-01459025a3f8" containerName="revoke-keys"
Nov 03 17:44:47.877044 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:47.877036    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="71e9707b-d130-45a1-ba0d-e7536a4c7dbe" containerName="payload-pre-processing"
Nov 03 17:44:47.882754 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:47.882726    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r"
Nov 03 17:44:47.940508 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:47.940471    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r"]
Nov 03 17:44:47.952137 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:47.952103    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e72b7ae-f8ef-4274-8942-3044fb23a9d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r\" (UID: \"8e72b7ae-f8ef-4274-8942-3044fb23a9d5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r"
Nov 03 17:44:47.952463 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:47.952436    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnk7f\" (UniqueName: \"kubernetes.io/projected/8e72b7ae-f8ef-4274-8942-3044fb23a9d5-kube-api-access-dnk7f\") pod \"payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r\" (UID: \"8e72b7ae-f8ef-4274-8942-3044fb23a9d5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r"
Nov 03 17:44:48.013617 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.006440    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-49c123d1\""
Nov 03 17:44:48.027497 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.027448    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-49c123d1-dockercfg-9qmht\""
Nov 03 17:44:48.081380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.068431    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnk7f\" (UniqueName: \"kubernetes.io/projected/8e72b7ae-f8ef-4274-8942-3044fb23a9d5-kube-api-access-dnk7f\") pod \"payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r\" (UID: \"8e72b7ae-f8ef-4274-8942-3044fb23a9d5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r"
Nov 03 17:44:48.081380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.068568    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e72b7ae-f8ef-4274-8942-3044fb23a9d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r\" (UID: \"8e72b7ae-f8ef-4274-8942-3044fb23a9d5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r"
Nov 03 17:44:48.081380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.069567    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e72b7ae-f8ef-4274-8942-3044fb23a9d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r\" (UID: \"8e72b7ae-f8ef-4274-8942-3044fb23a9d5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r"
Nov 03 17:44:48.127737 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.124778    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt"]
Nov 03 17:44:48.132712 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.132684    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt"
Nov 03 17:44:48.137170 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.137118    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt"]
Nov 03 17:44:48.156814 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.145185    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnk7f\" (UniqueName: \"kubernetes.io/projected/8e72b7ae-f8ef-4274-8942-3044fb23a9d5-kube-api-access-dnk7f\") pod \"payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r\" (UID: \"8e72b7ae-f8ef-4274-8942-3044fb23a9d5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r"
Nov 03 17:44:48.196078 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.196043    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r"
Nov 03 17:44:48.275857 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.271496    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe318530-ad39-46f8-ae0f-0b220a3c181b-plugins-config-volume\") pod \"payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt\" (UID: \"fe318530-ad39-46f8-ae0f-0b220a3c181b\") " pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt"
Nov 03 17:44:48.275857 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.271569    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg4tj\" (UniqueName: \"kubernetes.io/projected/fe318530-ad39-46f8-ae0f-0b220a3c181b-kube-api-access-cg4tj\") pod \"payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt\" (UID: \"fe318530-ad39-46f8-ae0f-0b220a3c181b\") " pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt"
Nov 03 17:44:48.373166 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.373119    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe318530-ad39-46f8-ae0f-0b220a3c181b-plugins-config-volume\") pod \"payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt\" (UID: \"fe318530-ad39-46f8-ae0f-0b220a3c181b\") " pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt"
Nov 03 17:44:48.373343 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.373184    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg4tj\" (UniqueName: \"kubernetes.io/projected/fe318530-ad39-46f8-ae0f-0b220a3c181b-kube-api-access-cg4tj\") pod \"payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt\" (UID: \"fe318530-ad39-46f8-ae0f-0b220a3c181b\") " pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt"
Nov 03 17:44:48.374121 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.374093    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe318530-ad39-46f8-ae0f-0b220a3c181b-plugins-config-volume\") pod \"payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt\" (UID: \"fe318530-ad39-46f8-ae0f-0b220a3c181b\") " pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt"
Nov 03 17:44:48.437126 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.437052    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg4tj\" (UniqueName: \"kubernetes.io/projected/fe318530-ad39-46f8-ae0f-0b220a3c181b-kube-api-access-cg4tj\") pod \"payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt\" (UID: \"fe318530-ad39-46f8-ae0f-0b220a3c181b\") " pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt"
Nov 03 17:44:48.474392 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.474334    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt"
Nov 03 17:44:48.581655 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.581621    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r"]
Nov 03 17:44:48.857532 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.857503    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt"]
Nov 03 17:44:48.858507 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:44:48.858478    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe318530_ad39_46f8_ae0f_0b220a3c181b.slice/crio-edbdb7d8f58524e91f9323f1317a84bb4982791623f9ca989d00db2472f397a9 WatchSource:0}: Error finding container edbdb7d8f58524e91f9323f1317a84bb4982791623f9ca989d00db2472f397a9: Status 404 returned error can't find the container with id edbdb7d8f58524e91f9323f1317a84bb4982791623f9ca989d00db2472f397a9
Nov 03 17:44:48.890823 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.890789    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r" event={"ID":"8e72b7ae-f8ef-4274-8942-3044fb23a9d5","Type":"ContainerStarted","Data":"2f0ca3282a84a9769973a69465514ed65450dde379760d50b96d4924e4192d11"}
Nov 03 17:44:48.891028 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.890844    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r" event={"ID":"8e72b7ae-f8ef-4274-8942-3044fb23a9d5","Type":"ContainerStarted","Data":"e1a8b91661e62fd44a8899ef467d4d0239acb2ecf8ba72b7f66bf42a6c668ddc"}
Nov 03 17:44:48.891028 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.890898    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r"
Nov 03 17:44:48.892227 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:48.892202    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt" event={"ID":"fe318530-ad39-46f8-ae0f-0b220a3c181b","Type":"ContainerStarted","Data":"edbdb7d8f58524e91f9323f1317a84bb4982791623f9ca989d00db2472f397a9"}
Nov 03 17:44:49.015489 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:49.015432    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r" podStartSLOduration=2.015414718 podStartE2EDuration="2.015414718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:49.011705325 +0000 UTC m=+1629.809418107" watchObservedRunningTime="2026-11-03 17:44:49.015414718 +0000 UTC m=+1629.813127499"
Nov 03 17:44:49.054316 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:49.054275    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2db4500-q695g"]
Nov 03 17:44:49.057445 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:49.057419    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2db4500-q695g"]
Nov 03 17:44:49.338954 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:49.338920    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7630c9bd-ea2d-45ed-9440-d53e539935d3" path="/var/lib/kubelet/pods/7630c9bd-ea2d-45ed-9440-d53e539935d3/volumes"
Nov 03 17:44:49.901155 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:49.901118    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt" event={"ID":"fe318530-ad39-46f8-ae0f-0b220a3c181b","Type":"ContainerStarted","Data":"093fb40d3cbfc8b5824594aa4687b67a5853c2cb4b1af4f84d540b724f0c51e0"}
Nov 03 17:44:49.901633 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:49.901227    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt"
Nov 03 17:44:50.024386 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:50.024309    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt" podStartSLOduration=3.024291755 podStartE2EDuration="3.024291755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:50.020451879 +0000 UTC m=+1630.818164660" watchObservedRunningTime="2026-11-03 17:44:50.024291755 +0000 UTC m=+1630.822004535"
Nov 03 17:44:51.631705 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:51.631671    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:56.250010 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:56.249953    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:44:58.072316 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:58.072278    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm"]
Nov 03 17:44:58.124104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:58.124068    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-695b8f95-cs2cm"]
Nov 03 17:44:59.336892 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:59.336858    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d65d49cb-7dea-46dd-b8fa-6d785939fb1e" path="/var/lib/kubelet/pods/d65d49cb-7dea-46dd-b8fa-6d785939fb1e/volumes"
Nov 03 17:44:59.905048 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:44:59.905017    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r"
Nov 03 17:45:00.202093 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.201999    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj"]
Nov 03 17:45:00.207132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.207107    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj"
Nov 03 17:45:00.220752 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.220723    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x"]
Nov 03 17:45:00.224933 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.224910    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x"
Nov 03 17:45:00.225849 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.225824    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x"]
Nov 03 17:45:00.230449 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.230428    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x"
Nov 03 17:45:00.241016 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.240987    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x\" (UID: \"e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x"
Nov 03 17:45:00.241170 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.241149    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfgd9\" (UniqueName: \"kubernetes.io/projected/e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7-kube-api-access-wfgd9\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x\" (UID: \"e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x"
Nov 03 17:45:00.241333 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.241307    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f88f65-51ef-4d2c-a2a3-451f883d8e15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x\" (UID: \"26f88f65-51ef-4d2c-a2a3-451f883d8e15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x"
Nov 03 17:45:00.241471 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.241451    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd8vv\" (UniqueName: \"kubernetes.io/projected/7c7a3bea-c23b-49e9-8398-e435046e1187-kube-api-access-cd8vv\") pod \"maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj\" (UID: \"7c7a3bea-c23b-49e9-8398-e435046e1187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj"
Nov 03 17:45:00.241545 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.241520    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7a3bea-c23b-49e9-8398-e435046e1187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj\" (UID: \"7c7a3bea-c23b-49e9-8398-e435046e1187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj"
Nov 03 17:45:00.241763 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.241618    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk4bm\" (UniqueName: \"kubernetes.io/projected/26f88f65-51ef-4d2c-a2a3-451f883d8e15-kube-api-access-gk4bm\") pod \"maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x\" (UID: \"26f88f65-51ef-4d2c-a2a3-451f883d8e15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x"
Nov 03 17:45:00.261634 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.261588    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:00.265547 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.265515    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:45:00.265737 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.265624    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:45:00.265737 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.265675    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"]
Nov 03 17:45:00.265960 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.265933    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj" podUID="1aec66b4-f062-41dc-b9e7-f69143f51117" containerName="istio-proxy" containerID="cri-o://f18165f3089827e03034dd2ff4dec419d7659bfc9fcfa49ad0ce481b3026e895" gracePeriod=30
Nov 03 17:45:00.267281 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.267258    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:45:00.277519 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.277475    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"]
Nov 03 17:45:00.279911 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.279845    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5" podUID="63c795d8-dfff-43d8-88a2-de65cc01a51f" containerName="istio-proxy" containerID="cri-o://15526d5aeba76b625d24ca1f4d29cdc47da676cf93b15405e21565e458e6e775" gracePeriod=30
Nov 03 17:45:00.340639 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.327797    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"]
Nov 03 17:45:00.340639 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.328072    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925" podUID="49075270-6931-42bc-948c-d4cdfc72c5d0" containerName="istio-proxy" containerID="cri-o://14a0f0f9e1947f122d8b78e76ea728129f0e762da1267733798a1d743b836bc0" gracePeriod=30
Nov 03 17:45:00.340639 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.333568    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"]
Nov 03 17:45:00.340639 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.333795    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds" podUID="4f9ab6a0-a561-47fa-8cab-156ef9b8f331" containerName="istio-proxy" containerID="cri-o://0fc98d82881d86a9533ff3d0bfbef9ec3a6f9ac25f88ccda9f8c00d7907c8c3a" gracePeriod=30
Nov 03 17:45:00.341699 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.341554    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj"]
Nov 03 17:45:00.346438 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.342845    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfgd9\" (UniqueName: \"kubernetes.io/projected/e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7-kube-api-access-wfgd9\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x\" (UID: \"e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x"
Nov 03 17:45:00.346438 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.342971    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f88f65-51ef-4d2c-a2a3-451f883d8e15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x\" (UID: \"26f88f65-51ef-4d2c-a2a3-451f883d8e15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x"
Nov 03 17:45:00.346438 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.343041    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd8vv\" (UniqueName: \"kubernetes.io/projected/7c7a3bea-c23b-49e9-8398-e435046e1187-kube-api-access-cd8vv\") pod \"maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj\" (UID: \"7c7a3bea-c23b-49e9-8398-e435046e1187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj"
Nov 03 17:45:00.346438 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.343102    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7a3bea-c23b-49e9-8398-e435046e1187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj\" (UID: \"7c7a3bea-c23b-49e9-8398-e435046e1187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj"
Nov 03 17:45:00.346438 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.343140    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk4bm\" (UniqueName: \"kubernetes.io/projected/26f88f65-51ef-4d2c-a2a3-451f883d8e15-kube-api-access-gk4bm\") pod \"maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x\" (UID: \"26f88f65-51ef-4d2c-a2a3-451f883d8e15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x"
Nov 03 17:45:00.346438 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.343236    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x\" (UID: \"e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x"
Nov 03 17:45:00.346438 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.344891    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x\" (UID: \"e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x"
Nov 03 17:45:00.346438 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.345715    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f88f65-51ef-4d2c-a2a3-451f883d8e15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x\" (UID: \"26f88f65-51ef-4d2c-a2a3-451f883d8e15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x"
Nov 03 17:45:00.346438 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.345734    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7a3bea-c23b-49e9-8398-e435046e1187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj\" (UID: \"7c7a3bea-c23b-49e9-8398-e435046e1187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj"
Nov 03 17:45:00.356311 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.349780    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c"]
Nov 03 17:45:00.358478 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.356902    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x"]
Nov 03 17:45:00.358478 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.357041    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c"
Nov 03 17:45:00.358478 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.358041    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x"]
Nov 03 17:45:00.415580 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.413713    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c"]
Nov 03 17:45:00.431987 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.431062    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfgd9\" (UniqueName: \"kubernetes.io/projected/e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7-kube-api-access-wfgd9\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x\" (UID: \"e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x"
Nov 03 17:45:00.436109 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.436075    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd8vv\" (UniqueName: \"kubernetes.io/projected/7c7a3bea-c23b-49e9-8398-e435046e1187-kube-api-access-cd8vv\") pod \"maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj\" (UID: \"7c7a3bea-c23b-49e9-8398-e435046e1187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj"
Nov 03 17:45:00.444236 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.444199    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b235543-821b-4bd5-9a77-e020368fbcf9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895465-2ds7c\" (UID: \"4b235543-821b-4bd5-9a77-e020368fbcf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c"
Nov 03 17:45:00.444453 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.444251    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmpf9\" (UniqueName: \"kubernetes.io/projected/4b235543-821b-4bd5-9a77-e020368fbcf9-kube-api-access-jmpf9\") pod \"maas-api-key-cleanup-29895465-2ds7c\" (UID: \"4b235543-821b-4bd5-9a77-e020368fbcf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c"
Nov 03 17:45:00.445277 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.445250    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk4bm\" (UniqueName: \"kubernetes.io/projected/26f88f65-51ef-4d2c-a2a3-451f883d8e15-kube-api-access-gk4bm\") pod \"maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x\" (UID: \"26f88f65-51ef-4d2c-a2a3-451f883d8e15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x"
Nov 03 17:45:00.525393 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.524620    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj"
Nov 03 17:45:00.543105 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.543067    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x"
Nov 03 17:45:00.547661 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.547400    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b235543-821b-4bd5-9a77-e020368fbcf9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895465-2ds7c\" (UID: \"4b235543-821b-4bd5-9a77-e020368fbcf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c"
Nov 03 17:45:00.547661 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.547449    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmpf9\" (UniqueName: \"kubernetes.io/projected/4b235543-821b-4bd5-9a77-e020368fbcf9-kube-api-access-jmpf9\") pod \"maas-api-key-cleanup-29895465-2ds7c\" (UID: \"4b235543-821b-4bd5-9a77-e020368fbcf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c"
Nov 03 17:45:00.556646 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.548730    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b235543-821b-4bd5-9a77-e020368fbcf9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895465-2ds7c\" (UID: \"4b235543-821b-4bd5-9a77-e020368fbcf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c"
Nov 03 17:45:00.556646 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.551741    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x"
Nov 03 17:45:00.617497 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.617464    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmpf9\" (UniqueName: \"kubernetes.io/projected/4b235543-821b-4bd5-9a77-e020368fbcf9-kube-api-access-jmpf9\") pod \"maas-api-key-cleanup-29895465-2ds7c\" (UID: \"4b235543-821b-4bd5-9a77-e020368fbcf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c"
Nov 03 17:45:00.674590 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.674284    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c"
Nov 03 17:45:00.912944 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.912917    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt"
Nov 03 17:45:00.975806 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:45:00.975774    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a0bc00_4ddf_4b2f_ac18_c6fcc92be4c7.slice/crio-1b38ff43b9fdad310f52093f620f07ddb92a5e1907ab22532d6fe14621135f0a WatchSource:0}: Error finding container 1b38ff43b9fdad310f52093f620f07ddb92a5e1907ab22532d6fe14621135f0a: Status 404 returned error can't find the container with id 1b38ff43b9fdad310f52093f620f07ddb92a5e1907ab22532d6fe14621135f0a
Nov 03 17:45:00.979181 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.979159    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x"]
Nov 03 17:45:00.990511 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:00.990471    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x" event={"ID":"e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7","Type":"ContainerStarted","Data":"1b38ff43b9fdad310f52093f620f07ddb92a5e1907ab22532d6fe14621135f0a"}
Nov 03 17:45:01.018312 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:45:01.018248    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7a3bea_c23b_49e9_8398_e435046e1187.slice/crio-826390d786a897c176639e8f55a1b84c5bbd7249b564b65064c812a8374d998d WatchSource:0}: Error finding container 826390d786a897c176639e8f55a1b84c5bbd7249b564b65064c812a8374d998d: Status 404 returned error can't find the container with id 826390d786a897c176639e8f55a1b84c5bbd7249b564b65064c812a8374d998d
Nov 03 17:45:01.022612 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:01.022576    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj"]
Nov 03 17:45:01.025774 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:45:01.025744    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f88f65_51ef_4d2c_a2a3_451f883d8e15.slice/crio-510f378a4be6531903bf713e1edabd89ece0797dd9dcbd34e814fced063de73b WatchSource:0}: Error finding container 510f378a4be6531903bf713e1edabd89ece0797dd9dcbd34e814fced063de73b: Status 404 returned error can't find the container with id 510f378a4be6531903bf713e1edabd89ece0797dd9dcbd34e814fced063de73b
Nov 03 17:45:01.027414 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:01.027392    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x"]
Nov 03 17:45:01.099505 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:01.099472    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c"]
Nov 03 17:45:01.132190 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:45:01.132159    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b235543_821b_4bd5_9a77_e020368fbcf9.slice/crio-30bf6626cfea2d2a5cca98859a782b4558512c608e1b23e518a92eb94efe6f8e WatchSource:0}: Error finding container 30bf6626cfea2d2a5cca98859a782b4558512c608e1b23e518a92eb94efe6f8e: Status 404 returned error can't find the container with id 30bf6626cfea2d2a5cca98859a782b4558512c608e1b23e518a92eb94efe6f8e
Nov 03 17:45:01.876585 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:01.876552    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:02.000119 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:02.000076    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x" event={"ID":"26f88f65-51ef-4d2c-a2a3-451f883d8e15","Type":"ContainerStarted","Data":"babad6aa1cd856d4913f200f703e9ae191fc11d2a3b67f63d6cb4c28c3c1d18f"}
Nov 03 17:45:02.000314 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:02.000144    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x" event={"ID":"26f88f65-51ef-4d2c-a2a3-451f883d8e15","Type":"ContainerStarted","Data":"510f378a4be6531903bf713e1edabd89ece0797dd9dcbd34e814fced063de73b"}
Nov 03 17:45:02.002006 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:02.001978    6059 generic.go:356] "Generic (PLEG): container finished" podID="e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7" containerID="721c02da3b47d09e674b12af439d9e33d78d30bfe1eb187cb5b5ccd793e5fc34" exitCode=60
Nov 03 17:45:02.002152 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:02.002062    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x" event={"ID":"e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7","Type":"ContainerDied","Data":"721c02da3b47d09e674b12af439d9e33d78d30bfe1eb187cb5b5ccd793e5fc34"}
Nov 03 17:45:02.002312 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:02.002296    6059 scope.go:122] "RemoveContainer" containerID="721c02da3b47d09e674b12af439d9e33d78d30bfe1eb187cb5b5ccd793e5fc34"
Nov 03 17:45:02.004195 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:02.004097    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c" event={"ID":"4b235543-821b-4bd5-9a77-e020368fbcf9","Type":"ContainerStarted","Data":"9d1a4ae6470f270153ba38b7bab61d359adad473228032bb3ffc73c8e5935df2"}
Nov 03 17:45:02.004195 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:02.004145    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c" event={"ID":"4b235543-821b-4bd5-9a77-e020368fbcf9","Type":"ContainerStarted","Data":"30bf6626cfea2d2a5cca98859a782b4558512c608e1b23e518a92eb94efe6f8e"}
Nov 03 17:45:02.006017 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:02.005935    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj" event={"ID":"7c7a3bea-c23b-49e9-8398-e435046e1187","Type":"ContainerStarted","Data":"92a901dd8cfe0ef9aeaa6089e9774e96938eaef6a797b906930857414a09de7d"}
Nov 03 17:45:02.006017 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:02.005970    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj" event={"ID":"7c7a3bea-c23b-49e9-8398-e435046e1187","Type":"ContainerStarted","Data":"826390d786a897c176639e8f55a1b84c5bbd7249b564b65064c812a8374d998d"}
Nov 03 17:45:02.122976 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:02.122916    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x" podStartSLOduration=2.12289986 podStartE2EDuration="2.12289986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:02.119121592 +0000 UTC m=+1642.916834373" watchObservedRunningTime="2026-11-03 17:45:02.12289986 +0000 UTC m=+1642.920612640"
Nov 03 17:45:02.128430 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:02.128321    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:02.314038 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:02.314004    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:02.316490 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:02.316463    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz"]
Nov 03 17:45:02.322416 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:02.322385    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b55a36f-zrwdz"]
Nov 03 17:45:02.377174 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:02.377112    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c" podStartSLOduration=2.377099787 podStartE2EDuration="2.377099787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:02.376699109 +0000 UTC m=+1643.174411891" watchObservedRunningTime="2026-11-03 17:45:02.377099787 +0000 UTC m=+1643.174812567"
Nov 03 17:45:02.498222 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:02.498164    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj" podStartSLOduration=2.498148399 podStartE2EDuration="2.498148399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:02.496522669 +0000 UTC m=+1643.294235451" watchObservedRunningTime="2026-11-03 17:45:02.498148399 +0000 UTC m=+1643.295861199"
Nov 03 17:45:03.056834 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:03.056713    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x" event={"ID":"e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7","Type":"ContainerStarted","Data":"f99999964bfa240d8f8867787104350fac4be9c2d26fa5ba100af222fb8d86c2"}
Nov 03 17:45:03.065691 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:03.065645    6059 generic.go:356] "Generic (PLEG): container finished" podID="4b235543-821b-4bd5-9a77-e020368fbcf9" containerID="9d1a4ae6470f270153ba38b7bab61d359adad473228032bb3ffc73c8e5935df2" exitCode=60
Nov 03 17:45:03.065864 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:03.065750    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c" event={"ID":"4b235543-821b-4bd5-9a77-e020368fbcf9","Type":"ContainerDied","Data":"9d1a4ae6470f270153ba38b7bab61d359adad473228032bb3ffc73c8e5935df2"}
Nov 03 17:45:03.069417 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:03.066223    6059 scope.go:122] "RemoveContainer" containerID="9d1a4ae6470f270153ba38b7bab61d359adad473228032bb3ffc73c8e5935df2"
Nov 03 17:45:03.072019 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:03.071986    6059 generic.go:356] "Generic (PLEG): container finished" podID="7c7a3bea-c23b-49e9-8398-e435046e1187" containerID="92a901dd8cfe0ef9aeaa6089e9774e96938eaef6a797b906930857414a09de7d" exitCode=60
Nov 03 17:45:03.072209 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:03.072158    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj" event={"ID":"7c7a3bea-c23b-49e9-8398-e435046e1187","Type":"ContainerDied","Data":"92a901dd8cfe0ef9aeaa6089e9774e96938eaef6a797b906930857414a09de7d"}
Nov 03 17:45:03.072839 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:03.072813    6059 scope.go:122] "RemoveContainer" containerID="92a901dd8cfe0ef9aeaa6089e9774e96938eaef6a797b906930857414a09de7d"
Nov 03 17:45:03.075560 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:03.075536    6059 generic.go:356] "Generic (PLEG): container finished" podID="26f88f65-51ef-4d2c-a2a3-451f883d8e15" containerID="babad6aa1cd856d4913f200f703e9ae191fc11d2a3b67f63d6cb4c28c3c1d18f" exitCode=60
Nov 03 17:45:03.075737 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:03.075721    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x" event={"ID":"26f88f65-51ef-4d2c-a2a3-451f883d8e15","Type":"ContainerDied","Data":"babad6aa1cd856d4913f200f703e9ae191fc11d2a3b67f63d6cb4c28c3c1d18f"}
Nov 03 17:45:03.076109 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:03.076093    6059 scope.go:122] "RemoveContainer" containerID="babad6aa1cd856d4913f200f703e9ae191fc11d2a3b67f63d6cb4c28c3c1d18f"
Nov 03 17:45:03.180573 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:03.180517    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x" podStartSLOduration=3.180502071 podStartE2EDuration="3.180502071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:03.177602171 +0000 UTC m=+1643.975314952" watchObservedRunningTime="2026-11-03 17:45:03.180502071 +0000 UTC m=+1643.978214880"
Nov 03 17:45:03.339766 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:03.339694    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a42fd008-dd2d-43e4-b3b9-01459025a3f8" path="/var/lib/kubelet/pods/a42fd008-dd2d-43e4-b3b9-01459025a3f8/volumes"
Nov 03 17:45:03.907422 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:03.907386    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"]
Nov 03 17:45:03.907651 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:03.907594    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq" podUID="bcf48b79-ec61-4698-9c03-74dd7ff368a1" containerName="istio-proxy" containerID="cri-o://078807d2690b5a86b55e34a577414bcfed122a1b77650e82a9f186359a269fc2" gracePeriod=30
Nov 03 17:45:03.982958 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:03.982925    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:04.093455 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:04.092735    6059 generic.go:356] "Generic (PLEG): container finished" podID="e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7" containerID="f99999964bfa240d8f8867787104350fac4be9c2d26fa5ba100af222fb8d86c2" exitCode=60
Nov 03 17:45:04.093455 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:04.092883    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x" event={"ID":"e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7","Type":"ContainerDied","Data":"f99999964bfa240d8f8867787104350fac4be9c2d26fa5ba100af222fb8d86c2"}
Nov 03 17:45:04.093455 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:04.092938    6059 scope.go:122] "RemoveContainer" containerID="721c02da3b47d09e674b12af439d9e33d78d30bfe1eb187cb5b5ccd793e5fc34"
Nov 03 17:45:04.094058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:04.093498    6059 scope.go:122] "RemoveContainer" containerID="f99999964bfa240d8f8867787104350fac4be9c2d26fa5ba100af222fb8d86c2"
Nov 03 17:45:04.094058 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:45:04.093847    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x_odh-ai-gateway-infra(e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x" podUID="e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7"
Nov 03 17:45:04.097881 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:04.097843    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c" event={"ID":"4b235543-821b-4bd5-9a77-e020368fbcf9","Type":"ContainerStarted","Data":"178af4967979186813fa9362412a206e1bff362dcfbb516d60d4cc038dfabe4a"}
Nov 03 17:45:04.101248 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:04.101217    6059 generic.go:356] "Generic (PLEG): container finished" podID="7c7a3bea-c23b-49e9-8398-e435046e1187" containerID="9e3260e7a438964428cd5f3ea0efbc08dfe6e42ed09389f3713acd71c71d994d" exitCode=60
Nov 03 17:45:04.101399 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:04.101374    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj" event={"ID":"7c7a3bea-c23b-49e9-8398-e435046e1187","Type":"ContainerDied","Data":"9e3260e7a438964428cd5f3ea0efbc08dfe6e42ed09389f3713acd71c71d994d"}
Nov 03 17:45:04.101708 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:04.101689    6059 scope.go:122] "RemoveContainer" containerID="9e3260e7a438964428cd5f3ea0efbc08dfe6e42ed09389f3713acd71c71d994d"
Nov 03 17:45:04.102111 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:45:04.102084    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj_odh-ai-gateway-infra(7c7a3bea-c23b-49e9-8398-e435046e1187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj" podUID="7c7a3bea-c23b-49e9-8398-e435046e1187"
Nov 03 17:45:04.103923 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:04.103894    6059 generic.go:356] "Generic (PLEG): container finished" podID="26f88f65-51ef-4d2c-a2a3-451f883d8e15" containerID="d6e850aa21de6407f92955752505a457c5dcd2aefe79aa3eb1fa19e2467769d8" exitCode=60
Nov 03 17:45:04.104156 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:04.104130    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x" event={"ID":"26f88f65-51ef-4d2c-a2a3-451f883d8e15","Type":"ContainerDied","Data":"d6e850aa21de6407f92955752505a457c5dcd2aefe79aa3eb1fa19e2467769d8"}
Nov 03 17:45:04.104786 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:04.104767    6059 scope.go:122] "RemoveContainer" containerID="d6e850aa21de6407f92955752505a457c5dcd2aefe79aa3eb1fa19e2467769d8"
Nov 03 17:45:04.105149 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:45:04.105125    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x_odh-ai-gateway-infra(26f88f65-51ef-4d2c-a2a3-451f883d8e15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x" podUID="26f88f65-51ef-4d2c-a2a3-451f883d8e15"
Nov 03 17:45:04.186678 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:04.186580    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:45:04.186678 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:04.186645    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds" podUID="4f9ab6a0-a561-47fa-8cab-156ef9b8f331" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:45:04.199952 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:04.199919    6059 scope.go:122] "RemoveContainer" containerID="92a901dd8cfe0ef9aeaa6089e9774e96938eaef6a797b906930857414a09de7d"
Nov 03 17:45:04.215117 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:04.215085    6059 scope.go:122] "RemoveContainer" containerID="babad6aa1cd856d4913f200f703e9ae191fc11d2a3b67f63d6cb4c28c3c1d18f"
Nov 03 17:45:04.460670 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:04.460627    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:45:04.460837 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:04.460685    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925" podUID="49075270-6931-42bc-948c-d4cdfc72c5d0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:45:04.491227 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:04.491188    6059 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:45:04.491443 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:04.491255    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj" podUID="1aec66b4-f062-41dc-b9e7-f69143f51117" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:45:04.624541 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:04.624507    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:05.113953 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.113919    6059 scope.go:122] "RemoveContainer" containerID="d6e850aa21de6407f92955752505a457c5dcd2aefe79aa3eb1fa19e2467769d8"
Nov 03 17:45:05.114447 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:45:05.114341    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x_odh-ai-gateway-infra(26f88f65-51ef-4d2c-a2a3-451f883d8e15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x" podUID="26f88f65-51ef-4d2c-a2a3-451f883d8e15"
Nov 03 17:45:05.115970 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.115947    6059 scope.go:122] "RemoveContainer" containerID="f99999964bfa240d8f8867787104350fac4be9c2d26fa5ba100af222fb8d86c2"
Nov 03 17:45:05.116303 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:45:05.116280    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x_odh-ai-gateway-infra(e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x" podUID="e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7"
Nov 03 17:45:05.117719 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.117696    6059 generic.go:356] "Generic (PLEG): container finished" podID="4b235543-821b-4bd5-9a77-e020368fbcf9" containerID="178af4967979186813fa9362412a206e1bff362dcfbb516d60d4cc038dfabe4a" exitCode=60
Nov 03 17:45:05.117837 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.117787    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c" event={"ID":"4b235543-821b-4bd5-9a77-e020368fbcf9","Type":"ContainerDied","Data":"178af4967979186813fa9362412a206e1bff362dcfbb516d60d4cc038dfabe4a"}
Nov 03 17:45:05.117909 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.117854    6059 scope.go:122] "RemoveContainer" containerID="9d1a4ae6470f270153ba38b7bab61d359adad473228032bb3ffc73c8e5935df2"
Nov 03 17:45:05.118034 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.118015    6059 scope.go:122] "RemoveContainer" containerID="178af4967979186813fa9362412a206e1bff362dcfbb516d60d4cc038dfabe4a"
Nov 03 17:45:05.118344 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:45:05.118320    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895465-2ds7c_odh-ai-gateway-infra(4b235543-821b-4bd5-9a77-e020368fbcf9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c" podUID="4b235543-821b-4bd5-9a77-e020368fbcf9"
Nov 03 17:45:05.119821 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.119803    6059 scope.go:122] "RemoveContainer" containerID="9e3260e7a438964428cd5f3ea0efbc08dfe6e42ed09389f3713acd71c71d994d"
Nov 03 17:45:05.120090 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:45:05.120070    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj_odh-ai-gateway-infra(7c7a3bea-c23b-49e9-8398-e435046e1187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj" podUID="7c7a3bea-c23b-49e9-8398-e435046e1187"
Nov 03 17:45:05.180791 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.180752    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"]
Nov 03 17:45:05.285839 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.284199    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.325099 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.314755    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"]
Nov 03 17:45:05.335304 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.335265    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:05.340707 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.340648    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3756d5e7-openshift-default-dockercfg-rngrs\""
Nov 03 17:45:05.381933 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.381847    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-podinfo\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.381933 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.381916    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-envoy\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.382181 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.382110    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq6lq\" (UniqueName: \"kubernetes.io/projected/17bc44f2-4185-4ce3-a127-cf0dc29b9583-kube-api-access-jq6lq\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.382181 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.382159    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-token\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.382306 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.382230    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-workload-socket\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.382377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.382328    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-workload-certs\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.383141 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.382444    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-credential-socket\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.383141 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.382523    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-data\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.383141 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.382564    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istiod-ca-cert\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.485875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.483946    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-envoy\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.485875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.484040    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq6lq\" (UniqueName: \"kubernetes.io/projected/17bc44f2-4185-4ce3-a127-cf0dc29b9583-kube-api-access-jq6lq\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.485875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.484057    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-token\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.485875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.484099    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-workload-socket\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.485875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.484155    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-workload-certs\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.485875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.484216    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-credential-socket\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.485875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.484239    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-data\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.485875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.484264    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istiod-ca-cert\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.485875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.484328    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-podinfo\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.485875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.485162    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-workload-socket\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.485875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.485656    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-credential-socket\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.485875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.485700    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-data\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.486577 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.486002    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-workload-certs\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.487167 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.487120    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istiod-ca-cert\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.487311 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.487290    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-podinfo\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.495478 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.494796    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-envoy\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.550110 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.550009    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-token\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.552091 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.551997    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq6lq\" (UniqueName: \"kubernetes.io/projected/17bc44f2-4185-4ce3-a127-cf0dc29b9583-kube-api-access-jq6lq\") pod \"e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") " pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.587332 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.585828    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:05.619462 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.617264    6059 patch_prober.go:29] interesting pod/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:45:05.619462 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.617334    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq" podUID="bcf48b79-ec61-4698-9c03-74dd7ff368a1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:45:05.638442 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.637455    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:05.680760 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.680654    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:45:05.765300 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.765276    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:45:05.775568 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.775538    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:45:05.780464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.780434    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:45:05.788314 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.788282    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aec66b4-f062-41dc-b9e7-f69143f51117-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aec66b4-f062-41dc-b9e7-f69143f51117-istiod-ca-cert\") pod \"1aec66b4-f062-41dc-b9e7-f69143f51117\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") "
Nov 03 17:45:05.788503 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.788345    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-workload-socket\") pod \"1aec66b4-f062-41dc-b9e7-f69143f51117\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") "
Nov 03 17:45:05.788503 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.788451    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-podinfo\") pod \"1aec66b4-f062-41dc-b9e7-f69143f51117\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") "
Nov 03 17:45:05.788503 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.788488    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-workload-certs\") pod \"1aec66b4-f062-41dc-b9e7-f69143f51117\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") "
Nov 03 17:45:05.788674 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.788601    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-envoy\") pod \"1aec66b4-f062-41dc-b9e7-f69143f51117\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") "
Nov 03 17:45:05.788674 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.788648    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-token\" (UniqueName: \"kubernetes.io/projected/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-token\") pod \"1aec66b4-f062-41dc-b9e7-f69143f51117\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") "
Nov 03 17:45:05.788777 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.788705    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-workload-socket" pod "1aec66b4-f062-41dc-b9e7-f69143f51117" (UID: "1aec66b4-f062-41dc-b9e7-f69143f51117"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:05.788777 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.788728    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-credential-socket\") pod \"1aec66b4-f062-41dc-b9e7-f69143f51117\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") "
Nov 03 17:45:05.788777 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.788733    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aec66b4-f062-41dc-b9e7-f69143f51117-istiod-ca-cert" pod "1aec66b4-f062-41dc-b9e7-f69143f51117" (UID: "1aec66b4-f062-41dc-b9e7-f69143f51117"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:45:05.788931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.788788    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aec66b4-f062-41dc-b9e7-f69143f51117-kube-api-access-mcxnc\" (UniqueName: \"kubernetes.io/projected/1aec66b4-f062-41dc-b9e7-f69143f51117-kube-api-access-mcxnc\") pod \"1aec66b4-f062-41dc-b9e7-f69143f51117\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") "
Nov 03 17:45:05.788931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.788864    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-data\") pod \"1aec66b4-f062-41dc-b9e7-f69143f51117\" (UID: \"1aec66b4-f062-41dc-b9e7-f69143f51117\") "
Nov 03 17:45:05.789041 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.788923    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-workload-certs" pod "1aec66b4-f062-41dc-b9e7-f69143f51117" (UID: "1aec66b4-f062-41dc-b9e7-f69143f51117"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:05.789254 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.789234    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-credential-socket" pod "1aec66b4-f062-41dc-b9e7-f69143f51117" (UID: "1aec66b4-f062-41dc-b9e7-f69143f51117"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:05.789474 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.789447    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.789575 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.789476    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.789575 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.789492    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aec66b4-f062-41dc-b9e7-f69143f51117-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.789575 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.789506    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.789740 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.789726    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-data" pod "1aec66b4-f062-41dc-b9e7-f69143f51117" (UID: "1aec66b4-f062-41dc-b9e7-f69143f51117"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:05.792630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.792604    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aec66b4-f062-41dc-b9e7-f69143f51117-kube-api-access-mcxnc" pod "1aec66b4-f062-41dc-b9e7-f69143f51117" (UID: "1aec66b4-f062-41dc-b9e7-f69143f51117"). InnerVolumeSpecName "kube-api-access-mcxnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:45:05.792630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.792615    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-envoy" pod "1aec66b4-f062-41dc-b9e7-f69143f51117" (UID: "1aec66b4-f062-41dc-b9e7-f69143f51117"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:05.792781 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.792732    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-podinfo" pod "1aec66b4-f062-41dc-b9e7-f69143f51117" (UID: "1aec66b4-f062-41dc-b9e7-f69143f51117"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:45:05.794916 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.794898    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-token" pod "1aec66b4-f062-41dc-b9e7-f69143f51117" (UID: "1aec66b4-f062-41dc-b9e7-f69143f51117"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:45:05.890473 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.890297    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-kube-api-access-qh4bq\" (UniqueName: \"kubernetes.io/projected/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-kube-api-access-qh4bq\") pod \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") "
Nov 03 17:45:05.890473 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.890394    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-podinfo\") pod \"63c795d8-dfff-43d8-88a2-de65cc01a51f\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") "
Nov 03 17:45:05.890473 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.890429    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-data\") pod \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") "
Nov 03 17:45:05.890812 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.890479    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-envoy\") pod \"49075270-6931-42bc-948c-d4cdfc72c5d0\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") "
Nov 03 17:45:05.890812 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.890508    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-token\" (UniqueName: \"kubernetes.io/projected/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-token\") pod \"63c795d8-dfff-43d8-88a2-de65cc01a51f\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") "
Nov 03 17:45:05.890812 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.890559    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-credential-socket\") pod \"63c795d8-dfff-43d8-88a2-de65cc01a51f\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") "
Nov 03 17:45:05.890812 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.890612    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-envoy\") pod \"63c795d8-dfff-43d8-88a2-de65cc01a51f\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") "
Nov 03 17:45:05.890812 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.890657    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-workload-certs\") pod \"49075270-6931-42bc-948c-d4cdfc72c5d0\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") "
Nov 03 17:45:05.890812 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.890734    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-credential-socket\") pod \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") "
Nov 03 17:45:05.890812 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.890765    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c795d8-dfff-43d8-88a2-de65cc01a51f-kube-api-access-btkrm\" (UniqueName: \"kubernetes.io/projected/63c795d8-dfff-43d8-88a2-de65cc01a51f-kube-api-access-btkrm\") pod \"63c795d8-dfff-43d8-88a2-de65cc01a51f\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") "
Nov 03 17:45:05.891188 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.891054    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-credential-socket" pod "63c795d8-dfff-43d8-88a2-de65cc01a51f" (UID: "63c795d8-dfff-43d8-88a2-de65cc01a51f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:05.891188 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.891154    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-data\") pod \"63c795d8-dfff-43d8-88a2-de65cc01a51f\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") "
Nov 03 17:45:05.891292 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.891247    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-token\" (UniqueName: \"kubernetes.io/projected/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-token\") pod \"49075270-6931-42bc-948c-d4cdfc72c5d0\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") "
Nov 03 17:45:05.891292 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.891257    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-workload-certs" pod "49075270-6931-42bc-948c-d4cdfc72c5d0" (UID: "49075270-6931-42bc-948c-d4cdfc72c5d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:05.891292 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.891273    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-credential-socket" pod "4f9ab6a0-a561-47fa-8cab-156ef9b8f331" (UID: "4f9ab6a0-a561-47fa-8cab-156ef9b8f331"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:05.891503 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.891310    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-workload-certs\") pod \"63c795d8-dfff-43d8-88a2-de65cc01a51f\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") "
Nov 03 17:45:05.891503 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.891199    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-data" pod "4f9ab6a0-a561-47fa-8cab-156ef9b8f331" (UID: "4f9ab6a0-a561-47fa-8cab-156ef9b8f331"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:05.891503 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.891379    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-data\") pod \"49075270-6931-42bc-948c-d4cdfc72c5d0\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") "
Nov 03 17:45:05.891503 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.891412    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-credential-socket\") pod \"49075270-6931-42bc-948c-d4cdfc72c5d0\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") "
Nov 03 17:45:05.891503 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.891452    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-podinfo\") pod \"49075270-6931-42bc-948c-d4cdfc72c5d0\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") "
Nov 03 17:45:05.891503 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.891490    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-envoy\") pod \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") "
Nov 03 17:45:05.891851 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.891532    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49075270-6931-42bc-948c-d4cdfc72c5d0-kube-api-access-tgtq7\" (UniqueName: \"kubernetes.io/projected/49075270-6931-42bc-948c-d4cdfc72c5d0-kube-api-access-tgtq7\") pod \"49075270-6931-42bc-948c-d4cdfc72c5d0\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") "
Nov 03 17:45:05.891851 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.891564    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-workload-socket\") pod \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") "
Nov 03 17:45:05.891851 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.891594    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-podinfo\") pod \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") "
Nov 03 17:45:05.891851 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.891638    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-workload-socket\") pod \"49075270-6931-42bc-948c-d4cdfc72c5d0\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") "
Nov 03 17:45:05.891851 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.891677    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istiod-ca-cert\") pod \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") "
Nov 03 17:45:05.891851 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.891709    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-workload-certs\") pod \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") "
Nov 03 17:45:05.891851 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.891791    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-token\" (UniqueName: \"kubernetes.io/projected/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-token\") pod \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\" (UID: \"4f9ab6a0-a561-47fa-8cab-156ef9b8f331\") "
Nov 03 17:45:05.891851 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.891848    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-workload-socket\") pod \"63c795d8-dfff-43d8-88a2-de65cc01a51f\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") "
Nov 03 17:45:05.892264 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.892096    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-data" pod "63c795d8-dfff-43d8-88a2-de65cc01a51f" (UID: "63c795d8-dfff-43d8-88a2-de65cc01a51f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:05.892428 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.892406    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c795d8-dfff-43d8-88a2-de65cc01a51f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c795d8-dfff-43d8-88a2-de65cc01a51f-istiod-ca-cert\") pod \"63c795d8-dfff-43d8-88a2-de65cc01a51f\" (UID: \"63c795d8-dfff-43d8-88a2-de65cc01a51f\") "
Nov 03 17:45:05.892868 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.892824    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-workload-socket" pod "49075270-6931-42bc-948c-d4cdfc72c5d0" (UID: "49075270-6931-42bc-948c-d4cdfc72c5d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:05.893012 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.892905    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-workload-certs" pod "63c795d8-dfff-43d8-88a2-de65cc01a51f" (UID: "63c795d8-dfff-43d8-88a2-de65cc01a51f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:05.893927 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.893097    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-data" pod "49075270-6931-42bc-948c-d4cdfc72c5d0" (UID: "49075270-6931-42bc-948c-d4cdfc72c5d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:05.893927 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.893257    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-workload-certs" pod "4f9ab6a0-a561-47fa-8cab-156ef9b8f331" (UID: "4f9ab6a0-a561-47fa-8cab-156ef9b8f331"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:05.894084 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.893271    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-credential-socket" pod "49075270-6931-42bc-948c-d4cdfc72c5d0" (UID: "49075270-6931-42bc-948c-d4cdfc72c5d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:05.894084 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.893591    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-workload-socket" pod "4f9ab6a0-a561-47fa-8cab-156ef9b8f331" (UID: "4f9ab6a0-a561-47fa-8cab-156ef9b8f331"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:05.894084 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.893621    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c795d8-dfff-43d8-88a2-de65cc01a51f-istiod-ca-cert" pod "63c795d8-dfff-43d8-88a2-de65cc01a51f" (UID: "63c795d8-dfff-43d8-88a2-de65cc01a51f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:45:05.894084 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.893888    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istiod-ca-cert" pod "4f9ab6a0-a561-47fa-8cab-156ef9b8f331" (UID: "4f9ab6a0-a561-47fa-8cab-156ef9b8f331"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:45:05.894084 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.893645    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49075270-6931-42bc-948c-d4cdfc72c5d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49075270-6931-42bc-948c-d4cdfc72c5d0-istiod-ca-cert\") pod \"49075270-6931-42bc-948c-d4cdfc72c5d0\" (UID: \"49075270-6931-42bc-948c-d4cdfc72c5d0\") "
Nov 03 17:45:05.894084 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.894040    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49075270-6931-42bc-948c-d4cdfc72c5d0-istiod-ca-cert" pod "49075270-6931-42bc-948c-d4cdfc72c5d0" (UID: "49075270-6931-42bc-948c-d4cdfc72c5d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:45:05.894433 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.894382    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-workload-socket" pod "63c795d8-dfff-43d8-88a2-de65cc01a51f" (UID: "63c795d8-dfff-43d8-88a2-de65cc01a51f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:05.894900 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.894875    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.894980 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.894909    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.894980 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.894926    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.894980 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.894941    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.894980 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.894955    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.894980 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.894969    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.895307 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.894983    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.895307 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.894998    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c795d8-dfff-43d8-88a2-de65cc01a51f-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.895307 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.895013    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.895307 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.895028    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49075270-6931-42bc-948c-d4cdfc72c5d0-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.895307 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.895044    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.895307 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.895059    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.895307 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.895074    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.895307 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.895089    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.895307 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.895105    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.895307 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.895121    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.895307 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.895137    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.895307 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.895152    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcxnc\" (UniqueName: \"kubernetes.io/projected/1aec66b4-f062-41dc-b9e7-f69143f51117-kube-api-access-mcxnc\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.895307 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.895168    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aec66b4-f062-41dc-b9e7-f69143f51117-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.895307 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.895183    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.896050 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.895602    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-podinfo" pod "49075270-6931-42bc-948c-d4cdfc72c5d0" (UID: "49075270-6931-42bc-948c-d4cdfc72c5d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:45:05.896166 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.896121    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-token" pod "4f9ab6a0-a561-47fa-8cab-156ef9b8f331" (UID: "4f9ab6a0-a561-47fa-8cab-156ef9b8f331"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:45:05.896963 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.896941    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49075270-6931-42bc-948c-d4cdfc72c5d0-kube-api-access-tgtq7" pod "49075270-6931-42bc-948c-d4cdfc72c5d0" (UID: "49075270-6931-42bc-948c-d4cdfc72c5d0"). InnerVolumeSpecName "kube-api-access-tgtq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:45:05.897190 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.897166    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-podinfo" pod "4f9ab6a0-a561-47fa-8cab-156ef9b8f331" (UID: "4f9ab6a0-a561-47fa-8cab-156ef9b8f331"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:45:05.897311 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.897178    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-envoy" pod "4f9ab6a0-a561-47fa-8cab-156ef9b8f331" (UID: "4f9ab6a0-a561-47fa-8cab-156ef9b8f331"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:05.897464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.897444    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-envoy" pod "63c795d8-dfff-43d8-88a2-de65cc01a51f" (UID: "63c795d8-dfff-43d8-88a2-de65cc01a51f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:05.897540 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.897474    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-kube-api-access-qh4bq" pod "4f9ab6a0-a561-47fa-8cab-156ef9b8f331" (UID: "4f9ab6a0-a561-47fa-8cab-156ef9b8f331"). InnerVolumeSpecName "kube-api-access-qh4bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:45:05.897540 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.897498    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-token" pod "63c795d8-dfff-43d8-88a2-de65cc01a51f" (UID: "63c795d8-dfff-43d8-88a2-de65cc01a51f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:45:05.897690 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.897615    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-podinfo" pod "63c795d8-dfff-43d8-88a2-de65cc01a51f" (UID: "63c795d8-dfff-43d8-88a2-de65cc01a51f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:45:05.897793 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.897704    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-envoy" pod "49075270-6931-42bc-948c-d4cdfc72c5d0" (UID: "49075270-6931-42bc-948c-d4cdfc72c5d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:05.898536 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.898515    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c795d8-dfff-43d8-88a2-de65cc01a51f-kube-api-access-btkrm" pod "63c795d8-dfff-43d8-88a2-de65cc01a51f" (UID: "63c795d8-dfff-43d8-88a2-de65cc01a51f"). InnerVolumeSpecName "kube-api-access-btkrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:45:05.898917 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.898900    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-token" pod "49075270-6931-42bc-948c-d4cdfc72c5d0" (UID: "49075270-6931-42bc-948c-d4cdfc72c5d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:45:05.996966 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.996928    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.996966 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.996966    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btkrm\" (UniqueName: \"kubernetes.io/projected/63c795d8-dfff-43d8-88a2-de65cc01a51f-kube-api-access-btkrm\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.997257 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.996982    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.997257 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.996996    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.997257 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.997009    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.997257 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.997023    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgtq7\" (UniqueName: \"kubernetes.io/projected/49075270-6931-42bc-948c-d4cdfc72c5d0-kube-api-access-tgtq7\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.997257 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.997036    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.997257 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.997049    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.997257 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.997063    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh4bq\" (UniqueName: \"kubernetes.io/projected/4f9ab6a0-a561-47fa-8cab-156ef9b8f331-kube-api-access-qh4bq\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.997257 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.997078    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.997257 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.997093    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49075270-6931-42bc-948c-d4cdfc72c5d0-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:05.997257 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:05.997107    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c795d8-dfff-43d8-88a2-de65cc01a51f-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:06.101614 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:45:06.101583    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17bc44f2_4185_4ce3_a127_cf0dc29b9583.slice/crio-8cfb5c457a05a5b22baa703b8dfd835fb7df23460c4600a3e0a088d6c8d08861 WatchSource:0}: Error finding container 8cfb5c457a05a5b22baa703b8dfd835fb7df23460c4600a3e0a088d6c8d08861: Status 404 returned error can't find the container with id 8cfb5c457a05a5b22baa703b8dfd835fb7df23460c4600a3e0a088d6c8d08861
Nov 03 17:45:06.104210 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.104179    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:45:06.104300 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.104251    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:45:06.104300 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.104281    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:45:06.104455 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.104433    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"]
Nov 03 17:45:06.136758 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.136438    6059 generic.go:356] "Generic (PLEG): container finished" podID="63c795d8-dfff-43d8-88a2-de65cc01a51f" containerID="15526d5aeba76b625d24ca1f4d29cdc47da676cf93b15405e21565e458e6e775" exitCode=0
Nov 03 17:45:06.136758 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.136503    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"
Nov 03 17:45:06.136758 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.136573    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5" event={"ID":"63c795d8-dfff-43d8-88a2-de65cc01a51f","Type":"ContainerDied","Data":"15526d5aeba76b625d24ca1f4d29cdc47da676cf93b15405e21565e458e6e775"}
Nov 03 17:45:06.136758 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.136624    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5" event={"ID":"63c795d8-dfff-43d8-88a2-de65cc01a51f","Type":"ContainerDied","Data":"8ee8bb931e915fda871b21389bca5ae6dd22f41adb0bc406bf6ed734335532a9"}
Nov 03 17:45:06.136758 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.136655    6059 scope.go:122] "RemoveContainer" containerID="15526d5aeba76b625d24ca1f4d29cdc47da676cf93b15405e21565e458e6e775"
Nov 03 17:45:06.142730 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.142701    6059 scope.go:122] "RemoveContainer" containerID="178af4967979186813fa9362412a206e1bff362dcfbb516d60d4cc038dfabe4a"
Nov 03 17:45:06.143125 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:45:06.143066    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895465-2ds7c_odh-ai-gateway-infra(4b235543-821b-4bd5-9a77-e020368fbcf9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c" podUID="4b235543-821b-4bd5-9a77-e020368fbcf9"
Nov 03 17:45:06.144291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.144263    6059 generic.go:356] "Generic (PLEG): container finished" podID="1aec66b4-f062-41dc-b9e7-f69143f51117" containerID="f18165f3089827e03034dd2ff4dec419d7659bfc9fcfa49ad0ce481b3026e895" exitCode=0
Nov 03 17:45:06.144584 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.144564    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"
Nov 03 17:45:06.144869 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.144776    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj" event={"ID":"1aec66b4-f062-41dc-b9e7-f69143f51117","Type":"ContainerDied","Data":"f18165f3089827e03034dd2ff4dec419d7659bfc9fcfa49ad0ce481b3026e895"}
Nov 03 17:45:06.144869 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.144839    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj" event={"ID":"1aec66b4-f062-41dc-b9e7-f69143f51117","Type":"ContainerDied","Data":"6457431bad9dd8a77ccad54343c8adf85b297e94d0cfec81d4791bebe8f2b6d9"}
Nov 03 17:45:06.146801 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.146769    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x" event={"ID":"17bc44f2-4185-4ce3-a127-cf0dc29b9583","Type":"ContainerStarted","Data":"8cfb5c457a05a5b22baa703b8dfd835fb7df23460c4600a3e0a088d6c8d08861"}
Nov 03 17:45:06.148233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.148208    6059 generic.go:356] "Generic (PLEG): container finished" podID="4f9ab6a0-a561-47fa-8cab-156ef9b8f331" containerID="0fc98d82881d86a9533ff3d0bfbef9ec3a6f9ac25f88ccda9f8c00d7907c8c3a" exitCode=0
Nov 03 17:45:06.148340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.148304    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds" event={"ID":"4f9ab6a0-a561-47fa-8cab-156ef9b8f331","Type":"ContainerDied","Data":"0fc98d82881d86a9533ff3d0bfbef9ec3a6f9ac25f88ccda9f8c00d7907c8c3a"}
Nov 03 17:45:06.148435 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.148343    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds" event={"ID":"4f9ab6a0-a561-47fa-8cab-156ef9b8f331","Type":"ContainerDied","Data":"dd6f501d577f25d8cb0e299a3b34e0d355e71cb7ba0c3f3e6c45d9f6a8fbdf56"}
Nov 03 17:45:06.148435 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.148428    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"
Nov 03 17:45:06.152261 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.150814    6059 generic.go:356] "Generic (PLEG): container finished" podID="49075270-6931-42bc-948c-d4cdfc72c5d0" containerID="14a0f0f9e1947f122d8b78e76ea728129f0e762da1267733798a1d743b836bc0" exitCode=0
Nov 03 17:45:06.152261 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.150856    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925" event={"ID":"49075270-6931-42bc-948c-d4cdfc72c5d0","Type":"ContainerDied","Data":"14a0f0f9e1947f122d8b78e76ea728129f0e762da1267733798a1d743b836bc0"}
Nov 03 17:45:06.152261 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.150893    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925" event={"ID":"49075270-6931-42bc-948c-d4cdfc72c5d0","Type":"ContainerDied","Data":"e7254068f0b87a45c6e91805c5eb4fd71d19dbd60e3f177edace9b62c634e477"}
Nov 03 17:45:06.152261 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.150954    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"
Nov 03 17:45:06.164824 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.164800    6059 scope.go:122] "RemoveContainer" containerID="30e23af5381fd9a80066543d1207c917691f7f5d1d864d76eabe6a0fc5d737db"
Nov 03 17:45:06.193873 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.193845    6059 scope.go:122] "RemoveContainer" containerID="15526d5aeba76b625d24ca1f4d29cdc47da676cf93b15405e21565e458e6e775"
Nov 03 17:45:06.194240 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:45:06.194216    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15526d5aeba76b625d24ca1f4d29cdc47da676cf93b15405e21565e458e6e775\": container with ID starting with 15526d5aeba76b625d24ca1f4d29cdc47da676cf93b15405e21565e458e6e775 not found: ID does not exist" containerID="15526d5aeba76b625d24ca1f4d29cdc47da676cf93b15405e21565e458e6e775"
Nov 03 17:45:06.194299 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.194248    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15526d5aeba76b625d24ca1f4d29cdc47da676cf93b15405e21565e458e6e775"} err="failed to get container status \"15526d5aeba76b625d24ca1f4d29cdc47da676cf93b15405e21565e458e6e775\": rpc error: code = NotFound desc = could not find container \"15526d5aeba76b625d24ca1f4d29cdc47da676cf93b15405e21565e458e6e775\": container with ID starting with 15526d5aeba76b625d24ca1f4d29cdc47da676cf93b15405e21565e458e6e775 not found: ID does not exist"
Nov 03 17:45:06.194299 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.194267    6059 scope.go:122] "RemoveContainer" containerID="30e23af5381fd9a80066543d1207c917691f7f5d1d864d76eabe6a0fc5d737db"
Nov 03 17:45:06.194581 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:45:06.194558    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e23af5381fd9a80066543d1207c917691f7f5d1d864d76eabe6a0fc5d737db\": container with ID starting with 30e23af5381fd9a80066543d1207c917691f7f5d1d864d76eabe6a0fc5d737db not found: ID does not exist" containerID="30e23af5381fd9a80066543d1207c917691f7f5d1d864d76eabe6a0fc5d737db"
Nov 03 17:45:06.194633 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.194591    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e23af5381fd9a80066543d1207c917691f7f5d1d864d76eabe6a0fc5d737db"} err="failed to get container status \"30e23af5381fd9a80066543d1207c917691f7f5d1d864d76eabe6a0fc5d737db\": rpc error: code = NotFound desc = could not find container \"30e23af5381fd9a80066543d1207c917691f7f5d1d864d76eabe6a0fc5d737db\": container with ID starting with 30e23af5381fd9a80066543d1207c917691f7f5d1d864d76eabe6a0fc5d737db not found: ID does not exist"
Nov 03 17:45:06.194633 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.194611    6059 scope.go:122] "RemoveContainer" containerID="f18165f3089827e03034dd2ff4dec419d7659bfc9fcfa49ad0ce481b3026e895"
Nov 03 17:45:06.205535 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.205507    6059 scope.go:122] "RemoveContainer" containerID="f18165f3089827e03034dd2ff4dec419d7659bfc9fcfa49ad0ce481b3026e895"
Nov 03 17:45:06.205917 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:45:06.205892    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f18165f3089827e03034dd2ff4dec419d7659bfc9fcfa49ad0ce481b3026e895\": container with ID starting with f18165f3089827e03034dd2ff4dec419d7659bfc9fcfa49ad0ce481b3026e895 not found: ID does not exist" containerID="f18165f3089827e03034dd2ff4dec419d7659bfc9fcfa49ad0ce481b3026e895"
Nov 03 17:45:06.205967 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.205931    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18165f3089827e03034dd2ff4dec419d7659bfc9fcfa49ad0ce481b3026e895"} err="failed to get container status \"f18165f3089827e03034dd2ff4dec419d7659bfc9fcfa49ad0ce481b3026e895\": rpc error: code = NotFound desc = could not find container \"f18165f3089827e03034dd2ff4dec419d7659bfc9fcfa49ad0ce481b3026e895\": container with ID starting with f18165f3089827e03034dd2ff4dec419d7659bfc9fcfa49ad0ce481b3026e895 not found: ID does not exist"
Nov 03 17:45:06.205967 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.205959    6059 scope.go:122] "RemoveContainer" containerID="0fc98d82881d86a9533ff3d0bfbef9ec3a6f9ac25f88ccda9f8c00d7907c8c3a"
Nov 03 17:45:06.216588 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.216565    6059 scope.go:122] "RemoveContainer" containerID="0fc98d82881d86a9533ff3d0bfbef9ec3a6f9ac25f88ccda9f8c00d7907c8c3a"
Nov 03 17:45:06.216938 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:45:06.216916    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc98d82881d86a9533ff3d0bfbef9ec3a6f9ac25f88ccda9f8c00d7907c8c3a\": container with ID starting with 0fc98d82881d86a9533ff3d0bfbef9ec3a6f9ac25f88ccda9f8c00d7907c8c3a not found: ID does not exist" containerID="0fc98d82881d86a9533ff3d0bfbef9ec3a6f9ac25f88ccda9f8c00d7907c8c3a"
Nov 03 17:45:06.217035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.216945    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc98d82881d86a9533ff3d0bfbef9ec3a6f9ac25f88ccda9f8c00d7907c8c3a"} err="failed to get container status \"0fc98d82881d86a9533ff3d0bfbef9ec3a6f9ac25f88ccda9f8c00d7907c8c3a\": rpc error: code = NotFound desc = could not find container \"0fc98d82881d86a9533ff3d0bfbef9ec3a6f9ac25f88ccda9f8c00d7907c8c3a\": container with ID starting with 0fc98d82881d86a9533ff3d0bfbef9ec3a6f9ac25f88ccda9f8c00d7907c8c3a not found: ID does not exist"
Nov 03 17:45:06.217035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.216972    6059 scope.go:122] "RemoveContainer" containerID="14a0f0f9e1947f122d8b78e76ea728129f0e762da1267733798a1d743b836bc0"
Nov 03 17:45:06.231619 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.229007    6059 scope.go:122] "RemoveContainer" containerID="14a0f0f9e1947f122d8b78e76ea728129f0e762da1267733798a1d743b836bc0"
Nov 03 17:45:06.231619 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:45:06.229586    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a0f0f9e1947f122d8b78e76ea728129f0e762da1267733798a1d743b836bc0\": container with ID starting with 14a0f0f9e1947f122d8b78e76ea728129f0e762da1267733798a1d743b836bc0 not found: ID does not exist" containerID="14a0f0f9e1947f122d8b78e76ea728129f0e762da1267733798a1d743b836bc0"
Nov 03 17:45:06.231619 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.229649    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a0f0f9e1947f122d8b78e76ea728129f0e762da1267733798a1d743b836bc0"} err="failed to get container status \"14a0f0f9e1947f122d8b78e76ea728129f0e762da1267733798a1d743b836bc0\": rpc error: code = NotFound desc = could not find container \"14a0f0f9e1947f122d8b78e76ea728129f0e762da1267733798a1d743b836bc0\": container with ID starting with 14a0f0f9e1947f122d8b78e76ea728129f0e762da1267733798a1d743b836bc0 not found: ID does not exist"
Nov 03 17:45:06.381635 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.381442    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"]
Nov 03 17:45:06.385994 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.385966    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-c9925"]
Nov 03 17:45:06.504223 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.504185    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"]
Nov 03 17:45:06.510880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.510850    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-rntdj"]
Nov 03 17:45:06.747264 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.747229    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"]
Nov 03 17:45:06.754113 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.754084    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-8vpc5"]
Nov 03 17:45:06.862678 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:45:06.862595    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:45:06.869696 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.869661    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"]
Nov 03 17:45:06.874130 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:06.874103    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-wdlds"]
Nov 03 17:45:07.163396 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:07.163290    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x" event={"ID":"17bc44f2-4185-4ce3-a127-cf0dc29b9583","Type":"ContainerStarted","Data":"81eff07997580368b18f8080d233d0f6c3a34929a413476ebf4e935c4ec4d076"}
Nov 03 17:45:07.184185 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:07.184130    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:07.288498 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:07.288438    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x" podStartSLOduration=2.2884218130000002 podStartE2EDuration="2.288421813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:07.287916475 +0000 UTC m=+1648.085629269" watchObservedRunningTime="2026-11-03 17:45:07.288421813 +0000 UTC m=+1648.086134593"
Nov 03 17:45:07.337390 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:07.337332    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aec66b4-f062-41dc-b9e7-f69143f51117" path="/var/lib/kubelet/pods/1aec66b4-f062-41dc-b9e7-f69143f51117/volumes"
Nov 03 17:45:07.338102 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:07.338083    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49075270-6931-42bc-948c-d4cdfc72c5d0" path="/var/lib/kubelet/pods/49075270-6931-42bc-948c-d4cdfc72c5d0/volumes"
Nov 03 17:45:07.338682 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:07.338664    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f9ab6a0-a561-47fa-8cab-156ef9b8f331" path="/var/lib/kubelet/pods/4f9ab6a0-a561-47fa-8cab-156ef9b8f331/volumes"
Nov 03 17:45:07.339750 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:07.339731    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c795d8-dfff-43d8-88a2-de65cc01a51f" path="/var/lib/kubelet/pods/63c795d8-dfff-43d8-88a2-de65cc01a51f/volumes"
Nov 03 17:45:07.637972 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:07.637933    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:07.643453 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:07.643428    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:08.177208 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:08.177177    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:08.178328 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:08.178310    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:45:09.051779 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.051744    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7"]
Nov 03 17:45:09.103176 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.103143    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95f94805-pv2h7"]
Nov 03 17:45:09.186332 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.186264    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:45:09.188637 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.188607    6059 generic.go:356] "Generic (PLEG): container finished" podID="bcf48b79-ec61-4698-9c03-74dd7ff368a1" containerID="078807d2690b5a86b55e34a577414bcfed122a1b77650e82a9f186359a269fc2" exitCode=0
Nov 03 17:45:09.188767 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.188653    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"
Nov 03 17:45:09.188767 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.188678    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq" event={"ID":"bcf48b79-ec61-4698-9c03-74dd7ff368a1","Type":"ContainerDied","Data":"078807d2690b5a86b55e34a577414bcfed122a1b77650e82a9f186359a269fc2"}
Nov 03 17:45:09.188767 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.188729    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq" event={"ID":"bcf48b79-ec61-4698-9c03-74dd7ff368a1","Type":"ContainerDied","Data":"a11143ffff6a319795296f8736e794c5e2b99d78b0cb2c4bfb3dec2cf5630512"}
Nov 03 17:45:09.188767 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.188752    6059 scope.go:122] "RemoveContainer" containerID="078807d2690b5a86b55e34a577414bcfed122a1b77650e82a9f186359a269fc2"
Nov 03 17:45:09.199476 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.199455    6059 scope.go:122] "RemoveContainer" containerID="1a3f721a9863b2f8422a71b8322d68ae1244ea1ac51c4c2d5455b235796c1fff"
Nov 03 17:45:09.209951 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.209926    6059 scope.go:122] "RemoveContainer" containerID="078807d2690b5a86b55e34a577414bcfed122a1b77650e82a9f186359a269fc2"
Nov 03 17:45:09.210275 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:45:09.210253    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"078807d2690b5a86b55e34a577414bcfed122a1b77650e82a9f186359a269fc2\": container with ID starting with 078807d2690b5a86b55e34a577414bcfed122a1b77650e82a9f186359a269fc2 not found: ID does not exist" containerID="078807d2690b5a86b55e34a577414bcfed122a1b77650e82a9f186359a269fc2"
Nov 03 17:45:09.210447 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.210283    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"078807d2690b5a86b55e34a577414bcfed122a1b77650e82a9f186359a269fc2"} err="failed to get container status \"078807d2690b5a86b55e34a577414bcfed122a1b77650e82a9f186359a269fc2\": rpc error: code = NotFound desc = could not find container \"078807d2690b5a86b55e34a577414bcfed122a1b77650e82a9f186359a269fc2\": container with ID starting with 078807d2690b5a86b55e34a577414bcfed122a1b77650e82a9f186359a269fc2 not found: ID does not exist"
Nov 03 17:45:09.210447 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.210305    6059 scope.go:122] "RemoveContainer" containerID="1a3f721a9863b2f8422a71b8322d68ae1244ea1ac51c4c2d5455b235796c1fff"
Nov 03 17:45:09.210623 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:45:09.210605    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a3f721a9863b2f8422a71b8322d68ae1244ea1ac51c4c2d5455b235796c1fff\": container with ID starting with 1a3f721a9863b2f8422a71b8322d68ae1244ea1ac51c4c2d5455b235796c1fff not found: ID does not exist" containerID="1a3f721a9863b2f8422a71b8322d68ae1244ea1ac51c4c2d5455b235796c1fff"
Nov 03 17:45:09.210660 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.210629    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3f721a9863b2f8422a71b8322d68ae1244ea1ac51c4c2d5455b235796c1fff"} err="failed to get container status \"1a3f721a9863b2f8422a71b8322d68ae1244ea1ac51c4c2d5455b235796c1fff\": rpc error: code = NotFound desc = could not find container \"1a3f721a9863b2f8422a71b8322d68ae1244ea1ac51c4c2d5455b235796c1fff\": container with ID starting with 1a3f721a9863b2f8422a71b8322d68ae1244ea1ac51c4c2d5455b235796c1fff not found: ID does not exist"
Nov 03 17:45:09.339343 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.339300    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9031a661-8d7b-4a6a-ae19-0a9b1a594b99" path="/var/lib/kubelet/pods/9031a661-8d7b-4a6a-ae19-0a9b1a594b99/volumes"
Nov 03 17:45:09.341380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.341342    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-token\" (UniqueName: \"kubernetes.io/projected/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-token\") pod \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") "
Nov 03 17:45:09.341505 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.341434    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf48b79-ec61-4698-9c03-74dd7ff368a1-kube-api-access-qhcwp\" (UniqueName: \"kubernetes.io/projected/bcf48b79-ec61-4698-9c03-74dd7ff368a1-kube-api-access-qhcwp\") pod \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") "
Nov 03 17:45:09.341505 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.341477    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-data\") pod \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") "
Nov 03 17:45:09.341682 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.341553    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-credential-socket\") pod \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") "
Nov 03 17:45:09.341682 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.341593    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-workload-certs\") pod \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") "
Nov 03 17:45:09.341796 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.341733    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-envoy\") pod \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") "
Nov 03 17:45:09.341796 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.341777    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istiod-ca-cert\") pod \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") "
Nov 03 17:45:09.341902 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.341816    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-workload-socket\") pod \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") "
Nov 03 17:45:09.341902 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.341819    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-data" pod "bcf48b79-ec61-4698-9c03-74dd7ff368a1" (UID: "bcf48b79-ec61-4698-9c03-74dd7ff368a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:09.341902 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.341846    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-credential-socket" pod "bcf48b79-ec61-4698-9c03-74dd7ff368a1" (UID: "bcf48b79-ec61-4698-9c03-74dd7ff368a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:09.341902 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.341886    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-podinfo\") pod \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\" (UID: \"bcf48b79-ec61-4698-9c03-74dd7ff368a1\") "
Nov 03 17:45:09.342112 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.342090    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-workload-socket" pod "bcf48b79-ec61-4698-9c03-74dd7ff368a1" (UID: "bcf48b79-ec61-4698-9c03-74dd7ff368a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:09.342250 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.342220    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istiod-ca-cert" pod "bcf48b79-ec61-4698-9c03-74dd7ff368a1" (UID: "bcf48b79-ec61-4698-9c03-74dd7ff368a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:45:09.342250 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.342229    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-workload-certs" pod "bcf48b79-ec61-4698-9c03-74dd7ff368a1" (UID: "bcf48b79-ec61-4698-9c03-74dd7ff368a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:09.343009 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.342984    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:09.343118 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.343013    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:09.343118 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.343028    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:09.343118 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.343045    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:09.343118 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.343059    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:09.344037 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.344009    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-token" pod "bcf48b79-ec61-4698-9c03-74dd7ff368a1" (UID: "bcf48b79-ec61-4698-9c03-74dd7ff368a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:45:09.344333 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.344306    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf48b79-ec61-4698-9c03-74dd7ff368a1-kube-api-access-qhcwp" pod "bcf48b79-ec61-4698-9c03-74dd7ff368a1" (UID: "bcf48b79-ec61-4698-9c03-74dd7ff368a1"). InnerVolumeSpecName "kube-api-access-qhcwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:45:09.344333 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.344313    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-podinfo" pod "bcf48b79-ec61-4698-9c03-74dd7ff368a1" (UID: "bcf48b79-ec61-4698-9c03-74dd7ff368a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:45:09.345724 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.345694    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-envoy" pod "bcf48b79-ec61-4698-9c03-74dd7ff368a1" (UID: "bcf48b79-ec61-4698-9c03-74dd7ff368a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:45:09.354927 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.354893    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:09.445066 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.444948    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:09.445066 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.444982    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhcwp\" (UniqueName: \"kubernetes.io/projected/bcf48b79-ec61-4698-9c03-74dd7ff368a1-kube-api-access-qhcwp\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:09.445066 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.444999    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:09.445066 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.445014    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf48b79-ec61-4698-9c03-74dd7ff368a1-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:09.542057 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.541976    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:09.620421 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.620385    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"]
Nov 03 17:45:09.623285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:09.623260    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0b55a36f-openshift-default-86dd48c87c-2bqcq"]
Nov 03 17:45:11.337877 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:11.337837    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcf48b79-ec61-4698-9c03-74dd7ff368a1" path="/var/lib/kubelet/pods/bcf48b79-ec61-4698-9c03-74dd7ff368a1/volumes"
Nov 03 17:45:11.876534 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:11.876495    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:12.075769 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.075737    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:12.435649 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.435608    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:12.673259 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.673220    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:12.843391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.843340    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894"]
Nov 03 17:45:12.844288 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.844268    6059 cpu_manager.go:414] "containerMap: removing container" podUID="63c795d8-dfff-43d8-88a2-de65cc01a51f" containerName="istio-proxy"
Nov 03 17:45:12.844340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.844291    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c795d8-dfff-43d8-88a2-de65cc01a51f" containerName="istio-proxy"
Nov 03 17:45:12.844340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.844307    6059 cpu_manager.go:414] "containerMap: removing container" podUID="49075270-6931-42bc-948c-d4cdfc72c5d0" containerName="istio-proxy"
Nov 03 17:45:12.844340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.844315    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49075270-6931-42bc-948c-d4cdfc72c5d0" containerName="istio-proxy"
Nov 03 17:45:12.844454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.844344    6059 cpu_manager.go:414] "containerMap: removing container" podUID="bcf48b79-ec61-4698-9c03-74dd7ff368a1" containerName="istio-proxy"
Nov 03 17:45:12.844454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.844368    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf48b79-ec61-4698-9c03-74dd7ff368a1" containerName="istio-proxy"
Nov 03 17:45:12.844454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.844381    6059 cpu_manager.go:414] "containerMap: removing container" podUID="63c795d8-dfff-43d8-88a2-de65cc01a51f" containerName="istio-proxy"
Nov 03 17:45:12.844454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.844388    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c795d8-dfff-43d8-88a2-de65cc01a51f" containerName="istio-proxy"
Nov 03 17:45:12.844454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.844448    6059 cpu_manager.go:414] "containerMap: removing container" podUID="1aec66b4-f062-41dc-b9e7-f69143f51117" containerName="istio-proxy"
Nov 03 17:45:12.844597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.844456    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aec66b4-f062-41dc-b9e7-f69143f51117" containerName="istio-proxy"
Nov 03 17:45:12.844597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.844477    6059 cpu_manager.go:414] "containerMap: removing container" podUID="4f9ab6a0-a561-47fa-8cab-156ef9b8f331" containerName="istio-proxy"
Nov 03 17:45:12.844597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.844484    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f9ab6a0-a561-47fa-8cab-156ef9b8f331" containerName="istio-proxy"
Nov 03 17:45:12.844684 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.844638    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf48b79-ec61-4698-9c03-74dd7ff368a1" containerName="istio-proxy"
Nov 03 17:45:12.844684 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.844649    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c795d8-dfff-43d8-88a2-de65cc01a51f" containerName="istio-proxy"
Nov 03 17:45:12.844684 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.844664    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf48b79-ec61-4698-9c03-74dd7ff368a1" containerName="istio-proxy"
Nov 03 17:45:12.844684 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.844679    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aec66b4-f062-41dc-b9e7-f69143f51117" containerName="istio-proxy"
Nov 03 17:45:12.844814 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.844689    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f9ab6a0-a561-47fa-8cab-156ef9b8f331" containerName="istio-proxy"
Nov 03 17:45:12.844814 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.844698    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="49075270-6931-42bc-948c-d4cdfc72c5d0" containerName="istio-proxy"
Nov 03 17:45:12.851260 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.851227    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894"
Nov 03 17:45:12.892765 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.892728    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/361ca698-4267-4a8e-8423-a1eb86738052-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894\" (UID: \"361ca698-4267-4a8e-8423-a1eb86738052\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894"
Nov 03 17:45:12.892956 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.892906    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc7kx\" (UniqueName: \"kubernetes.io/projected/361ca698-4267-4a8e-8423-a1eb86738052-kube-api-access-cc7kx\") pod \"payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894\" (UID: \"361ca698-4267-4a8e-8423-a1eb86738052\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894"
Nov 03 17:45:12.905979 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.905945    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3756d5e7-dockercfg-gdrh7\""
Nov 03 17:45:12.909133 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.909087    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894"]
Nov 03 17:45:12.955098 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.955066    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3756d5e7\""
Nov 03 17:45:12.989945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.989910    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559"]
Nov 03 17:45:12.990594 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.990574    6059 cpu_manager.go:414] "containerMap: removing container" podUID="bcf48b79-ec61-4698-9c03-74dd7ff368a1" containerName="istio-proxy"
Nov 03 17:45:12.990594 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.990592    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf48b79-ec61-4698-9c03-74dd7ff368a1" containerName="istio-proxy"
Nov 03 17:45:12.990761 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.990703    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c795d8-dfff-43d8-88a2-de65cc01a51f" containerName="istio-proxy"
Nov 03 17:45:12.994886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.994694    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559"
Nov 03 17:45:12.994886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.994862    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc7kx\" (UniqueName: \"kubernetes.io/projected/361ca698-4267-4a8e-8423-a1eb86738052-kube-api-access-cc7kx\") pod \"payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894\" (UID: \"361ca698-4267-4a8e-8423-a1eb86738052\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894"
Nov 03 17:45:12.995130 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.994962    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/361ca698-4267-4a8e-8423-a1eb86738052-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894\" (UID: \"361ca698-4267-4a8e-8423-a1eb86738052\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894"
Nov 03 17:45:12.995960 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:12.995931    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/361ca698-4267-4a8e-8423-a1eb86738052-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894\" (UID: \"361ca698-4267-4a8e-8423-a1eb86738052\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894"
Nov 03 17:45:13.064443 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:13.057527    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559"]
Nov 03 17:45:13.064443 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:13.060446    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc7kx\" (UniqueName: \"kubernetes.io/projected/361ca698-4267-4a8e-8423-a1eb86738052-kube-api-access-cc7kx\") pod \"payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894\" (UID: \"361ca698-4267-4a8e-8423-a1eb86738052\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894"
Nov 03 17:45:13.096531 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:13.096438    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a60408a-78b9-40ba-a55f-2d62fae10b5c-plugins-config-volume\") pod \"payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559\" (UID: \"1a60408a-78b9-40ba-a55f-2d62fae10b5c\") " pod="openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559"
Nov 03 17:45:13.096531 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:13.096512    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqstw\" (UniqueName: \"kubernetes.io/projected/1a60408a-78b9-40ba-a55f-2d62fae10b5c-kube-api-access-lqstw\") pod \"payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559\" (UID: \"1a60408a-78b9-40ba-a55f-2d62fae10b5c\") " pod="openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559"
Nov 03 17:45:13.167211 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:13.167167    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894"
Nov 03 17:45:13.198314 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:13.198280    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a60408a-78b9-40ba-a55f-2d62fae10b5c-plugins-config-volume\") pod \"payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559\" (UID: \"1a60408a-78b9-40ba-a55f-2d62fae10b5c\") " pod="openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559"
Nov 03 17:45:13.198502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:13.198414    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqstw\" (UniqueName: \"kubernetes.io/projected/1a60408a-78b9-40ba-a55f-2d62fae10b5c-kube-api-access-lqstw\") pod \"payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559\" (UID: \"1a60408a-78b9-40ba-a55f-2d62fae10b5c\") " pod="openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559"
Nov 03 17:45:13.199209 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:13.199182    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a60408a-78b9-40ba-a55f-2d62fae10b5c-plugins-config-volume\") pod \"payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559\" (UID: \"1a60408a-78b9-40ba-a55f-2d62fae10b5c\") " pod="openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559"
Nov 03 17:45:13.262399 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:13.262350    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqstw\" (UniqueName: \"kubernetes.io/projected/1a60408a-78b9-40ba-a55f-2d62fae10b5c-kube-api-access-lqstw\") pod \"payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559\" (UID: \"1a60408a-78b9-40ba-a55f-2d62fae10b5c\") " pod="openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559"
Nov 03 17:45:13.314758 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:13.314724    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559"
Nov 03 17:45:13.571960 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:13.571933    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894"]
Nov 03 17:45:13.705293 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:45:13.705256    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a60408a_78b9_40ba_a55f_2d62fae10b5c.slice/crio-673e2c3be1a191d7ea8191b559d18b89f00a1218d282858a640b09f3e720542f WatchSource:0}: Error finding container 673e2c3be1a191d7ea8191b559d18b89f00a1218d282858a640b09f3e720542f: Status 404 returned error can't find the container with id 673e2c3be1a191d7ea8191b559d18b89f00a1218d282858a640b09f3e720542f
Nov 03 17:45:13.708423 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:13.708398    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559"]
Nov 03 17:45:14.236934 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:14.236888    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559" event={"ID":"1a60408a-78b9-40ba-a55f-2d62fae10b5c","Type":"ContainerStarted","Data":"d7e687727c070daa698bb500e95f2a1aec28cf88cafdc5fd53c90a87b24be785"}
Nov 03 17:45:14.237118 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:14.236959    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559" event={"ID":"1a60408a-78b9-40ba-a55f-2d62fae10b5c","Type":"ContainerStarted","Data":"673e2c3be1a191d7ea8191b559d18b89f00a1218d282858a640b09f3e720542f"}
Nov 03 17:45:14.237118 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:14.236996    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559"
Nov 03 17:45:14.238728 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:14.238693    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894" event={"ID":"361ca698-4267-4a8e-8423-a1eb86738052","Type":"ContainerStarted","Data":"f2167887f971d4e2d29f715c89f0ee3f9797894d684d50018cf675724cebb986"}
Nov 03 17:45:14.238869 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:14.238752    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894"
Nov 03 17:45:14.238869 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:14.238777    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894" event={"ID":"361ca698-4267-4a8e-8423-a1eb86738052","Type":"ContainerStarted","Data":"eae8926f2638dfcdac42ee12d1253b5bc361d5910d5f7df12167ea1b7b8b2117"}
Nov 03 17:45:14.328757 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:14.328726    6059 scope.go:122] "RemoveContainer" containerID="d6e850aa21de6407f92955752505a457c5dcd2aefe79aa3eb1fa19e2467769d8"
Nov 03 17:45:14.364157 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:14.364095    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559" podStartSLOduration=2.364080063 podStartE2EDuration="2.364080063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:14.36339327 +0000 UTC m=+1655.161106044" watchObservedRunningTime="2026-11-03 17:45:14.364080063 +0000 UTC m=+1655.161792844"
Nov 03 17:45:14.489070 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:14.488954    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894" podStartSLOduration=2.488931101 podStartE2EDuration="2.488931101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:14.486448762 +0000 UTC m=+1655.284161547" watchObservedRunningTime="2026-11-03 17:45:14.488931101 +0000 UTC m=+1655.286643886"
Nov 03 17:45:15.248204 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:15.248164    6059 generic.go:356] "Generic (PLEG): container finished" podID="26f88f65-51ef-4d2c-a2a3-451f883d8e15" containerID="a24e9c827cafa87c1623f6737cc496dbc44162541904a0f44a59135caa936d62" exitCode=60
Nov 03 17:45:15.248701 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:15.248240    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x" event={"ID":"26f88f65-51ef-4d2c-a2a3-451f883d8e15","Type":"ContainerDied","Data":"a24e9c827cafa87c1623f6737cc496dbc44162541904a0f44a59135caa936d62"}
Nov 03 17:45:15.248701 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:15.248307    6059 scope.go:122] "RemoveContainer" containerID="d6e850aa21de6407f92955752505a457c5dcd2aefe79aa3eb1fa19e2467769d8"
Nov 03 17:45:15.249132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:15.249101    6059 scope.go:122] "RemoveContainer" containerID="a24e9c827cafa87c1623f6737cc496dbc44162541904a0f44a59135caa936d62"
Nov 03 17:45:15.249455 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:45:15.249383    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x_odh-ai-gateway-infra(26f88f65-51ef-4d2c-a2a3-451f883d8e15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x" podUID="26f88f65-51ef-4d2c-a2a3-451f883d8e15"
Nov 03 17:45:16.329052 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:16.329016    6059 scope.go:122] "RemoveContainer" containerID="9e3260e7a438964428cd5f3ea0efbc08dfe6e42ed09389f3713acd71c71d994d"
Nov 03 17:45:16.395290 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:16.395246    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x"]
Nov 03 17:45:16.574893 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:16.574864    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x"
Nov 03 17:45:16.646618 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:16.646583    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f88f65-51ef-4d2c-a2a3-451f883d8e15-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f88f65-51ef-4d2c-a2a3-451f883d8e15-maas-api-service-ca\") pod \"26f88f65-51ef-4d2c-a2a3-451f883d8e15\" (UID: \"26f88f65-51ef-4d2c-a2a3-451f883d8e15\") "
Nov 03 17:45:16.646794 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:16.646655    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f88f65-51ef-4d2c-a2a3-451f883d8e15-kube-api-access-gk4bm\" (UniqueName: \"kubernetes.io/projected/26f88f65-51ef-4d2c-a2a3-451f883d8e15-kube-api-access-gk4bm\") pod \"26f88f65-51ef-4d2c-a2a3-451f883d8e15\" (UID: \"26f88f65-51ef-4d2c-a2a3-451f883d8e15\") "
Nov 03 17:45:16.647013 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:16.646986    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f88f65-51ef-4d2c-a2a3-451f883d8e15-maas-api-service-ca" pod "26f88f65-51ef-4d2c-a2a3-451f883d8e15" (UID: "26f88f65-51ef-4d2c-a2a3-451f883d8e15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:45:16.647184 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:16.647171    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f88f65-51ef-4d2c-a2a3-451f883d8e15-maas-api-service-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:16.648740 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:16.648717    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f88f65-51ef-4d2c-a2a3-451f883d8e15-kube-api-access-gk4bm" pod "26f88f65-51ef-4d2c-a2a3-451f883d8e15" (UID: "26f88f65-51ef-4d2c-a2a3-451f883d8e15"). InnerVolumeSpecName "kube-api-access-gk4bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:45:16.749457 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:16.749428    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk4bm\" (UniqueName: \"kubernetes.io/projected/26f88f65-51ef-4d2c-a2a3-451f883d8e15-kube-api-access-gk4bm\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:16.850017 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:16.849963    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:17.273648 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:17.273598    6059 generic.go:356] "Generic (PLEG): container finished" podID="7c7a3bea-c23b-49e9-8398-e435046e1187" containerID="191ceb91ae6f4d17160749b6747ee8cff287e2174e85c303fb591a911f18c260" exitCode=60
Nov 03 17:45:17.273821 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:17.273697    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj" event={"ID":"7c7a3bea-c23b-49e9-8398-e435046e1187","Type":"ContainerDied","Data":"191ceb91ae6f4d17160749b6747ee8cff287e2174e85c303fb591a911f18c260"}
Nov 03 17:45:17.273912 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:17.273841    6059 scope.go:122] "RemoveContainer" containerID="9e3260e7a438964428cd5f3ea0efbc08dfe6e42ed09389f3713acd71c71d994d"
Nov 03 17:45:17.274477 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:17.274454    6059 scope.go:122] "RemoveContainer" containerID="191ceb91ae6f4d17160749b6747ee8cff287e2174e85c303fb591a911f18c260"
Nov 03 17:45:17.274884 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:45:17.274856    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj_odh-ai-gateway-infra(7c7a3bea-c23b-49e9-8398-e435046e1187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj" podUID="7c7a3bea-c23b-49e9-8398-e435046e1187"
Nov 03 17:45:17.279301 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:17.276086    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x" event={"ID":"26f88f65-51ef-4d2c-a2a3-451f883d8e15","Type":"ContainerDied","Data":"510f378a4be6531903bf713e1edabd89ece0797dd9dcbd34e814fced063de73b"}
Nov 03 17:45:17.279301 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:17.276174    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x"
Nov 03 17:45:17.289621 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:17.289596    6059 scope.go:122] "RemoveContainer" containerID="a24e9c827cafa87c1623f6737cc496dbc44162541904a0f44a59135caa936d62"
Nov 03 17:45:17.329557 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:17.329528    6059 scope.go:122] "RemoveContainer" containerID="f99999964bfa240d8f8867787104350fac4be9c2d26fa5ba100af222fb8d86c2"
Nov 03 17:45:17.515206 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:17.515168    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x"]
Nov 03 17:45:17.520136 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:17.520096    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895465-q2k5x"]
Nov 03 17:45:18.285926 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:18.285892    6059 generic.go:356] "Generic (PLEG): container finished" podID="e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7" containerID="aaa1f8bb75941d16dcafe9c3c54f35f8dcb96ea36215bc4cf2e0861a0c1eb348" exitCode=60
Nov 03 17:45:18.286106 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:18.285966    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x" event={"ID":"e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7","Type":"ContainerDied","Data":"aaa1f8bb75941d16dcafe9c3c54f35f8dcb96ea36215bc4cf2e0861a0c1eb348"}
Nov 03 17:45:18.286106 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:18.286024    6059 scope.go:122] "RemoveContainer" containerID="f99999964bfa240d8f8867787104350fac4be9c2d26fa5ba100af222fb8d86c2"
Nov 03 17:45:18.286501 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:18.286479    6059 scope.go:122] "RemoveContainer" containerID="aaa1f8bb75941d16dcafe9c3c54f35f8dcb96ea36215bc4cf2e0861a0c1eb348"
Nov 03 17:45:18.286844 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:45:18.286813    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x_odh-ai-gateway-infra(e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x" podUID="e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7"
Nov 03 17:45:18.328940 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:18.328907    6059 scope.go:122] "RemoveContainer" containerID="178af4967979186813fa9362412a206e1bff362dcfbb516d60d4cc038dfabe4a"
Nov 03 17:45:18.375136 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:45:18.375102    6059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a0bc00_4ddf_4b2f_ac18_c6fcc92be4c7.slice/crio-conmon-aaa1f8bb75941d16dcafe9c3c54f35f8dcb96ea36215bc4cf2e0861a0c1eb348.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 17:45:18.415909 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:18.415877    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj"]
Nov 03 17:45:18.574935 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:18.574907    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj"
Nov 03 17:45:18.672681 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:18.672640    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7a3bea-c23b-49e9-8398-e435046e1187-kube-api-access-cd8vv\" (UniqueName: \"kubernetes.io/projected/7c7a3bea-c23b-49e9-8398-e435046e1187-kube-api-access-cd8vv\") pod \"7c7a3bea-c23b-49e9-8398-e435046e1187\" (UID: \"7c7a3bea-c23b-49e9-8398-e435046e1187\") "
Nov 03 17:45:18.672889 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:18.672739    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c7a3bea-c23b-49e9-8398-e435046e1187-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7a3bea-c23b-49e9-8398-e435046e1187-maas-api-service-ca\") pod \"7c7a3bea-c23b-49e9-8398-e435046e1187\" (UID: \"7c7a3bea-c23b-49e9-8398-e435046e1187\") "
Nov 03 17:45:18.673112 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:18.673084    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7a3bea-c23b-49e9-8398-e435046e1187-maas-api-service-ca" pod "7c7a3bea-c23b-49e9-8398-e435046e1187" (UID: "7c7a3bea-c23b-49e9-8398-e435046e1187"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:45:18.674787 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:18.674765    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7a3bea-c23b-49e9-8398-e435046e1187-kube-api-access-cd8vv" pod "7c7a3bea-c23b-49e9-8398-e435046e1187" (UID: "7c7a3bea-c23b-49e9-8398-e435046e1187"). InnerVolumeSpecName "kube-api-access-cd8vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:45:18.774686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:18.774639    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd8vv\" (UniqueName: \"kubernetes.io/projected/7c7a3bea-c23b-49e9-8398-e435046e1187-kube-api-access-cd8vv\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:18.774686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:18.774680    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7a3bea-c23b-49e9-8398-e435046e1187-maas-api-service-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:19.300958 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:19.300926    6059 generic.go:356] "Generic (PLEG): container finished" podID="4b235543-821b-4bd5-9a77-e020368fbcf9" containerID="784ae334882cfd78b9ee7f4d16df095698daccdd3562d178df41d52fc599275a" exitCode=60
Nov 03 17:45:19.301107 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:19.300997    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c" event={"ID":"4b235543-821b-4bd5-9a77-e020368fbcf9","Type":"ContainerDied","Data":"784ae334882cfd78b9ee7f4d16df095698daccdd3562d178df41d52fc599275a"}
Nov 03 17:45:19.301107 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:19.301060    6059 scope.go:122] "RemoveContainer" containerID="178af4967979186813fa9362412a206e1bff362dcfbb516d60d4cc038dfabe4a"
Nov 03 17:45:19.301537 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:19.301515    6059 scope.go:122] "RemoveContainer" containerID="784ae334882cfd78b9ee7f4d16df095698daccdd3562d178df41d52fc599275a"
Nov 03 17:45:19.301973 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:45:19.301947    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895465-2ds7c_odh-ai-gateway-infra(4b235543-821b-4bd5-9a77-e020368fbcf9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c" podUID="4b235543-821b-4bd5-9a77-e020368fbcf9"
Nov 03 17:45:19.307449 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:19.307415    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj" event={"ID":"7c7a3bea-c23b-49e9-8398-e435046e1187","Type":"ContainerDied","Data":"826390d786a897c176639e8f55a1b84c5bbd7249b564b65064c812a8374d998d"}
Nov 03 17:45:19.307606 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:19.307425    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj"
Nov 03 17:45:19.317611 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:19.317587    6059 scope.go:122] "RemoveContainer" containerID="191ceb91ae6f4d17160749b6747ee8cff287e2174e85c303fb591a911f18c260"
Nov 03 17:45:19.344895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:19.344860    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f88f65-51ef-4d2c-a2a3-451f883d8e15" path="/var/lib/kubelet/pods/26f88f65-51ef-4d2c-a2a3-451f883d8e15/volumes"
Nov 03 17:45:19.420920 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:19.420884    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x"]
Nov 03 17:45:19.564350 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:19.564256    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj"]
Nov 03 17:45:19.569087 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:19.569057    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895465-v4qrj"]
Nov 03 17:45:19.612884 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:19.612848    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x"
Nov 03 17:45:19.686122 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:19.686084    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7-maas-api-service-ca\") pod \"e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7\" (UID: \"e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7\") "
Nov 03 17:45:19.686273 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:19.686259    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7-kube-api-access-wfgd9\" (UniqueName: \"kubernetes.io/projected/e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7-kube-api-access-wfgd9\") pod \"e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7\" (UID: \"e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7\") "
Nov 03 17:45:19.686525 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:19.686499    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7-maas-api-service-ca" pod "e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7" (UID: "e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:45:19.686718 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:19.686696    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7-maas-api-service-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:19.688311 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:19.688288    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7-kube-api-access-wfgd9" pod "e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7" (UID: "e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7"). InnerVolumeSpecName "kube-api-access-wfgd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:45:19.787696 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:19.787653    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfgd9\" (UniqueName: \"kubernetes.io/projected/e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7-kube-api-access-wfgd9\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:20.316839 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:20.316808    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x"
Nov 03 17:45:20.317027 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:20.316802    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x" event={"ID":"e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7","Type":"ContainerDied","Data":"1b38ff43b9fdad310f52093f620f07ddb92a5e1907ab22532d6fe14621135f0a"}
Nov 03 17:45:20.317027 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:20.316983    6059 scope.go:122] "RemoveContainer" containerID="aaa1f8bb75941d16dcafe9c3c54f35f8dcb96ea36215bc4cf2e0861a0c1eb348"
Nov 03 17:45:20.438133 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:20.438095    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c"]
Nov 03 17:45:20.507745 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:20.507706    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x"]
Nov 03 17:45:20.512405 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:20.512350    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-af9e3c-29895465-xr66x"]
Nov 03 17:45:20.589651 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:20.589621    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c"
Nov 03 17:45:20.699270 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:20.699232    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b235543-821b-4bd5-9a77-e020368fbcf9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b235543-821b-4bd5-9a77-e020368fbcf9-maas-api-service-ca\") pod \"4b235543-821b-4bd5-9a77-e020368fbcf9\" (UID: \"4b235543-821b-4bd5-9a77-e020368fbcf9\") "
Nov 03 17:45:20.699463 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:20.699447    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b235543-821b-4bd5-9a77-e020368fbcf9-kube-api-access-jmpf9\" (UniqueName: \"kubernetes.io/projected/4b235543-821b-4bd5-9a77-e020368fbcf9-kube-api-access-jmpf9\") pod \"4b235543-821b-4bd5-9a77-e020368fbcf9\" (UID: \"4b235543-821b-4bd5-9a77-e020368fbcf9\") "
Nov 03 17:45:20.699667 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:20.699637    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b235543-821b-4bd5-9a77-e020368fbcf9-maas-api-service-ca" pod "4b235543-821b-4bd5-9a77-e020368fbcf9" (UID: "4b235543-821b-4bd5-9a77-e020368fbcf9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:45:20.699893 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:20.699877    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b235543-821b-4bd5-9a77-e020368fbcf9-maas-api-service-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:20.701743 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:20.701724    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b235543-821b-4bd5-9a77-e020368fbcf9-kube-api-access-jmpf9" pod "4b235543-821b-4bd5-9a77-e020368fbcf9" (UID: "4b235543-821b-4bd5-9a77-e020368fbcf9"). InnerVolumeSpecName "kube-api-access-jmpf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:45:20.800833 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:20.800798    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmpf9\" (UniqueName: \"kubernetes.io/projected/4b235543-821b-4bd5-9a77-e020368fbcf9-kube-api-access-jmpf9\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:45:21.330395 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:21.330352    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c"
Nov 03 17:45:21.336905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:21.336869    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c7a3bea-c23b-49e9-8398-e435046e1187" path="/var/lib/kubelet/pods/7c7a3bea-c23b-49e9-8398-e435046e1187/volumes"
Nov 03 17:45:21.337439 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:21.337416    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7" path="/var/lib/kubelet/pods/e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7/volumes"
Nov 03 17:45:21.338036 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:21.338013    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c" event={"ID":"4b235543-821b-4bd5-9a77-e020368fbcf9","Type":"ContainerDied","Data":"30bf6626cfea2d2a5cca98859a782b4558512c608e1b23e518a92eb94efe6f8e"}
Nov 03 17:45:21.338148 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:21.338077    6059 scope.go:122] "RemoveContainer" containerID="784ae334882cfd78b9ee7f4d16df095698daccdd3562d178df41d52fc599275a"
Nov 03 17:45:21.421032 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:21.420989    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:21.502327 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:21.502284    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c"]
Nov 03 17:45:21.506542 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:21.506515    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895465-2ds7c"]
Nov 03 17:45:23.337047 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:23.337007    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b235543-821b-4bd5-9a77-e020368fbcf9" path="/var/lib/kubelet/pods/4b235543-821b-4bd5-9a77-e020368fbcf9/volumes"
Nov 03 17:45:25.251705 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:25.251675    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559"
Nov 03 17:45:25.252218 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:25.251753    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894"
Nov 03 17:45:26.021425 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:26.021388    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:27.067850 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:27.067815    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm"]
Nov 03 17:45:27.118812 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:27.118774    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7535f460-tdccm"]
Nov 03 17:45:27.337134 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:27.337051    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46171787-934f-473e-9de9-b9d4e0f8f68a" path="/var/lib/kubelet/pods/46171787-934f-473e-9de9-b9d4e0f8f68a/volumes"
Nov 03 17:45:30.659759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:30.659716    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:35.126845 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:35.126806    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:36.863305 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:45:36.863263    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:45:39.621880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:39.621830    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:39.800784 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:39.800745    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:41.650710 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:41.650673    6059 scope.go:122] "RemoveContainer" containerID="a104a9ddd56ca96b02d8733540d318115986928bf0a40f23a3805e86e8436cb4"
Nov 03 17:45:41.663279 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:41.663247    6059 scope.go:122] "RemoveContainer" containerID="8c87a2c1de7b4ef354fc8a84264d5771ac91179a03806839abb335ce654ca2e1"
Nov 03 17:45:41.673424 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:41.673397    6059 scope.go:122] "RemoveContainer" containerID="a236d0f363bffcd13a49973e4983cb6955e9f284c32522e7a10cb2863cb3a379"
Nov 03 17:45:41.682628 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:41.682604    6059 scope.go:122] "RemoveContainer" containerID="4e78efa48f473360f68952f45c99c1c6b10ef31486f46d7672b84150cd814287"
Nov 03 17:45:41.692531 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:41.692510    6059 scope.go:122] "RemoveContainer" containerID="6d484d3a5a707b1cc50ef52c28452132ca6f8639b5583eeb943d3f411bf68a6b"
Nov 03 17:45:41.701813 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:41.701791    6059 scope.go:122] "RemoveContainer" containerID="bf97440cc175998241950cd45c4d75688b66ba93e3bfc3e3c8bc880ddbbf566f"
Nov 03 17:45:41.711191 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:41.711171    6059 scope.go:122] "RemoveContainer" containerID="dc3f25b7b60f61a336a4d4a0824c32d8331738e6c3f1fbe69d4b84fb9fe73f20"
Nov 03 17:45:43.052113 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:43.052065    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ddc4390-j9zbn"]
Nov 03 17:45:43.103306 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:43.103267    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ddc4390-j9zbn"]
Nov 03 17:45:43.337246 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:43.337166    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49fb5c34-7a16-4094-b33c-2c511e1c1bd9" path="/var/lib/kubelet/pods/49fb5c34-7a16-4094-b33c-2c511e1c1bd9/volumes"
Nov 03 17:45:44.161124 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:44.161086    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:48.603751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:48.603715    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:48.783295 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:48.783262    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:53.434901 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:53.434864    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:45:58.253420 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:45:58.253382    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:00.103382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:00.103315    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:02.918627 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:02.918588    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:04.106024 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.105987    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx"]
Nov 03 17:46:04.106583 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106568    6059 cpu_manager.go:414] "containerMap: removing container" podUID="4b235543-821b-4bd5-9a77-e020368fbcf9" containerName="cleanup"
Nov 03 17:46:04.106636 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106583    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b235543-821b-4bd5-9a77-e020368fbcf9" containerName="cleanup"
Nov 03 17:46:04.106636 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106602    6059 cpu_manager.go:414] "containerMap: removing container" podUID="7c7a3bea-c23b-49e9-8398-e435046e1187" containerName="cleanup"
Nov 03 17:46:04.106636 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106606    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7a3bea-c23b-49e9-8398-e435046e1187" containerName="cleanup"
Nov 03 17:46:04.106636 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106614    6059 cpu_manager.go:414] "containerMap: removing container" podUID="7c7a3bea-c23b-49e9-8398-e435046e1187" containerName="cleanup"
Nov 03 17:46:04.106636 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106620    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7a3bea-c23b-49e9-8398-e435046e1187" containerName="cleanup"
Nov 03 17:46:04.106636 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106628    6059 cpu_manager.go:414] "containerMap: removing container" podUID="e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7" containerName="cleanup"
Nov 03 17:46:04.106636 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106632    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7" containerName="cleanup"
Nov 03 17:46:04.106636 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106639    6059 cpu_manager.go:414] "containerMap: removing container" podUID="26f88f65-51ef-4d2c-a2a3-451f883d8e15" containerName="cleanup"
Nov 03 17:46:04.106886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106643    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f88f65-51ef-4d2c-a2a3-451f883d8e15" containerName="cleanup"
Nov 03 17:46:04.106886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106655    6059 cpu_manager.go:414] "containerMap: removing container" podUID="4b235543-821b-4bd5-9a77-e020368fbcf9" containerName="cleanup"
Nov 03 17:46:04.106886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106659    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b235543-821b-4bd5-9a77-e020368fbcf9" containerName="cleanup"
Nov 03 17:46:04.106886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106708    6059 cpu_manager.go:414] "containerMap: removing container" podUID="26f88f65-51ef-4d2c-a2a3-451f883d8e15" containerName="cleanup"
Nov 03 17:46:04.106886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106714    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f88f65-51ef-4d2c-a2a3-451f883d8e15" containerName="cleanup"
Nov 03 17:46:04.106886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106719    6059 cpu_manager.go:414] "containerMap: removing container" podUID="e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7" containerName="cleanup"
Nov 03 17:46:04.106886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106725    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7" containerName="cleanup"
Nov 03 17:46:04.106886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106797    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f88f65-51ef-4d2c-a2a3-451f883d8e15" containerName="cleanup"
Nov 03 17:46:04.106886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106806    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7" containerName="cleanup"
Nov 03 17:46:04.106886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106811    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b235543-821b-4bd5-9a77-e020368fbcf9" containerName="cleanup"
Nov 03 17:46:04.106886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106818    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7a3bea-c23b-49e9-8398-e435046e1187" containerName="cleanup"
Nov 03 17:46:04.106886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106824    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7a3bea-c23b-49e9-8398-e435046e1187" containerName="cleanup"
Nov 03 17:46:04.106886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106830    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7" containerName="cleanup"
Nov 03 17:46:04.106886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106835    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f88f65-51ef-4d2c-a2a3-451f883d8e15" containerName="cleanup"
Nov 03 17:46:04.106886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106842    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b235543-821b-4bd5-9a77-e020368fbcf9" containerName="cleanup"
Nov 03 17:46:04.106886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.106847    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f88f65-51ef-4d2c-a2a3-451f883d8e15" containerName="cleanup"
Nov 03 17:46:04.110163 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.110145    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx"
Nov 03 17:46:04.204706 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.204666    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwxwj\" (UniqueName: \"kubernetes.io/projected/60403592-f827-4ff1-856a-47ccc007eb09-kube-api-access-gwxwj\") pod \"maas-api-revoke-sub-3f3e4acc-wqgxx\" (UID: \"60403592-f827-4ff1-856a-47ccc007eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx"
Nov 03 17:46:04.204877 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.204819    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60403592-f827-4ff1-856a-47ccc007eb09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f3e4acc-wqgxx\" (UID: \"60403592-f827-4ff1-856a-47ccc007eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx"
Nov 03 17:46:04.204950 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.204924    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60403592-f827-4ff1-856a-47ccc007eb09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f3e4acc-wqgxx\" (UID: \"60403592-f827-4ff1-856a-47ccc007eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx"
Nov 03 17:46:04.218626 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.218595    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:46:04.218626 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.218612    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:46:04.218626 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.218612    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:46:04.218822 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.218661    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:46:04.221593 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.221574    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx"]
Nov 03 17:46:04.306104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.306047    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwxwj\" (UniqueName: \"kubernetes.io/projected/60403592-f827-4ff1-856a-47ccc007eb09-kube-api-access-gwxwj\") pod \"maas-api-revoke-sub-3f3e4acc-wqgxx\" (UID: \"60403592-f827-4ff1-856a-47ccc007eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx"
Nov 03 17:46:04.306297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.306185    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60403592-f827-4ff1-856a-47ccc007eb09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f3e4acc-wqgxx\" (UID: \"60403592-f827-4ff1-856a-47ccc007eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx"
Nov 03 17:46:04.306297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.306243    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60403592-f827-4ff1-856a-47ccc007eb09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f3e4acc-wqgxx\" (UID: \"60403592-f827-4ff1-856a-47ccc007eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx"
Nov 03 17:46:04.306833 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.306808    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60403592-f827-4ff1-856a-47ccc007eb09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f3e4acc-wqgxx\" (UID: \"60403592-f827-4ff1-856a-47ccc007eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx"
Nov 03 17:46:04.308556 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.308537    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60403592-f827-4ff1-856a-47ccc007eb09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f3e4acc-wqgxx\" (UID: \"60403592-f827-4ff1-856a-47ccc007eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx"
Nov 03 17:46:04.370140 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.370068    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwxwj\" (UniqueName: \"kubernetes.io/projected/60403592-f827-4ff1-856a-47ccc007eb09-kube-api-access-gwxwj\") pod \"maas-api-revoke-sub-3f3e4acc-wqgxx\" (UID: \"60403592-f827-4ff1-856a-47ccc007eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx"
Nov 03 17:46:04.424275 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.424240    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx"
Nov 03 17:46:04.821758 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:46:04.821722    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60403592_f827_4ff1_856a_47ccc007eb09.slice/crio-538a928a376cbae55b06ab46ef3600729748093b52b1d1db5bf931ec60a12889 WatchSource:0}: Error finding container 538a928a376cbae55b06ab46ef3600729748093b52b1d1db5bf931ec60a12889: Status 404 returned error can't find the container with id 538a928a376cbae55b06ab46ef3600729748093b52b1d1db5bf931ec60a12889
Nov 03 17:46:04.824021 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:04.823995    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx"]
Nov 03 17:46:05.388925 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:05.388839    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7481-50a6e4"]
Nov 03 17:46:05.389820 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:05.389797    6059 cpu_manager.go:414] "containerMap: removing container" podUID="7c7a3bea-c23b-49e9-8398-e435046e1187" containerName="cleanup"
Nov 03 17:46:05.389873 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:05.389824    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7a3bea-c23b-49e9-8398-e435046e1187" containerName="cleanup"
Nov 03 17:46:05.389873 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:05.389841    6059 cpu_manager.go:414] "containerMap: removing container" podUID="e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7" containerName="cleanup"
Nov 03 17:46:05.389873 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:05.389849    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7" containerName="cleanup"
Nov 03 17:46:05.389873 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:05.389865    6059 cpu_manager.go:414] "containerMap: removing container" podUID="26f88f65-51ef-4d2c-a2a3-451f883d8e15" containerName="cleanup"
Nov 03 17:46:05.390015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:05.389874    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f88f65-51ef-4d2c-a2a3-451f883d8e15" containerName="cleanup"
Nov 03 17:46:05.390015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:05.389895    6059 cpu_manager.go:414] "containerMap: removing container" podUID="4b235543-821b-4bd5-9a77-e020368fbcf9" containerName="cleanup"
Nov 03 17:46:05.390015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:05.389906    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b235543-821b-4bd5-9a77-e020368fbcf9" containerName="cleanup"
Nov 03 17:46:05.390129 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:05.390093    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7a3bea-c23b-49e9-8398-e435046e1187" containerName="cleanup"
Nov 03 17:46:05.390129 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:05.390105    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a0bc00-4ddf-4b2f-ac18-c6fcc92be4c7" containerName="cleanup"
Nov 03 17:46:05.390129 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:05.390116    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b235543-821b-4bd5-9a77-e020368fbcf9" containerName="cleanup"
Nov 03 17:46:05.394277 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:05.394255    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-50a6e4"
Nov 03 17:46:05.448669 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:05.448629    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-50a6e4"]
Nov 03 17:46:05.458004 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:05.457979    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hjccf\""
Nov 03 17:46:05.522939 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:05.522908    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjk4s\" (UniqueName: \"kubernetes.io/projected/98153070-238a-43ea-9a8d-ab48efdcfcdf-kube-api-access-sjk4s\") pod \"test-curl-7481-50a6e4\" (UID: \"98153070-238a-43ea-9a8d-ab48efdcfcdf\") " pod="opendatahub/test-curl-7481-50a6e4"
Nov 03 17:46:05.624670 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:05.624632    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjk4s\" (UniqueName: \"kubernetes.io/projected/98153070-238a-43ea-9a8d-ab48efdcfcdf-kube-api-access-sjk4s\") pod \"test-curl-7481-50a6e4\" (UID: \"98153070-238a-43ea-9a8d-ab48efdcfcdf\") " pod="opendatahub/test-curl-7481-50a6e4"
Nov 03 17:46:05.688053 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:05.687973    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjk4s\" (UniqueName: \"kubernetes.io/projected/98153070-238a-43ea-9a8d-ab48efdcfcdf-kube-api-access-sjk4s\") pod \"test-curl-7481-50a6e4\" (UID: \"98153070-238a-43ea-9a8d-ab48efdcfcdf\") " pod="opendatahub/test-curl-7481-50a6e4"
Nov 03 17:46:05.698560 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:05.698518    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx" event={"ID":"60403592-f827-4ff1-856a-47ccc007eb09","Type":"ContainerStarted","Data":"907ea931adc6c01dbd70c00fd414e76460f1fca13b4e60527830c38501a4ba77"}
Nov 03 17:46:05.698741 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:05.698588    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx" event={"ID":"60403592-f827-4ff1-856a-47ccc007eb09","Type":"ContainerStarted","Data":"538a928a376cbae55b06ab46ef3600729748093b52b1d1db5bf931ec60a12889"}
Nov 03 17:46:05.708934 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:05.708908    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-50a6e4"
Nov 03 17:46:05.820641 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:05.820585    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx" podStartSLOduration=1.8205711660000001 podStartE2EDuration="1.820571166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:05.818008422 +0000 UTC m=+1706.615721207" watchObservedRunningTime="2026-11-03 17:46:05.820571166 +0000 UTC m=+1706.618283947"
Nov 03 17:46:06.087344 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:46:06.087307    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98153070_238a_43ea_9a8d_ab48efdcfcdf.slice/crio-5314e37c9d4ebd4c624122e32043f0cd48e635c0250694ba1d3ff1cf9d6b0796 WatchSource:0}: Error finding container 5314e37c9d4ebd4c624122e32043f0cd48e635c0250694ba1d3ff1cf9d6b0796: Status 404 returned error can't find the container with id 5314e37c9d4ebd4c624122e32043f0cd48e635c0250694ba1d3ff1cf9d6b0796
Nov 03 17:46:06.090129 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:06.090101    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7481-50a6e4"]
Nov 03 17:46:06.708567 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:06.708531    6059 generic.go:356] "Generic (PLEG): container finished" podID="60403592-f827-4ff1-856a-47ccc007eb09" containerID="907ea931adc6c01dbd70c00fd414e76460f1fca13b4e60527830c38501a4ba77" exitCode=0
Nov 03 17:46:06.709025 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:06.708624    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx" event={"ID":"60403592-f827-4ff1-856a-47ccc007eb09","Type":"ContainerDied","Data":"907ea931adc6c01dbd70c00fd414e76460f1fca13b4e60527830c38501a4ba77"}
Nov 03 17:46:06.710385 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:06.710347    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-50a6e4" event={"ID":"98153070-238a-43ea-9a8d-ab48efdcfcdf","Type":"ContainerStarted","Data":"106ed3cbbed2087a321006ed7413f1070211a9ad8a3716e4e04a737bc804ff2d"}
Nov 03 17:46:06.710486 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:06.710396    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7481-50a6e4" event={"ID":"98153070-238a-43ea-9a8d-ab48efdcfcdf","Type":"ContainerStarted","Data":"5314e37c9d4ebd4c624122e32043f0cd48e635c0250694ba1d3ff1cf9d6b0796"}
Nov 03 17:46:06.864994 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:46:06.864940    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:46:06.968735 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:06.968633    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7481-50a6e4" podStartSLOduration=1.96861632 podStartE2EDuration="1.96861632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:06.967557846 +0000 UTC m=+1707.765270631" watchObservedRunningTime="2026-11-03 17:46:06.96861632 +0000 UTC m=+1707.766329094"
Nov 03 17:46:07.473258 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:07.473222    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:07.878614 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:07.878582    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx"
Nov 03 17:46:07.958684 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:07.958626    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60403592-f827-4ff1-856a-47ccc007eb09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60403592-f827-4ff1-856a-47ccc007eb09-maas-api-openshift-ca\") pod \"60403592-f827-4ff1-856a-47ccc007eb09\" (UID: \"60403592-f827-4ff1-856a-47ccc007eb09\") "
Nov 03 17:46:07.958890 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:07.958862    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60403592-f827-4ff1-856a-47ccc007eb09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60403592-f827-4ff1-856a-47ccc007eb09-maas-api-xks-ca\") pod \"60403592-f827-4ff1-856a-47ccc007eb09\" (UID: \"60403592-f827-4ff1-856a-47ccc007eb09\") "
Nov 03 17:46:07.958964 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:07.958917    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60403592-f827-4ff1-856a-47ccc007eb09-kube-api-access-gwxwj\" (UniqueName: \"kubernetes.io/projected/60403592-f827-4ff1-856a-47ccc007eb09-kube-api-access-gwxwj\") pod \"60403592-f827-4ff1-856a-47ccc007eb09\" (UID: \"60403592-f827-4ff1-856a-47ccc007eb09\") "
Nov 03 17:46:07.959023 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:07.958991    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60403592-f827-4ff1-856a-47ccc007eb09-maas-api-openshift-ca" pod "60403592-f827-4ff1-856a-47ccc007eb09" (UID: "60403592-f827-4ff1-856a-47ccc007eb09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:07.959346 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:07.959325    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60403592-f827-4ff1-856a-47ccc007eb09-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:07.960919 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:07.960890    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60403592-f827-4ff1-856a-47ccc007eb09-maas-api-xks-ca" pod "60403592-f827-4ff1-856a-47ccc007eb09" (UID: "60403592-f827-4ff1-856a-47ccc007eb09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:46:07.961032 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:07.961013    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60403592-f827-4ff1-856a-47ccc007eb09-kube-api-access-gwxwj" pod "60403592-f827-4ff1-856a-47ccc007eb09" (UID: "60403592-f827-4ff1-856a-47ccc007eb09"). InnerVolumeSpecName "kube-api-access-gwxwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:46:08.060666 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:08.060581    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60403592-f827-4ff1-856a-47ccc007eb09-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:08.060666 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:08.060611    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwxwj\" (UniqueName: \"kubernetes.io/projected/60403592-f827-4ff1-856a-47ccc007eb09-kube-api-access-gwxwj\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:08.403073 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:08.402877    6059 patch_prober.go:29] interesting pod/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": read tcp 10.133.0.2:39206->10.133.0.168:15021: read: connection reset by peer" start-of-body=
Nov 03 17:46:08.403073 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:08.402939    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x" podUID="17bc44f2-4185-4ce3-a127-cf0dc29b9583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": read tcp 10.133.0.2:39206->10.133.0.168:15021: read: connection reset by peer"
Nov 03 17:46:08.462800 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:08.462760    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7481-50a6e4"]
Nov 03 17:46:08.465766 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:08.465734    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7481-50a6e4"]
Nov 03 17:46:08.725726 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:08.725692    6059 generic.go:356] "Generic (PLEG): container finished" podID="17bc44f2-4185-4ce3-a127-cf0dc29b9583" containerID="81eff07997580368b18f8080d233d0f6c3a34929a413476ebf4e935c4ec4d076" exitCode=0
Nov 03 17:46:08.725908 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:08.725764    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x" event={"ID":"17bc44f2-4185-4ce3-a127-cf0dc29b9583","Type":"ContainerDied","Data":"81eff07997580368b18f8080d233d0f6c3a34929a413476ebf4e935c4ec4d076"}
Nov 03 17:46:08.726242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:08.726221    6059 scope.go:122] "RemoveContainer" containerID="81eff07997580368b18f8080d233d0f6c3a34929a413476ebf4e935c4ec4d076"
Nov 03 17:46:08.727418 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:08.727387    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx" event={"ID":"60403592-f827-4ff1-856a-47ccc007eb09","Type":"ContainerDied","Data":"538a928a376cbae55b06ab46ef3600729748093b52b1d1db5bf931ec60a12889"}
Nov 03 17:46:08.727537 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:08.727441    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="538a928a376cbae55b06ab46ef3600729748093b52b1d1db5bf931ec60a12889"
Nov 03 17:46:08.727537 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:08.727458    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx"
Nov 03 17:46:08.727537 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:08.727459    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7481-50a6e4" podUID="98153070-238a-43ea-9a8d-ab48efdcfcdf" containerName="test-curl-7481-50a6e4" containerID="cri-o://106ed3cbbed2087a321006ed7413f1070211a9ad8a3716e4e04a737bc804ff2d" gracePeriod=2
Nov 03 17:46:08.727881 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:08.727850    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:46:08.727962 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:08.727937    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:46:08.728015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:08.728004    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:46:08.955456 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:46:08.955415    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7481-50a6e4\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="98153070-238a-43ea-9a8d-ab48efdcfcdf" pod="opendatahub/test-curl-7481-50a6e4"
Nov 03 17:46:09.047634 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:09.047551    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:09.394967 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:46:09.394865    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7481-50a6e4\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="98153070-238a-43ea-9a8d-ab48efdcfcdf" pod="opendatahub/test-curl-7481-50a6e4"
Nov 03 17:46:09.757838 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:09.757799    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x" event={"ID":"17bc44f2-4185-4ce3-a127-cf0dc29b9583","Type":"ContainerStarted","Data":"30588c3d061767c44a9505418b9b67b1b0d9f5a282ff05ce4af8853a0759db21"}
Nov 03 17:46:10.653381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:10.652773    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:46:10.653381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:10.652918    6059 patch_prober.go:29] interesting pod/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 03 17:46:10.653381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:10.652965    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x" podUID="17bc44f2-4185-4ce3-a127-cf0dc29b9583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 03 17:46:11.073923 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:11.073893    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-50a6e4"
Nov 03 17:46:11.202660 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:11.202620    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98153070-238a-43ea-9a8d-ab48efdcfcdf-kube-api-access-sjk4s\" (UniqueName: \"kubernetes.io/projected/98153070-238a-43ea-9a8d-ab48efdcfcdf-kube-api-access-sjk4s\") pod \"98153070-238a-43ea-9a8d-ab48efdcfcdf\" (UID: \"98153070-238a-43ea-9a8d-ab48efdcfcdf\") "
Nov 03 17:46:11.204724 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:11.204695    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98153070-238a-43ea-9a8d-ab48efdcfcdf-kube-api-access-sjk4s" pod "98153070-238a-43ea-9a8d-ab48efdcfcdf" (UID: "98153070-238a-43ea-9a8d-ab48efdcfcdf"). InnerVolumeSpecName "kube-api-access-sjk4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:46:11.304070 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:11.304030    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjk4s\" (UniqueName: \"kubernetes.io/projected/98153070-238a-43ea-9a8d-ab48efdcfcdf-kube-api-access-sjk4s\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:11.358816 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:11.358780    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98153070-238a-43ea-9a8d-ab48efdcfcdf" path="/var/lib/kubelet/pods/98153070-238a-43ea-9a8d-ab48efdcfcdf/volumes"
Nov 03 17:46:11.653900 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:11.653803    6059 patch_prober.go:29] interesting pod/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 03 17:46:11.654513 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:11.654468    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x" podUID="17bc44f2-4185-4ce3-a127-cf0dc29b9583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 03 17:46:11.774193 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:11.774157    6059 generic.go:356] "Generic (PLEG): container finished" podID="98153070-238a-43ea-9a8d-ab48efdcfcdf" containerID="106ed3cbbed2087a321006ed7413f1070211a9ad8a3716e4e04a737bc804ff2d" exitCode=137
Nov 03 17:46:11.774405 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:11.774204    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7481-50a6e4"
Nov 03 17:46:11.774405 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:11.774226    6059 scope.go:122] "RemoveContainer" containerID="106ed3cbbed2087a321006ed7413f1070211a9ad8a3716e4e04a737bc804ff2d"
Nov 03 17:46:11.787732 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:11.787592    6059 scope.go:122] "RemoveContainer" containerID="106ed3cbbed2087a321006ed7413f1070211a9ad8a3716e4e04a737bc804ff2d"
Nov 03 17:46:11.788023 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:46:11.787992    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"106ed3cbbed2087a321006ed7413f1070211a9ad8a3716e4e04a737bc804ff2d\": container with ID starting with 106ed3cbbed2087a321006ed7413f1070211a9ad8a3716e4e04a737bc804ff2d not found: ID does not exist" containerID="106ed3cbbed2087a321006ed7413f1070211a9ad8a3716e4e04a737bc804ff2d"
Nov 03 17:46:11.788132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:11.788029    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"106ed3cbbed2087a321006ed7413f1070211a9ad8a3716e4e04a737bc804ff2d"} err="failed to get container status \"106ed3cbbed2087a321006ed7413f1070211a9ad8a3716e4e04a737bc804ff2d\": rpc error: code = NotFound desc = could not find container \"106ed3cbbed2087a321006ed7413f1070211a9ad8a3716e4e04a737bc804ff2d\": container with ID starting with 106ed3cbbed2087a321006ed7413f1070211a9ad8a3716e4e04a737bc804ff2d not found: ID does not exist"
Nov 03 17:46:11.951833 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:11.951723    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:12.646811 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:12.646771    6059 patch_prober.go:29] interesting pod/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 03 17:46:12.647098 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:12.647069    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x" podUID="17bc44f2-4185-4ce3-a127-cf0dc29b9583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 03 17:46:13.652380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:13.651799    6059 patch_prober.go:29] interesting pod/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 03 17:46:13.652380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:13.651860    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x" podUID="17bc44f2-4185-4ce3-a127-cf0dc29b9583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 03 17:46:14.063315 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:14.063276    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:14.641774 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:14.641734    6059 patch_prober.go:29] interesting pod/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 03 17:46:14.642066 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:14.642032    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x" podUID="17bc44f2-4185-4ce3-a127-cf0dc29b9583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 03 17:46:14.883594 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:14.883548    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b5a3c08-mhqqf"]
Nov 03 17:46:14.885102 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:14.885071    6059 cpu_manager.go:414] "containerMap: removing container" podUID="98153070-238a-43ea-9a8d-ab48efdcfcdf" containerName="test-curl-7481-50a6e4"
Nov 03 17:46:14.885102 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:14.885103    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98153070-238a-43ea-9a8d-ab48efdcfcdf" containerName="test-curl-7481-50a6e4"
Nov 03 17:46:14.885287 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:14.885202    6059 cpu_manager.go:414] "containerMap: removing container" podUID="60403592-f827-4ff1-856a-47ccc007eb09" containerName="revoke-keys"
Nov 03 17:46:14.885287 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:14.885212    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60403592-f827-4ff1-856a-47ccc007eb09" containerName="revoke-keys"
Nov 03 17:46:14.885533 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:14.885514    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="98153070-238a-43ea-9a8d-ab48efdcfcdf" containerName="test-curl-7481-50a6e4"
Nov 03 17:46:14.885597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:14.885560    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="60403592-f827-4ff1-856a-47ccc007eb09" containerName="revoke-keys"
Nov 03 17:46:14.892295 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:14.892221    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5a3c08-mhqqf"
Nov 03 17:46:14.946400 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:14.946348    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b5a3c08-mhqqf"]
Nov 03 17:46:14.974671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:14.974633    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzl2w\" (UniqueName: \"kubernetes.io/projected/a115b104-1420-446c-aa9e-5d426677627f-kube-api-access-xzl2w\") pod \"maas-api-revoke-sub-5b5a3c08-mhqqf\" (UID: \"a115b104-1420-446c-aa9e-5d426677627f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5a3c08-mhqqf"
Nov 03 17:46:14.974850 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:14.974688    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a115b104-1420-446c-aa9e-5d426677627f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b5a3c08-mhqqf\" (UID: \"a115b104-1420-446c-aa9e-5d426677627f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5a3c08-mhqqf"
Nov 03 17:46:14.974943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:14.974924    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a115b104-1420-446c-aa9e-5d426677627f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b5a3c08-mhqqf\" (UID: \"a115b104-1420-446c-aa9e-5d426677627f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5a3c08-mhqqf"
Nov 03 17:46:14.994209 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:14.994177    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:46:14.994557 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:14.994533    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:46:14.998201 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:14.998179    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:46:14.998449 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:14.998429    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:46:15.076276 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:15.076242    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a115b104-1420-446c-aa9e-5d426677627f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b5a3c08-mhqqf\" (UID: \"a115b104-1420-446c-aa9e-5d426677627f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5a3c08-mhqqf"
Nov 03 17:46:15.076506 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:15.076300    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzl2w\" (UniqueName: \"kubernetes.io/projected/a115b104-1420-446c-aa9e-5d426677627f-kube-api-access-xzl2w\") pod \"maas-api-revoke-sub-5b5a3c08-mhqqf\" (UID: \"a115b104-1420-446c-aa9e-5d426677627f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5a3c08-mhqqf"
Nov 03 17:46:15.076506 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:15.076325    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a115b104-1420-446c-aa9e-5d426677627f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b5a3c08-mhqqf\" (UID: \"a115b104-1420-446c-aa9e-5d426677627f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5a3c08-mhqqf"
Nov 03 17:46:15.076974 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:15.076951    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a115b104-1420-446c-aa9e-5d426677627f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b5a3c08-mhqqf\" (UID: \"a115b104-1420-446c-aa9e-5d426677627f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5a3c08-mhqqf"
Nov 03 17:46:15.078688 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:15.078664    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a115b104-1420-446c-aa9e-5d426677627f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b5a3c08-mhqqf\" (UID: \"a115b104-1420-446c-aa9e-5d426677627f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5a3c08-mhqqf"
Nov 03 17:46:15.139399 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:15.139369    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzl2w\" (UniqueName: \"kubernetes.io/projected/a115b104-1420-446c-aa9e-5d426677627f-kube-api-access-xzl2w\") pod \"maas-api-revoke-sub-5b5a3c08-mhqqf\" (UID: \"a115b104-1420-446c-aa9e-5d426677627f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5a3c08-mhqqf"
Nov 03 17:46:15.207693 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:15.207660    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5a3c08-mhqqf"
Nov 03 17:46:15.582383 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:46:15.582321    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda115b104_1420_446c_aa9e_5d426677627f.slice/crio-832e645f19283565878ab2f8f80cfb7665aaf61185277608b39536845f06a7e2 WatchSource:0}: Error finding container 832e645f19283565878ab2f8f80cfb7665aaf61185277608b39536845f06a7e2: Status 404 returned error can't find the container with id 832e645f19283565878ab2f8f80cfb7665aaf61185277608b39536845f06a7e2
Nov 03 17:46:15.583904 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:15.583875    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b5a3c08-mhqqf"]
Nov 03 17:46:15.641831 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:15.641796    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:46:15.642225 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:15.642206    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:46:15.642821 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:15.642801    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:46:15.806707 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:15.806668    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5a3c08-mhqqf" event={"ID":"a115b104-1420-446c-aa9e-5d426677627f","Type":"ContainerStarted","Data":"832e645f19283565878ab2f8f80cfb7665aaf61185277608b39536845f06a7e2"}
Nov 03 17:46:15.963904 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:15.963866    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"]
Nov 03 17:46:15.969345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:15.969317    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.042069 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.042034    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"]
Nov 03 17:46:16.091074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.090994    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-data\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.091074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.091032    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-podinfo\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.091249 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.091162    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-credential-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.091249 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.091189    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-workload-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.091318 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.091302    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-workload-certs\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.091384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.091337    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a04c0ae8-7886-46c1-98c1-b2053479c608-istiod-ca-cert\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.091439 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.091425    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-envoy\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.091524 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.091512    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-token\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.091562 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.091538    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjjr8\" (UniqueName: \"kubernetes.io/projected/a04c0ae8-7886-46c1-98c1-b2053479c608-kube-api-access-pjjr8\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.192775 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.192735    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-token\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.192775 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.192779    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjjr8\" (UniqueName: \"kubernetes.io/projected/a04c0ae8-7886-46c1-98c1-b2053479c608-kube-api-access-pjjr8\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.192996 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.192834    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-data\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.192996 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.192853    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-podinfo\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.192996 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.192906    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-credential-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.192996 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.192929    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-workload-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.192996 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.192984    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-workload-certs\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.193261 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.193078    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a04c0ae8-7886-46c1-98c1-b2053479c608-istiod-ca-cert\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.193261 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.193206    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-envoy\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.193385 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.193310    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-data\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.193450 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.193394    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-workload-certs\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.193450 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.193436    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-credential-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.193557 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.193473    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-workload-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.193755 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.193735    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a04c0ae8-7886-46c1-98c1-b2053479c608-istiod-ca-cert\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.195420 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.195401    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-envoy\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.195521 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.195487    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-podinfo\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.256182 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.256140    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-token\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.256370 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.256203    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjjr8\" (UniqueName: \"kubernetes.io/projected/a04c0ae8-7886-46c1-98c1-b2053479c608-kube-api-access-pjjr8\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.284833 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.284806    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:16.459770 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.459735    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:16.711041 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:46:16.711012    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda04c0ae8_7886_46c1_98c1_b2053479c608.slice/crio-f2d7dda6e9ed8a1e0ae7777b0e820e94be88705b012709a2c79ba664b1d1273a WatchSource:0}: Error finding container f2d7dda6e9ed8a1e0ae7777b0e820e94be88705b012709a2c79ba664b1d1273a: Status 404 returned error can't find the container with id f2d7dda6e9ed8a1e0ae7777b0e820e94be88705b012709a2c79ba664b1d1273a
Nov 03 17:46:16.713719 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.713683    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"]
Nov 03 17:46:16.713977 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.713947    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:46:16.714047 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.714025    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:46:16.714112 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.714071    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:46:16.817860 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.817823    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp" event={"ID":"a04c0ae8-7886-46c1-98c1-b2053479c608","Type":"ContainerStarted","Data":"9e9e14cac618766d4e04ff853ddba7be78f2cf83cee586db43a4998baac4975e"}
Nov 03 17:46:16.818077 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.817892    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp" event={"ID":"a04c0ae8-7886-46c1-98c1-b2053479c608","Type":"ContainerStarted","Data":"f2d7dda6e9ed8a1e0ae7777b0e820e94be88705b012709a2c79ba664b1d1273a"}
Nov 03 17:46:16.819742 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.819718    6059 generic.go:356] "Generic (PLEG): container finished" podID="a115b104-1420-446c-aa9e-5d426677627f" containerID="d99de479f8eee9d7daf039118acb8f8042d8595755f009e50403174e4a5014ee" exitCode=0
Nov 03 17:46:16.819882 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.819765    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5a3c08-mhqqf" event={"ID":"a115b104-1420-446c-aa9e-5d426677627f","Type":"ContainerDied","Data":"d99de479f8eee9d7daf039118acb8f8042d8595755f009e50403174e4a5014ee"}
Nov 03 17:46:16.948850 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:16.948786    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp" podStartSLOduration=1.9487676299999999 podStartE2EDuration="1.94876763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:16.944546721 +0000 UTC m=+1717.742259503" watchObservedRunningTime="2026-11-03 17:46:16.94876763 +0000 UTC m=+1717.746480412"
Nov 03 17:46:17.285949 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:17.285861    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:17.287315 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:17.287289    6059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 17:46:17.287414 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:17.287339    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp" podUID="a04c0ae8-7886-46c1-98c1-b2053479c608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 17:46:18.062282 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:18.062256    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5a3c08-mhqqf"
Nov 03 17:46:18.223465 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:18.223431    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a115b104-1420-446c-aa9e-5d426677627f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a115b104-1420-446c-aa9e-5d426677627f-maas-api-xks-ca\") pod \"a115b104-1420-446c-aa9e-5d426677627f\" (UID: \"a115b104-1420-446c-aa9e-5d426677627f\") "
Nov 03 17:46:18.223625 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:18.223545    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a115b104-1420-446c-aa9e-5d426677627f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a115b104-1420-446c-aa9e-5d426677627f-maas-api-openshift-ca\") pod \"a115b104-1420-446c-aa9e-5d426677627f\" (UID: \"a115b104-1420-446c-aa9e-5d426677627f\") "
Nov 03 17:46:18.223625 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:18.223617    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a115b104-1420-446c-aa9e-5d426677627f-kube-api-access-xzl2w\" (UniqueName: \"kubernetes.io/projected/a115b104-1420-446c-aa9e-5d426677627f-kube-api-access-xzl2w\") pod \"a115b104-1420-446c-aa9e-5d426677627f\" (UID: \"a115b104-1420-446c-aa9e-5d426677627f\") "
Nov 03 17:46:18.223917 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:18.223892    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a115b104-1420-446c-aa9e-5d426677627f-maas-api-openshift-ca" pod "a115b104-1420-446c-aa9e-5d426677627f" (UID: "a115b104-1420-446c-aa9e-5d426677627f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:18.224182 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:18.224166    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a115b104-1420-446c-aa9e-5d426677627f-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:18.225712 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:18.225691    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a115b104-1420-446c-aa9e-5d426677627f-maas-api-xks-ca" pod "a115b104-1420-446c-aa9e-5d426677627f" (UID: "a115b104-1420-446c-aa9e-5d426677627f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:46:18.225800 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:18.225715    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a115b104-1420-446c-aa9e-5d426677627f-kube-api-access-xzl2w" pod "a115b104-1420-446c-aa9e-5d426677627f" (UID: "a115b104-1420-446c-aa9e-5d426677627f"). InnerVolumeSpecName "kube-api-access-xzl2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:46:18.285769 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:18.285737    6059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 17:46:18.285902 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:18.285787    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp" podUID="a04c0ae8-7886-46c1-98c1-b2053479c608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 17:46:18.325940 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:18.325910    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a115b104-1420-446c-aa9e-5d426677627f-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:18.325940 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:18.325940    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzl2w\" (UniqueName: \"kubernetes.io/projected/a115b104-1420-446c-aa9e-5d426677627f-kube-api-access-xzl2w\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:18.866561 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:18.866522    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5a3c08-mhqqf" event={"ID":"a115b104-1420-446c-aa9e-5d426677627f","Type":"ContainerDied","Data":"832e645f19283565878ab2f8f80cfb7665aaf61185277608b39536845f06a7e2"}
Nov 03 17:46:18.866561 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:18.866553    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5a3c08-mhqqf"
Nov 03 17:46:18.866774 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:18.866578    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="832e645f19283565878ab2f8f80cfb7665aaf61185277608b39536845f06a7e2"
Nov 03 17:46:19.010717 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:19.010686    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/models-as-a-service_payload-processing-external-model-86cbdddf86-2k5kw_ebefb030-567e-47f1-bb77-f62b222d5280/payload-processing/0.log"
Nov 03 17:46:19.285866 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:19.285834    6059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 17:46:19.286037 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:19.285881    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp" podUID="a04c0ae8-7886-46c1-98c1-b2053479c608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 17:46:19.865510 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:19.865474    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:20.285774 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:20.285736    6059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 17:46:20.285949 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:20.285794    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp" podUID="a04c0ae8-7886-46c1-98c1-b2053479c608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 17:46:21.084190 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:21.084152    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:21.285760 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:21.285722    6059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 17:46:21.285961 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:21.285810    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp" podUID="a04c0ae8-7886-46c1-98c1-b2053479c608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 17:46:22.285554 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:22.285518    6059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 17:46:22.285965 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:22.285578    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp" podUID="a04c0ae8-7886-46c1-98c1-b2053479c608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 17:46:23.285621 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:23.285587    6059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 17:46:23.286191 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:23.285639    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp" podUID="a04c0ae8-7886-46c1-98c1-b2053479c608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 17:46:24.270535 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.270500    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"]
Nov 03 17:46:24.271204 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.271170    6059 cpu_manager.go:414] "containerMap: removing container" podUID="a115b104-1420-446c-aa9e-5d426677627f" containerName="revoke-keys"
Nov 03 17:46:24.271204 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.271191    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a115b104-1420-446c-aa9e-5d426677627f" containerName="revoke-keys"
Nov 03 17:46:24.271347 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.271284    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a115b104-1420-446c-aa9e-5d426677627f" containerName="revoke-keys"
Nov 03 17:46:24.274757 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.274738    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"
Nov 03 17:46:24.289713 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.289684    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:24.290055 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.289974    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:24.290780 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.290756    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:46:24.327539 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.327505    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmx2x\" (UniqueName: \"kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-kube-api-access-tmx2x\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv\" (UID: \"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"
Nov 03 17:46:24.327705 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.327559    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv\" (UID: \"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"
Nov 03 17:46:24.327705 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.327618    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv\" (UID: \"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"
Nov 03 17:46:24.327808 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.327712    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv\" (UID: \"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"
Nov 03 17:46:24.382582 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.382542    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-ec4400c42669\""
Nov 03 17:46:24.382582 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.382569    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-bf3839ce3fb2\""
Nov 03 17:46:24.382582 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.382585    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-d27edff159f1\""
Nov 03 17:46:24.382885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.382549    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-198bcf954428\""
Nov 03 17:46:24.382885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.382639    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-dbae42e6384f\""
Nov 03 17:46:24.382885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.382645    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"payload-processing-external-model-e2e--b603cb9e-dockercfg-78csf\""
Nov 03 17:46:24.382885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.382643    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"openshift-service-ca.crt\""
Nov 03 17:46:24.382885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.382693    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"routing-overlay\""
Nov 03 17:46:24.382885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.382741    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-8c9b0f\""
Nov 03 17:46:24.382885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.382780    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"kube-root-ca.crt\""
Nov 03 17:46:24.386036 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.386016    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"]
Nov 03 17:46:24.429285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.429250    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmx2x\" (UniqueName: \"kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-kube-api-access-tmx2x\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv\" (UID: \"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"
Nov 03 17:46:24.429481 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.429298    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv\" (UID: \"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"
Nov 03 17:46:24.429481 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.429385    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv\" (UID: \"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"
Nov 03 17:46:24.429481 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.429428    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv\" (UID: \"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"
Nov 03 17:46:24.430095 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.430067    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv\" (UID: \"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"
Nov 03 17:46:24.430211 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.430169    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv\" (UID: \"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"
Nov 03 17:46:24.432239 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.432216    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv\" (UID: \"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"
Nov 03 17:46:24.492530 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.492504    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmx2x\" (UniqueName: \"kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-kube-api-access-tmx2x\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv\" (UID: \"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"
Nov 03 17:46:24.589019 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.588922    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"
Nov 03 17:46:24.971493 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:46:24.971448    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3713f17a_8bdf_4d6b_9a20_e8d0a3c752a1.slice/crio-e479930a583e136bce42b4e6922c5246c292c842013111b587e14724af7fed8e WatchSource:0}: Error finding container e479930a583e136bce42b4e6922c5246c292c842013111b587e14724af7fed8e: Status 404 returned error can't find the container with id e479930a583e136bce42b4e6922c5246c292c842013111b587e14724af7fed8e
Nov 03 17:46:24.973601 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:24.973574    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"]
Nov 03 17:46:25.716337 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:25.716269    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:25.920661 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:25.920622    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv" event={"ID":"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1","Type":"ContainerStarted","Data":"a6d0386c1f1ccc35088e8db4c04b16675f2192ef5fb4ee7f139b311f6e8e183a"}
Nov 03 17:46:25.920871 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:25.920714    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv" event={"ID":"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1","Type":"ContainerStarted","Data":"e479930a583e136bce42b4e6922c5246c292c842013111b587e14724af7fed8e"}
Nov 03 17:46:25.920871 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:25.920831    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"
Nov 03 17:46:26.045292 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:26.045182    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv" podStartSLOduration=2.045169951 podStartE2EDuration="2.045169951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:26.044002931 +0000 UTC m=+1726.841715712" watchObservedRunningTime="2026-11-03 17:46:26.045169951 +0000 UTC m=+1726.842882733"
Nov 03 17:46:26.445723 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:26.445649    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl"]
Nov 03 17:46:26.449959 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:26.449940    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl"
Nov 03 17:46:26.506572 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:26.506541    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:46:26.508346 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:26.508319    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl"]
Nov 03 17:46:26.555490 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:26.555456    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0b259b1-95ce-4d47-9aa3-9021be99e165-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cd65e5f-5hwsl\" (UID: \"d0b259b1-95ce-4d47-9aa3-9021be99e165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl"
Nov 03 17:46:26.555715 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:26.555692    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b74hj\" (UniqueName: \"kubernetes.io/projected/d0b259b1-95ce-4d47-9aa3-9021be99e165-kube-api-access-b74hj\") pod \"maas-api-revoke-sub-1cd65e5f-5hwsl\" (UID: \"d0b259b1-95ce-4d47-9aa3-9021be99e165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl"
Nov 03 17:46:26.555793 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:26.555751    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0b259b1-95ce-4d47-9aa3-9021be99e165-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cd65e5f-5hwsl\" (UID: \"d0b259b1-95ce-4d47-9aa3-9021be99e165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl"
Nov 03 17:46:26.556138 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:26.556118    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:46:26.556387 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:26.556344    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:46:26.556475 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:26.556424    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:46:26.657267 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:26.657231    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b74hj\" (UniqueName: \"kubernetes.io/projected/d0b259b1-95ce-4d47-9aa3-9021be99e165-kube-api-access-b74hj\") pod \"maas-api-revoke-sub-1cd65e5f-5hwsl\" (UID: \"d0b259b1-95ce-4d47-9aa3-9021be99e165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl"
Nov 03 17:46:26.657467 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:26.657292    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0b259b1-95ce-4d47-9aa3-9021be99e165-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cd65e5f-5hwsl\" (UID: \"d0b259b1-95ce-4d47-9aa3-9021be99e165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl"
Nov 03 17:46:26.657467 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:26.657383    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0b259b1-95ce-4d47-9aa3-9021be99e165-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cd65e5f-5hwsl\" (UID: \"d0b259b1-95ce-4d47-9aa3-9021be99e165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl"
Nov 03 17:46:26.657969 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:26.657946    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0b259b1-95ce-4d47-9aa3-9021be99e165-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cd65e5f-5hwsl\" (UID: \"d0b259b1-95ce-4d47-9aa3-9021be99e165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl"
Nov 03 17:46:26.659762 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:26.659745    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0b259b1-95ce-4d47-9aa3-9021be99e165-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cd65e5f-5hwsl\" (UID: \"d0b259b1-95ce-4d47-9aa3-9021be99e165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl"
Nov 03 17:46:26.720588 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:26.720564    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b74hj\" (UniqueName: \"kubernetes.io/projected/d0b259b1-95ce-4d47-9aa3-9021be99e165-kube-api-access-b74hj\") pod \"maas-api-revoke-sub-1cd65e5f-5hwsl\" (UID: \"d0b259b1-95ce-4d47-9aa3-9021be99e165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl"
Nov 03 17:46:26.764015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:26.763988    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl"
Nov 03 17:46:27.140602 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:46:27.140568    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0b259b1_95ce_4d47_9aa3_9021be99e165.slice/crio-809deae773044d6a369ba816c18de0dc841855778d4f317895f03472cd41e981 WatchSource:0}: Error finding container 809deae773044d6a369ba816c18de0dc841855778d4f317895f03472cd41e981: Status 404 returned error can't find the container with id 809deae773044d6a369ba816c18de0dc841855778d4f317895f03472cd41e981
Nov 03 17:46:27.143315 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:27.143292    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl"]
Nov 03 17:46:27.935641 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:27.935601    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl" event={"ID":"d0b259b1-95ce-4d47-9aa3-9021be99e165","Type":"ContainerStarted","Data":"114aee1c147b06f4d76cca25a44faaa26b21b4fdff86d958fa15a87074bd90d7"}
Nov 03 17:46:27.936076 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:27.935675    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl" event={"ID":"d0b259b1-95ce-4d47-9aa3-9021be99e165","Type":"ContainerStarted","Data":"809deae773044d6a369ba816c18de0dc841855778d4f317895f03472cd41e981"}
Nov 03 17:46:28.056205 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:28.056150    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl" podStartSLOduration=2.056133676 podStartE2EDuration="2.056133676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:28.054614734 +0000 UTC m=+1728.852327515" watchObservedRunningTime="2026-11-03 17:46:28.056133676 +0000 UTC m=+1728.853846461"
Nov 03 17:46:28.953247 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:28.953209    6059 generic.go:356] "Generic (PLEG): container finished" podID="d0b259b1-95ce-4d47-9aa3-9021be99e165" containerID="114aee1c147b06f4d76cca25a44faaa26b21b4fdff86d958fa15a87074bd90d7" exitCode=0
Nov 03 17:46:28.953687 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:28.953294    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl" event={"ID":"d0b259b1-95ce-4d47-9aa3-9021be99e165","Type":"ContainerDied","Data":"114aee1c147b06f4d76cca25a44faaa26b21b4fdff86d958fa15a87074bd90d7"}
Nov 03 17:46:30.128776 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:30.128749    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl"
Nov 03 17:46:30.201708 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:30.201680    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0b259b1-95ce-4d47-9aa3-9021be99e165-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0b259b1-95ce-4d47-9aa3-9021be99e165-maas-api-xks-ca\") pod \"d0b259b1-95ce-4d47-9aa3-9021be99e165\" (UID: \"d0b259b1-95ce-4d47-9aa3-9021be99e165\") "
Nov 03 17:46:30.201926 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:30.201825    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0b259b1-95ce-4d47-9aa3-9021be99e165-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0b259b1-95ce-4d47-9aa3-9021be99e165-maas-api-openshift-ca\") pod \"d0b259b1-95ce-4d47-9aa3-9021be99e165\" (UID: \"d0b259b1-95ce-4d47-9aa3-9021be99e165\") "
Nov 03 17:46:30.201926 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:30.201852    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0b259b1-95ce-4d47-9aa3-9021be99e165-kube-api-access-b74hj\" (UniqueName: \"kubernetes.io/projected/d0b259b1-95ce-4d47-9aa3-9021be99e165-kube-api-access-b74hj\") pod \"d0b259b1-95ce-4d47-9aa3-9021be99e165\" (UID: \"d0b259b1-95ce-4d47-9aa3-9021be99e165\") "
Nov 03 17:46:30.202193 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:30.202169    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0b259b1-95ce-4d47-9aa3-9021be99e165-maas-api-openshift-ca" pod "d0b259b1-95ce-4d47-9aa3-9021be99e165" (UID: "d0b259b1-95ce-4d47-9aa3-9021be99e165"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:30.202346 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:30.202331    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0b259b1-95ce-4d47-9aa3-9021be99e165-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:30.203792 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:30.203764    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0b259b1-95ce-4d47-9aa3-9021be99e165-maas-api-xks-ca" pod "d0b259b1-95ce-4d47-9aa3-9021be99e165" (UID: "d0b259b1-95ce-4d47-9aa3-9021be99e165"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:46:30.203908 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:30.203891    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b259b1-95ce-4d47-9aa3-9021be99e165-kube-api-access-b74hj" pod "d0b259b1-95ce-4d47-9aa3-9021be99e165" (UID: "d0b259b1-95ce-4d47-9aa3-9021be99e165"). InnerVolumeSpecName "kube-api-access-b74hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:46:30.236252 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:30.236212    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:30.303754 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:30.303720    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b74hj\" (UniqueName: \"kubernetes.io/projected/d0b259b1-95ce-4d47-9aa3-9021be99e165-kube-api-access-b74hj\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:30.303754 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:30.303752    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0b259b1-95ce-4d47-9aa3-9021be99e165-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:30.971284 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:30.971254    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl"
Nov 03 17:46:30.971284 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:30.971277    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl" event={"ID":"d0b259b1-95ce-4d47-9aa3-9021be99e165","Type":"ContainerDied","Data":"809deae773044d6a369ba816c18de0dc841855778d4f317895f03472cd41e981"}
Nov 03 17:46:30.971517 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:30.971325    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="809deae773044d6a369ba816c18de0dc841855778d4f317895f03472cd41e981"
Nov 03 17:46:34.926784 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:34.926735    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:36.861295 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:46:36.861256    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:46:36.930163 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:36.930136    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"
Nov 03 17:46:37.348540 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.348505    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf"]
Nov 03 17:46:37.349451 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.349432    6059 cpu_manager.go:414] "containerMap: removing container" podUID="d0b259b1-95ce-4d47-9aa3-9021be99e165" containerName="revoke-keys"
Nov 03 17:46:37.349510 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.349455    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0b259b1-95ce-4d47-9aa3-9021be99e165" containerName="revoke-keys"
Nov 03 17:46:37.349693 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.349679    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0b259b1-95ce-4d47-9aa3-9021be99e165" containerName="revoke-keys"
Nov 03 17:46:37.353924 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.353904    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf"
Nov 03 17:46:37.420074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.418423    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf"]
Nov 03 17:46:37.457834 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.457799    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:46:37.457988 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.457817    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:46:37.458063 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.458035    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:46:37.458165 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.458151    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:46:37.462950 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.462927    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r"]
Nov 03 17:46:37.467182 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.467166    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r"
Nov 03 17:46:37.512442 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.512408    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r"]
Nov 03 17:46:37.518173 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.518140    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085915c8-55e5-42e5-a258-bdf2f026f12c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3756d5e7-852lf\" (UID: \"085915c8-55e5-42e5-a258-bdf2f026f12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf"
Nov 03 17:46:37.518312 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.518202    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmqb4\" (UniqueName: \"kubernetes.io/projected/085915c8-55e5-42e5-a258-bdf2f026f12c-kube-api-access-tmqb4\") pod \"maas-api-revoke-keys-e2e-ait-3756d5e7-852lf\" (UID: \"085915c8-55e5-42e5-a258-bdf2f026f12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf"
Nov 03 17:46:37.518424 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.518394    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085915c8-55e5-42e5-a258-bdf2f026f12c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3756d5e7-852lf\" (UID: \"085915c8-55e5-42e5-a258-bdf2f026f12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf"
Nov 03 17:46:37.619918 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.619814    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dcf428c-j5h8r\" (UID: \"6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r"
Nov 03 17:46:37.619918 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.619865    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztc6g\" (UniqueName: \"kubernetes.io/projected/6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c-kube-api-access-ztc6g\") pod \"maas-api-revoke-sub-6dcf428c-j5h8r\" (UID: \"6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r"
Nov 03 17:46:37.620114 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.619924    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085915c8-55e5-42e5-a258-bdf2f026f12c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3756d5e7-852lf\" (UID: \"085915c8-55e5-42e5-a258-bdf2f026f12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf"
Nov 03 17:46:37.620114 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.619976    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmqb4\" (UniqueName: \"kubernetes.io/projected/085915c8-55e5-42e5-a258-bdf2f026f12c-kube-api-access-tmqb4\") pod \"maas-api-revoke-keys-e2e-ait-3756d5e7-852lf\" (UID: \"085915c8-55e5-42e5-a258-bdf2f026f12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf"
Nov 03 17:46:37.620114 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.620086    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085915c8-55e5-42e5-a258-bdf2f026f12c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3756d5e7-852lf\" (UID: \"085915c8-55e5-42e5-a258-bdf2f026f12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf"
Nov 03 17:46:37.620210 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.620183    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dcf428c-j5h8r\" (UID: \"6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r"
Nov 03 17:46:37.621548 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.621524    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085915c8-55e5-42e5-a258-bdf2f026f12c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3756d5e7-852lf\" (UID: \"085915c8-55e5-42e5-a258-bdf2f026f12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf"
Nov 03 17:46:37.622764 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.622739    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085915c8-55e5-42e5-a258-bdf2f026f12c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3756d5e7-852lf\" (UID: \"085915c8-55e5-42e5-a258-bdf2f026f12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf"
Nov 03 17:46:37.692004 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.691967    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmqb4\" (UniqueName: \"kubernetes.io/projected/085915c8-55e5-42e5-a258-bdf2f026f12c-kube-api-access-tmqb4\") pod \"maas-api-revoke-keys-e2e-ait-3756d5e7-852lf\" (UID: \"085915c8-55e5-42e5-a258-bdf2f026f12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf"
Nov 03 17:46:37.722039 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.722002    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dcf428c-j5h8r\" (UID: \"6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r"
Nov 03 17:46:37.722250 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.722096    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dcf428c-j5h8r\" (UID: \"6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r"
Nov 03 17:46:37.722326 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.722241    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztc6g\" (UniqueName: \"kubernetes.io/projected/6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c-kube-api-access-ztc6g\") pod \"maas-api-revoke-sub-6dcf428c-j5h8r\" (UID: \"6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r"
Nov 03 17:46:37.722790 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.722769    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dcf428c-j5h8r\" (UID: \"6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r"
Nov 03 17:46:37.724607 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.724583    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dcf428c-j5h8r\" (UID: \"6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r"
Nov 03 17:46:37.790822 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.790784    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztc6g\" (UniqueName: \"kubernetes.io/projected/6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c-kube-api-access-ztc6g\") pod \"maas-api-revoke-sub-6dcf428c-j5h8r\" (UID: \"6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r"
Nov 03 17:46:37.967980 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:37.967950    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf"
Nov 03 17:46:38.083744 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:38.083711    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r"
Nov 03 17:46:38.350877 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:38.350850    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf"]
Nov 03 17:46:38.458041 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:46:38.458010    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5e4bba_7d1f_4b3c_8746_a77c794b3a8c.slice/crio-2e0404d1c232b1274b422999c4f00c55833a3593610bebd3a3beba27a1875d04 WatchSource:0}: Error finding container 2e0404d1c232b1274b422999c4f00c55833a3593610bebd3a3beba27a1875d04: Status 404 returned error can't find the container with id 2e0404d1c232b1274b422999c4f00c55833a3593610bebd3a3beba27a1875d04
Nov 03 17:46:38.461481 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:38.461459    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r"]
Nov 03 17:46:39.041802 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:39.041770    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf" event={"ID":"085915c8-55e5-42e5-a258-bdf2f026f12c","Type":"ContainerStarted","Data":"06067fceb6824cf75a50d6bf40747845a477a2f486dc937b9e6fd3d7e7b35737"}
Nov 03 17:46:39.042262 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:39.041824    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf" event={"ID":"085915c8-55e5-42e5-a258-bdf2f026f12c","Type":"ContainerStarted","Data":"b013aaa08f86e8f4efc39fd0274c806d12e736d319c73deb7d2b85c07f526e2a"}
Nov 03 17:46:39.043728 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:39.043701    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r" event={"ID":"6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c","Type":"ContainerStarted","Data":"e5d375629f06f9182487179b5a75a2a2cb619447cc332e91f7e79c50e7b6d5a9"}
Nov 03 17:46:39.043854 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:39.043744    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r" event={"ID":"6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c","Type":"ContainerStarted","Data":"2e0404d1c232b1274b422999c4f00c55833a3593610bebd3a3beba27a1875d04"}
Nov 03 17:46:39.165996 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:39.165939    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf" podStartSLOduration=2.165925807 podStartE2EDuration="2.165925807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:39.163654993 +0000 UTC m=+1739.961367755" watchObservedRunningTime="2026-11-03 17:46:39.165925807 +0000 UTC m=+1739.963638587"
Nov 03 17:46:39.294561 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:39.294463    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r" podStartSLOduration=2.294449844 podStartE2EDuration="2.294449844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:39.290889235 +0000 UTC m=+1740.088602029" watchObservedRunningTime="2026-11-03 17:46:39.294449844 +0000 UTC m=+1740.092162645"
Nov 03 17:46:40.031894 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:40.031864    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:40.057728 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:40.057689    6059 generic.go:356] "Generic (PLEG): container finished" podID="6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c" containerID="e5d375629f06f9182487179b5a75a2a2cb619447cc332e91f7e79c50e7b6d5a9" exitCode=0
Nov 03 17:46:40.058252 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:40.057777    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r" event={"ID":"6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c","Type":"ContainerDied","Data":"e5d375629f06f9182487179b5a75a2a2cb619447cc332e91f7e79c50e7b6d5a9"}
Nov 03 17:46:40.059809 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:40.059782    6059 generic.go:356] "Generic (PLEG): container finished" podID="085915c8-55e5-42e5-a258-bdf2f026f12c" containerID="06067fceb6824cf75a50d6bf40747845a477a2f486dc937b9e6fd3d7e7b35737" exitCode=0
Nov 03 17:46:40.059941 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:40.059929    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf" event={"ID":"085915c8-55e5-42e5-a258-bdf2f026f12c","Type":"ContainerDied","Data":"06067fceb6824cf75a50d6bf40747845a477a2f486dc937b9e6fd3d7e7b35737"}
Nov 03 17:46:40.219762 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:40.219728    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:41.249608 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.249580    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf"
Nov 03 17:46:41.252776 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.252756    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r"
Nov 03 17:46:41.373668 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.373592    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/085915c8-55e5-42e5-a258-bdf2f026f12c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085915c8-55e5-42e5-a258-bdf2f026f12c-maas-api-openshift-ca\") pod \"085915c8-55e5-42e5-a258-bdf2f026f12c\" (UID: \"085915c8-55e5-42e5-a258-bdf2f026f12c\") "
Nov 03 17:46:41.373668 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.373657    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c-kube-api-access-ztc6g\" (UniqueName: \"kubernetes.io/projected/6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c-kube-api-access-ztc6g\") pod \"6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c\" (UID: \"6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c\") "
Nov 03 17:46:41.373886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.373695    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c-maas-api-openshift-ca\") pod \"6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c\" (UID: \"6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c\") "
Nov 03 17:46:41.373886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.373764    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/085915c8-55e5-42e5-a258-bdf2f026f12c-kube-api-access-tmqb4\" (UniqueName: \"kubernetes.io/projected/085915c8-55e5-42e5-a258-bdf2f026f12c-kube-api-access-tmqb4\") pod \"085915c8-55e5-42e5-a258-bdf2f026f12c\" (UID: \"085915c8-55e5-42e5-a258-bdf2f026f12c\") "
Nov 03 17:46:41.373886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.373878    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c-maas-api-xks-ca\") pod \"6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c\" (UID: \"6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c\") "
Nov 03 17:46:41.374059 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.373983    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/085915c8-55e5-42e5-a258-bdf2f026f12c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085915c8-55e5-42e5-a258-bdf2f026f12c-maas-api-xks-ca\") pod \"085915c8-55e5-42e5-a258-bdf2f026f12c\" (UID: \"085915c8-55e5-42e5-a258-bdf2f026f12c\") "
Nov 03 17:46:41.374059 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.374036    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/085915c8-55e5-42e5-a258-bdf2f026f12c-maas-api-openshift-ca" pod "085915c8-55e5-42e5-a258-bdf2f026f12c" (UID: "085915c8-55e5-42e5-a258-bdf2f026f12c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:41.374182 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.374142    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c-maas-api-openshift-ca" pod "6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c" (UID: "6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:41.374580 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.374554    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:41.374707 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.374582    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085915c8-55e5-42e5-a258-bdf2f026f12c-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:41.376157 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.376129    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/085915c8-55e5-42e5-a258-bdf2f026f12c-maas-api-xks-ca" pod "085915c8-55e5-42e5-a258-bdf2f026f12c" (UID: "085915c8-55e5-42e5-a258-bdf2f026f12c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:46:41.376350 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.376329    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085915c8-55e5-42e5-a258-bdf2f026f12c-kube-api-access-tmqb4" pod "085915c8-55e5-42e5-a258-bdf2f026f12c" (UID: "085915c8-55e5-42e5-a258-bdf2f026f12c"). InnerVolumeSpecName "kube-api-access-tmqb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:46:41.376551 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.376531    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c-maas-api-xks-ca" pod "6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c" (UID: "6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:46:41.376629 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.376543    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c-kube-api-access-ztc6g" pod "6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c" (UID: "6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c"). InnerVolumeSpecName "kube-api-access-ztc6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:46:41.476293 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.476258    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:41.476293 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.476289    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085915c8-55e5-42e5-a258-bdf2f026f12c-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:41.476293 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.476300    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztc6g\" (UniqueName: \"kubernetes.io/projected/6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c-kube-api-access-ztc6g\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:41.476545 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.476311    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmqb4\" (UniqueName: \"kubernetes.io/projected/085915c8-55e5-42e5-a258-bdf2f026f12c-kube-api-access-tmqb4\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:41.824291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.824255    6059 scope.go:122] "RemoveContainer" containerID="2ee4c8ab6ae4bee7718d4353c1fad1b36e11c0b17e3da7620c3493e233dcc9ae"
Nov 03 17:46:41.834957 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.834936    6059 scope.go:122] "RemoveContainer" containerID="7f47e37d15811db9605c40b0125b7124fa19c34835ac94e5fa545d5263793c5b"
Nov 03 17:46:41.844062 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.844041    6059 scope.go:122] "RemoveContainer" containerID="ec904366f6c9e5935da62d7ef27d75ebffc271c46727287f706909ca1bd1430b"
Nov 03 17:46:41.852810 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.852792    6059 scope.go:122] "RemoveContainer" containerID="f4386b6bf0c68ca3438c818d8c4f92da1e467754ff75f89a33cbfe8f3a42f631"
Nov 03 17:46:41.861645 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.861623    6059 scope.go:122] "RemoveContainer" containerID="4adc512d67bf6febbf79294018a2064dbb713f930f6c70a264fa7143b02faf4e"
Nov 03 17:46:41.869864 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:41.869845    6059 scope.go:122] "RemoveContainer" containerID="943b3c1e3c7184b62a47d27ce650dbee4bc1e04555fc90a67d8c03c4f88d09aa"
Nov 03 17:46:42.078079 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:42.077995    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r"
Nov 03 17:46:42.078079 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:42.078038    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r" event={"ID":"6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c","Type":"ContainerDied","Data":"2e0404d1c232b1274b422999c4f00c55833a3593610bebd3a3beba27a1875d04"}
Nov 03 17:46:42.078326 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:42.078097    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e0404d1c232b1274b422999c4f00c55833a3593610bebd3a3beba27a1875d04"
Nov 03 17:46:42.079543 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:42.079524    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf"
Nov 03 17:46:42.079655 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:42.079551    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf" event={"ID":"085915c8-55e5-42e5-a258-bdf2f026f12c","Type":"ContainerDied","Data":"b013aaa08f86e8f4efc39fd0274c806d12e736d319c73deb7d2b85c07f526e2a"}
Nov 03 17:46:42.079655 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:42.079585    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b013aaa08f86e8f4efc39fd0274c806d12e736d319c73deb7d2b85c07f526e2a"
Nov 03 17:46:44.815479 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:44.815447    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:48.826858 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:48.826820    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559"]
Nov 03 17:46:48.922177 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:48.922146    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894"]
Nov 03 17:46:49.209875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:49.209830    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:49.450816 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:49.450778    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:51.334882 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:51.334844    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:51.767922 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:51.767869    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:52.283489 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:52.283441    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt"]
Nov 03 17:46:52.393130 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:52.393094    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r"]
Nov 03 17:46:52.656453 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:52.656371    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:52.835776 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:52.835742    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:53.827590 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:53.827536    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559" podUID="1a60408a-78b9-40ba-a55f-2d62fae10b5c" containerName="payload-processing" containerID="cri-o://d7e687727c070daa698bb500e95f2a1aec28cf88cafdc5fd53c90a87b24be785" gracePeriod=25
Nov 03 17:46:54.119093 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.119068    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559"
Nov 03 17:46:54.127806 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.127762    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894" podUID="361ca698-4267-4a8e-8423-a1eb86738052" containerName="payload-pre-processing" containerID="cri-o://f2167887f971d4e2d29f715c89f0ee3f9797894d684d50018cf675724cebb986" gracePeriod=25
Nov 03 17:46:54.167165 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.167138    6059 generic.go:356] "Generic (PLEG): container finished" podID="1a60408a-78b9-40ba-a55f-2d62fae10b5c" containerID="d7e687727c070daa698bb500e95f2a1aec28cf88cafdc5fd53c90a87b24be785" exitCode=0
Nov 03 17:46:54.167302 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.167199    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559"
Nov 03 17:46:54.167302 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.167206    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559" event={"ID":"1a60408a-78b9-40ba-a55f-2d62fae10b5c","Type":"ContainerDied","Data":"d7e687727c070daa698bb500e95f2a1aec28cf88cafdc5fd53c90a87b24be785"}
Nov 03 17:46:54.167430 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.167298    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559" event={"ID":"1a60408a-78b9-40ba-a55f-2d62fae10b5c","Type":"ContainerDied","Data":"673e2c3be1a191d7ea8191b559d18b89f00a1218d282858a640b09f3e720542f"}
Nov 03 17:46:54.167430 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.167333    6059 scope.go:122] "RemoveContainer" containerID="d7e687727c070daa698bb500e95f2a1aec28cf88cafdc5fd53c90a87b24be785"
Nov 03 17:46:54.243018 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.242903    6059 scope.go:122] "RemoveContainer" containerID="d7e687727c070daa698bb500e95f2a1aec28cf88cafdc5fd53c90a87b24be785"
Nov 03 17:46:54.243292 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:46:54.243262    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e687727c070daa698bb500e95f2a1aec28cf88cafdc5fd53c90a87b24be785\": container with ID starting with d7e687727c070daa698bb500e95f2a1aec28cf88cafdc5fd53c90a87b24be785 not found: ID does not exist" containerID="d7e687727c070daa698bb500e95f2a1aec28cf88cafdc5fd53c90a87b24be785"
Nov 03 17:46:54.243440 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.243300    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e687727c070daa698bb500e95f2a1aec28cf88cafdc5fd53c90a87b24be785"} err="failed to get container status \"d7e687727c070daa698bb500e95f2a1aec28cf88cafdc5fd53c90a87b24be785\": rpc error: code = NotFound desc = could not find container \"d7e687727c070daa698bb500e95f2a1aec28cf88cafdc5fd53c90a87b24be785\": container with ID starting with d7e687727c070daa698bb500e95f2a1aec28cf88cafdc5fd53c90a87b24be785 not found: ID does not exist"
Nov 03 17:46:54.253550 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.253518    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a60408a-78b9-40ba-a55f-2d62fae10b5c-kube-api-access-lqstw\" (UniqueName: \"kubernetes.io/projected/1a60408a-78b9-40ba-a55f-2d62fae10b5c-kube-api-access-lqstw\") pod \"1a60408a-78b9-40ba-a55f-2d62fae10b5c\" (UID: \"1a60408a-78b9-40ba-a55f-2d62fae10b5c\") "
Nov 03 17:46:54.253684 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.253589    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a60408a-78b9-40ba-a55f-2d62fae10b5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a60408a-78b9-40ba-a55f-2d62fae10b5c-plugins-config-volume\") pod \"1a60408a-78b9-40ba-a55f-2d62fae10b5c\" (UID: \"1a60408a-78b9-40ba-a55f-2d62fae10b5c\") "
Nov 03 17:46:54.254035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.254006    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a60408a-78b9-40ba-a55f-2d62fae10b5c-plugins-config-volume" pod "1a60408a-78b9-40ba-a55f-2d62fae10b5c" (UID: "1a60408a-78b9-40ba-a55f-2d62fae10b5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:54.254199 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.254179    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a60408a-78b9-40ba-a55f-2d62fae10b5c-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:54.256176 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.256138    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a60408a-78b9-40ba-a55f-2d62fae10b5c-kube-api-access-lqstw" pod "1a60408a-78b9-40ba-a55f-2d62fae10b5c" (UID: "1a60408a-78b9-40ba-a55f-2d62fae10b5c"). InnerVolumeSpecName "kube-api-access-lqstw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:46:54.356017 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.355914    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqstw\" (UniqueName: \"kubernetes.io/projected/1a60408a-78b9-40ba-a55f-2d62fae10b5c-kube-api-access-lqstw\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:54.381103 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.381074    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894"
Nov 03 17:46:54.456597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.456547    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/361ca698-4267-4a8e-8423-a1eb86738052-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/361ca698-4267-4a8e-8423-a1eb86738052-plugins-config-volume\") pod \"361ca698-4267-4a8e-8423-a1eb86738052\" (UID: \"361ca698-4267-4a8e-8423-a1eb86738052\") "
Nov 03 17:46:54.456880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.456862    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/361ca698-4267-4a8e-8423-a1eb86738052-kube-api-access-cc7kx\" (UniqueName: \"kubernetes.io/projected/361ca698-4267-4a8e-8423-a1eb86738052-kube-api-access-cc7kx\") pod \"361ca698-4267-4a8e-8423-a1eb86738052\" (UID: \"361ca698-4267-4a8e-8423-a1eb86738052\") "
Nov 03 17:46:54.456976 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.456915    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361ca698-4267-4a8e-8423-a1eb86738052-plugins-config-volume" pod "361ca698-4267-4a8e-8423-a1eb86738052" (UID: "361ca698-4267-4a8e-8423-a1eb86738052"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:54.457481 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.457449    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/361ca698-4267-4a8e-8423-a1eb86738052-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:54.458997 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.458974    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361ca698-4267-4a8e-8423-a1eb86738052-kube-api-access-cc7kx" pod "361ca698-4267-4a8e-8423-a1eb86738052" (UID: "361ca698-4267-4a8e-8423-a1eb86738052"). InnerVolumeSpecName "kube-api-access-cc7kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:46:54.477009 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.476962    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:54.560459 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.560417    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc7kx\" (UniqueName: \"kubernetes.io/projected/361ca698-4267-4a8e-8423-a1eb86738052-kube-api-access-cc7kx\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:54.618735 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.618652    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559"]
Nov 03 17:46:54.625963 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:54.625934    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3756d5e7-6c6fc6c8bc-f6559"]
Nov 03 17:46:55.177788 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:55.177752    6059 generic.go:356] "Generic (PLEG): container finished" podID="361ca698-4267-4a8e-8423-a1eb86738052" containerID="f2167887f971d4e2d29f715c89f0ee3f9797894d684d50018cf675724cebb986" exitCode=0
Nov 03 17:46:55.178282 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:55.177794    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894"
Nov 03 17:46:55.178282 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:55.177794    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894" event={"ID":"361ca698-4267-4a8e-8423-a1eb86738052","Type":"ContainerDied","Data":"f2167887f971d4e2d29f715c89f0ee3f9797894d684d50018cf675724cebb986"}
Nov 03 17:46:55.178282 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:55.177897    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894" event={"ID":"361ca698-4267-4a8e-8423-a1eb86738052","Type":"ContainerDied","Data":"eae8926f2638dfcdac42ee12d1253b5bc361d5910d5f7df12167ea1b7b8b2117"}
Nov 03 17:46:55.178282 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:55.177938    6059 scope.go:122] "RemoveContainer" containerID="f2167887f971d4e2d29f715c89f0ee3f9797894d684d50018cf675724cebb986"
Nov 03 17:46:55.188752 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:55.188734    6059 scope.go:122] "RemoveContainer" containerID="f2167887f971d4e2d29f715c89f0ee3f9797894d684d50018cf675724cebb986"
Nov 03 17:46:55.189013 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:46:55.188996    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2167887f971d4e2d29f715c89f0ee3f9797894d684d50018cf675724cebb986\": container with ID starting with f2167887f971d4e2d29f715c89f0ee3f9797894d684d50018cf675724cebb986 not found: ID does not exist" containerID="f2167887f971d4e2d29f715c89f0ee3f9797894d684d50018cf675724cebb986"
Nov 03 17:46:55.189068 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:55.189020    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2167887f971d4e2d29f715c89f0ee3f9797894d684d50018cf675724cebb986"} err="failed to get container status \"f2167887f971d4e2d29f715c89f0ee3f9797894d684d50018cf675724cebb986\": rpc error: code = NotFound desc = could not find container \"f2167887f971d4e2d29f715c89f0ee3f9797894d684d50018cf675724cebb986\": container with ID starting with f2167887f971d4e2d29f715c89f0ee3f9797894d684d50018cf675724cebb986 not found: ID does not exist"
Nov 03 17:46:55.336466 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:55.336421    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a60408a-78b9-40ba-a55f-2d62fae10b5c" path="/var/lib/kubelet/pods/1a60408a-78b9-40ba-a55f-2d62fae10b5c/volumes"
Nov 03 17:46:55.349620 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:55.349590    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894"]
Nov 03 17:46:55.401524 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:55.401494    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3756d5e7-f8dbf8d5f-69894"]
Nov 03 17:46:57.283856 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:57.283812    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt" podUID="fe318530-ad39-46f8-ae0f-0b220a3c181b" containerName="payload-processing" containerID="cri-o://093fb40d3cbfc8b5824594aa4687b67a5853c2cb4b1af4f84d540b724f0c51e0" gracePeriod=25
Nov 03 17:46:57.337136 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:57.337109    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="361ca698-4267-4a8e-8423-a1eb86738052" path="/var/lib/kubelet/pods/361ca698-4267-4a8e-8423-a1eb86738052/volumes"
Nov 03 17:46:57.394034 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:57.393967    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r" podUID="8e72b7ae-f8ef-4274-8942-3044fb23a9d5" containerName="payload-pre-processing" containerID="cri-o://2f0ca3282a84a9769973a69465514ed65450dde379760d50b96d4924e4192d11" gracePeriod=25
Nov 03 17:46:57.552638 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:57.552573    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt"
Nov 03 17:46:57.682034 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:57.682007    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r"
Nov 03 17:46:57.704237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:57.704210    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe318530-ad39-46f8-ae0f-0b220a3c181b-kube-api-access-cg4tj\" (UniqueName: \"kubernetes.io/projected/fe318530-ad39-46f8-ae0f-0b220a3c181b-kube-api-access-cg4tj\") pod \"fe318530-ad39-46f8-ae0f-0b220a3c181b\" (UID: \"fe318530-ad39-46f8-ae0f-0b220a3c181b\") "
Nov 03 17:46:57.704393 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:57.704299    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe318530-ad39-46f8-ae0f-0b220a3c181b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe318530-ad39-46f8-ae0f-0b220a3c181b-plugins-config-volume\") pod \"fe318530-ad39-46f8-ae0f-0b220a3c181b\" (UID: \"fe318530-ad39-46f8-ae0f-0b220a3c181b\") "
Nov 03 17:46:57.704781 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:57.704747    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe318530-ad39-46f8-ae0f-0b220a3c181b-plugins-config-volume" pod "fe318530-ad39-46f8-ae0f-0b220a3c181b" (UID: "fe318530-ad39-46f8-ae0f-0b220a3c181b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:57.704954 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:57.704930    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe318530-ad39-46f8-ae0f-0b220a3c181b-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:57.706462 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:57.706434    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe318530-ad39-46f8-ae0f-0b220a3c181b-kube-api-access-cg4tj" pod "fe318530-ad39-46f8-ae0f-0b220a3c181b" (UID: "fe318530-ad39-46f8-ae0f-0b220a3c181b"). InnerVolumeSpecName "kube-api-access-cg4tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:46:57.730074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:57.730051    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:46:57.805997 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:57.805920    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e72b7ae-f8ef-4274-8942-3044fb23a9d5-kube-api-access-dnk7f\" (UniqueName: \"kubernetes.io/projected/8e72b7ae-f8ef-4274-8942-3044fb23a9d5-kube-api-access-dnk7f\") pod \"8e72b7ae-f8ef-4274-8942-3044fb23a9d5\" (UID: \"8e72b7ae-f8ef-4274-8942-3044fb23a9d5\") "
Nov 03 17:46:57.806173 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:57.806155    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e72b7ae-f8ef-4274-8942-3044fb23a9d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e72b7ae-f8ef-4274-8942-3044fb23a9d5-plugins-config-volume\") pod \"8e72b7ae-f8ef-4274-8942-3044fb23a9d5\" (UID: \"8e72b7ae-f8ef-4274-8942-3044fb23a9d5\") "
Nov 03 17:46:57.806709 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:57.806680    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg4tj\" (UniqueName: \"kubernetes.io/projected/fe318530-ad39-46f8-ae0f-0b220a3c181b-kube-api-access-cg4tj\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:57.806830 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:57.806710    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e72b7ae-f8ef-4274-8942-3044fb23a9d5-plugins-config-volume" pod "8e72b7ae-f8ef-4274-8942-3044fb23a9d5" (UID: "8e72b7ae-f8ef-4274-8942-3044fb23a9d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:57.808688 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:57.808661    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e72b7ae-f8ef-4274-8942-3044fb23a9d5-kube-api-access-dnk7f" pod "8e72b7ae-f8ef-4274-8942-3044fb23a9d5" (UID: "8e72b7ae-f8ef-4274-8942-3044fb23a9d5"). InnerVolumeSpecName "kube-api-access-dnk7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:46:57.907928 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:57.907893    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e72b7ae-f8ef-4274-8942-3044fb23a9d5-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:57.907928 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:57.907924    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnk7f\" (UniqueName: \"kubernetes.io/projected/8e72b7ae-f8ef-4274-8942-3044fb23a9d5-kube-api-access-dnk7f\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:58.201780 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:58.201693    6059 generic.go:356] "Generic (PLEG): container finished" podID="fe318530-ad39-46f8-ae0f-0b220a3c181b" containerID="093fb40d3cbfc8b5824594aa4687b67a5853c2cb4b1af4f84d540b724f0c51e0" exitCode=0
Nov 03 17:46:58.201780 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:58.201744    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt"
Nov 03 17:46:58.201979 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:58.201777    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt" event={"ID":"fe318530-ad39-46f8-ae0f-0b220a3c181b","Type":"ContainerDied","Data":"093fb40d3cbfc8b5824594aa4687b67a5853c2cb4b1af4f84d540b724f0c51e0"}
Nov 03 17:46:58.201979 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:58.201857    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt" event={"ID":"fe318530-ad39-46f8-ae0f-0b220a3c181b","Type":"ContainerDied","Data":"edbdb7d8f58524e91f9323f1317a84bb4982791623f9ca989d00db2472f397a9"}
Nov 03 17:46:58.201979 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:58.201883    6059 scope.go:122] "RemoveContainer" containerID="093fb40d3cbfc8b5824594aa4687b67a5853c2cb4b1af4f84d540b724f0c51e0"
Nov 03 17:46:58.203290 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:58.203266    6059 generic.go:356] "Generic (PLEG): container finished" podID="8e72b7ae-f8ef-4274-8942-3044fb23a9d5" containerID="2f0ca3282a84a9769973a69465514ed65450dde379760d50b96d4924e4192d11" exitCode=0
Nov 03 17:46:58.203391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:58.203323    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r"
Nov 03 17:46:58.203391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:58.203345    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r" event={"ID":"8e72b7ae-f8ef-4274-8942-3044fb23a9d5","Type":"ContainerDied","Data":"2f0ca3282a84a9769973a69465514ed65450dde379760d50b96d4924e4192d11"}
Nov 03 17:46:58.203511 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:58.203392    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r" event={"ID":"8e72b7ae-f8ef-4274-8942-3044fb23a9d5","Type":"ContainerDied","Data":"e1a8b91661e62fd44a8899ef467d4d0239acb2ecf8ba72b7f66bf42a6c668ddc"}
Nov 03 17:46:58.220085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:58.220067    6059 scope.go:122] "RemoveContainer" containerID="093fb40d3cbfc8b5824594aa4687b67a5853c2cb4b1af4f84d540b724f0c51e0"
Nov 03 17:46:58.220398 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:46:58.220376    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"093fb40d3cbfc8b5824594aa4687b67a5853c2cb4b1af4f84d540b724f0c51e0\": container with ID starting with 093fb40d3cbfc8b5824594aa4687b67a5853c2cb4b1af4f84d540b724f0c51e0 not found: ID does not exist" containerID="093fb40d3cbfc8b5824594aa4687b67a5853c2cb4b1af4f84d540b724f0c51e0"
Nov 03 17:46:58.220474 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:58.220404    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"093fb40d3cbfc8b5824594aa4687b67a5853c2cb4b1af4f84d540b724f0c51e0"} err="failed to get container status \"093fb40d3cbfc8b5824594aa4687b67a5853c2cb4b1af4f84d540b724f0c51e0\": rpc error: code = NotFound desc = could not find container \"093fb40d3cbfc8b5824594aa4687b67a5853c2cb4b1af4f84d540b724f0c51e0\": container with ID starting with 093fb40d3cbfc8b5824594aa4687b67a5853c2cb4b1af4f84d540b724f0c51e0 not found: ID does not exist"
Nov 03 17:46:58.220474 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:58.220423    6059 scope.go:122] "RemoveContainer" containerID="2f0ca3282a84a9769973a69465514ed65450dde379760d50b96d4924e4192d11"
Nov 03 17:46:58.229426 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:58.229408    6059 scope.go:122] "RemoveContainer" containerID="2f0ca3282a84a9769973a69465514ed65450dde379760d50b96d4924e4192d11"
Nov 03 17:46:58.229723 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:46:58.229701    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f0ca3282a84a9769973a69465514ed65450dde379760d50b96d4924e4192d11\": container with ID starting with 2f0ca3282a84a9769973a69465514ed65450dde379760d50b96d4924e4192d11 not found: ID does not exist" containerID="2f0ca3282a84a9769973a69465514ed65450dde379760d50b96d4924e4192d11"
Nov 03 17:46:58.229787 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:58.229730    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0ca3282a84a9769973a69465514ed65450dde379760d50b96d4924e4192d11"} err="failed to get container status \"2f0ca3282a84a9769973a69465514ed65450dde379760d50b96d4924e4192d11\": rpc error: code = NotFound desc = could not find container \"2f0ca3282a84a9769973a69465514ed65450dde379760d50b96d4924e4192d11\": container with ID starting with 2f0ca3282a84a9769973a69465514ed65450dde379760d50b96d4924e4192d11 not found: ID does not exist"
Nov 03 17:46:58.374391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:58.374327    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r"]
Nov 03 17:46:58.377784 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:58.377759    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-49c123d1-7496876dc-tc25r"]
Nov 03 17:46:58.490760 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:58.490727    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt"]
Nov 03 17:46:58.495736 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:58.495710    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-49c123d1-78fcdd7965-9xnnt"]
Nov 03 17:46:59.336051 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:59.336017    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e72b7ae-f8ef-4274-8942-3044fb23a9d5" path="/var/lib/kubelet/pods/8e72b7ae-f8ef-4274-8942-3044fb23a9d5/volumes"
Nov 03 17:46:59.336436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:59.336422    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe318530-ad39-46f8-ae0f-0b220a3c181b" path="/var/lib/kubelet/pods/fe318530-ad39-46f8-ae0f-0b220a3c181b/volumes"
Nov 03 17:46:59.497209 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:46:59.497176    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:04.232814 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:04.232776    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:06.861344 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:47:06.861307    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:47:08.828537 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:08.828481    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:12.725495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:12.725460    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:13.604202 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.604174    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"]
Nov 03 17:47:13.604766 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.604751    6059 cpu_manager.go:414] "containerMap: removing container" podUID="fe318530-ad39-46f8-ae0f-0b220a3c181b" containerName="payload-processing"
Nov 03 17:47:13.604843 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.604766    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe318530-ad39-46f8-ae0f-0b220a3c181b" containerName="payload-processing"
Nov 03 17:47:13.604843 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.604780    6059 cpu_manager.go:414] "containerMap: removing container" podUID="1a60408a-78b9-40ba-a55f-2d62fae10b5c" containerName="payload-processing"
Nov 03 17:47:13.604843 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.604785    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a60408a-78b9-40ba-a55f-2d62fae10b5c" containerName="payload-processing"
Nov 03 17:47:13.604843 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.604798    6059 cpu_manager.go:414] "containerMap: removing container" podUID="8e72b7ae-f8ef-4274-8942-3044fb23a9d5" containerName="payload-pre-processing"
Nov 03 17:47:13.604843 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.604803    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e72b7ae-f8ef-4274-8942-3044fb23a9d5" containerName="payload-pre-processing"
Nov 03 17:47:13.604843 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.604824    6059 cpu_manager.go:414] "containerMap: removing container" podUID="6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c" containerName="revoke-keys"
Nov 03 17:47:13.604843 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.604829    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c" containerName="revoke-keys"
Nov 03 17:47:13.604843 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.604841    6059 cpu_manager.go:414] "containerMap: removing container" podUID="085915c8-55e5-42e5-a258-bdf2f026f12c" containerName="revoke-keys"
Nov 03 17:47:13.605113 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.604884    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="085915c8-55e5-42e5-a258-bdf2f026f12c" containerName="revoke-keys"
Nov 03 17:47:13.605113 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.604899    6059 cpu_manager.go:414] "containerMap: removing container" podUID="361ca698-4267-4a8e-8423-a1eb86738052" containerName="payload-pre-processing"
Nov 03 17:47:13.605113 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.604903    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="361ca698-4267-4a8e-8423-a1eb86738052" containerName="payload-pre-processing"
Nov 03 17:47:13.605113 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.605012    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe318530-ad39-46f8-ae0f-0b220a3c181b" containerName="payload-processing"
Nov 03 17:47:13.605113 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.605024    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a60408a-78b9-40ba-a55f-2d62fae10b5c" containerName="payload-processing"
Nov 03 17:47:13.605113 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.605032    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e72b7ae-f8ef-4274-8942-3044fb23a9d5" containerName="payload-pre-processing"
Nov 03 17:47:13.605113 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.605042    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c" containerName="revoke-keys"
Nov 03 17:47:13.605113 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.605050    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="085915c8-55e5-42e5-a258-bdf2f026f12c" containerName="revoke-keys"
Nov 03 17:47:13.605113 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.605058    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="361ca698-4267-4a8e-8423-a1eb86738052" containerName="payload-pre-processing"
Nov 03 17:47:13.610303 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.610280    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.712747 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.712711    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"]
Nov 03 17:47:13.720270 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.720237    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-credential-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.720601 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.720574    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1e64fd8-6343-4408-9303-a5b712e16186-istiod-ca-cert\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.720791 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.720771    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e64fd8-6343-4408-9303-a5b712e16186-istio-token\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.721065 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.721043    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmbx6\" (UniqueName: \"kubernetes.io/projected/e1e64fd8-6343-4408-9303-a5b712e16186-kube-api-access-rmbx6\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.721233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.721212    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1e64fd8-6343-4408-9303-a5b712e16186-istio-podinfo\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.721393 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.721375    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-istio-data\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.721559 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.721539    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-workload-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.721715 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.721699    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-istio-envoy\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.721842 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.721826    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-workload-certs\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.822998 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.822960    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-istio-envoy\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.822998 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.823003    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-workload-certs\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.823597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.823057    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-credential-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.823597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.823108    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1e64fd8-6343-4408-9303-a5b712e16186-istiod-ca-cert\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.823597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.823130    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e64fd8-6343-4408-9303-a5b712e16186-istio-token\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.823597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.823186    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmbx6\" (UniqueName: \"kubernetes.io/projected/e1e64fd8-6343-4408-9303-a5b712e16186-kube-api-access-rmbx6\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.823597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.823221    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1e64fd8-6343-4408-9303-a5b712e16186-istio-podinfo\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.823597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.823258    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-istio-data\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.823597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.823303    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-workload-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.823597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.823534    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-credential-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.823597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.823561    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-workload-certs\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.824130 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.823749    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-workload-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.824130 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.823805    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-istio-data\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.824130 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.823935    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1e64fd8-6343-4408-9303-a5b712e16186-istiod-ca-cert\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.825500 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.825479    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-istio-envoy\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.825687 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.825670    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1e64fd8-6343-4408-9303-a5b712e16186-istio-podinfo\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.867325 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.867234    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:13.886716 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.886683    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e64fd8-6343-4408-9303-a5b712e16186-istio-token\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.886849 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.886757    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmbx6\" (UniqueName: \"kubernetes.io/projected/e1e64fd8-6343-4408-9303-a5b712e16186-kube-api-access-rmbx6\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:13.925336 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:13.925297    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:14.223338 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:14.223311    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:14.371851 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:47:14.371814    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e64fd8_6343_4408_9303_a5b712e16186.slice/crio-096ca15e1c422cb35cfd673b6fd51664964237c57b5350bfc80f40cdb3ff5369 WatchSource:0}: Error finding container 096ca15e1c422cb35cfd673b6fd51664964237c57b5350bfc80f40cdb3ff5369: Status 404 returned error can't find the container with id 096ca15e1c422cb35cfd673b6fd51664964237c57b5350bfc80f40cdb3ff5369
Nov 03 17:47:14.374411 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:14.374379    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"]
Nov 03 17:47:14.374510 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:14.374385    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:47:14.374554 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:14.374523    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:47:14.374590 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:14.374552    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:47:14.542028 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:14.541944    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:14.959039 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:14.959011    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf"]
Nov 03 17:47:14.976596 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:14.976561    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3756d5e7-852lf"]
Nov 03 17:47:15.361967 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:15.361881    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="085915c8-55e5-42e5-a258-bdf2f026f12c" path="/var/lib/kubelet/pods/085915c8-55e5-42e5-a258-bdf2f026f12c/volumes"
Nov 03 17:47:15.362237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:15.362218    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4" event={"ID":"e1e64fd8-6343-4408-9303-a5b712e16186","Type":"ContainerStarted","Data":"f74b261521d51e01a2eedaab6c0e2080e62ed80118dc944c8781ec46ffb79ae5"}
Nov 03 17:47:15.362299 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:15.362266    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4" event={"ID":"e1e64fd8-6343-4408-9303-a5b712e16186","Type":"ContainerStarted","Data":"096ca15e1c422cb35cfd673b6fd51664964237c57b5350bfc80f40cdb3ff5369"}
Nov 03 17:47:15.513773 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:15.513725    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4" podStartSLOduration=2.513710092 podStartE2EDuration="2.513710092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:15.472584199 +0000 UTC m=+1776.270296981" watchObservedRunningTime="2026-11-03 17:47:15.513710092 +0000 UTC m=+1776.311422873"
Nov 03 17:47:15.527600 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:15.527573    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"]
Nov 03 17:47:15.527813 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:15.527790    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x" podUID="17bc44f2-4185-4ce3-a127-cf0dc29b9583" containerName="istio-proxy" containerID="cri-o://30588c3d061767c44a9505418b9b67b1b0d9f5a282ff05ce4af8853a0759db21" gracePeriod=30
Nov 03 17:47:15.590499 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:15.590460    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:15.657292 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:15.657202    6059 patch_prober.go:29] interesting pod/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:47:15.657292 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:15.657265    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x" podUID="17bc44f2-4185-4ce3-a127-cf0dc29b9583" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:47:15.805410 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:15.804228    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:15.925867 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:15.925777    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:15.927548 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:15.927519    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Nov 03 17:47:15.927712 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:15.927581    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4" podUID="e1e64fd8-6343-4408-9303-a5b712e16186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Nov 03 17:47:16.241378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:16.236826    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:16.926478 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:16.926447    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Nov 03 17:47:16.926657 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:16.926501    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4" podUID="e1e64fd8-6343-4408-9303-a5b712e16186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Nov 03 17:47:17.709540 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:17.709500    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:17.906203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:17.906170    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:17.925579 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:17.925549    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Nov 03 17:47:17.925753 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:17.925597    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4" podUID="e1e64fd8-6343-4408-9303-a5b712e16186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Nov 03 17:47:18.925817 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:18.925785    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Nov 03 17:47:18.926227 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:18.925837    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4" podUID="e1e64fd8-6343-4408-9303-a5b712e16186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Nov 03 17:47:19.925929 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:19.925892    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Nov 03 17:47:19.926447 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:19.925946    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4" podUID="e1e64fd8-6343-4408-9303-a5b712e16186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Nov 03 17:47:20.722775 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.722741    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:47:20.757400 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.757082    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:20.865833 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.865739    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istiod-ca-cert\") pod \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") "
Nov 03 17:47:20.866016 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.865833    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-workload-certs\") pod \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") "
Nov 03 17:47:20.866016 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.865869    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-envoy\") pod \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") "
Nov 03 17:47:20.866016 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.865953    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-token\" (UniqueName: \"kubernetes.io/projected/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-token\") pod \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") "
Nov 03 17:47:20.866016 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.865999    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-workload-socket\") pod \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") "
Nov 03 17:47:20.866286 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.866043    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-podinfo\") pod \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") "
Nov 03 17:47:20.866286 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.866081    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-credential-socket\") pod \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") "
Nov 03 17:47:20.866286 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.866119    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-workload-certs" pod "17bc44f2-4185-4ce3-a127-cf0dc29b9583" (UID: "17bc44f2-4185-4ce3-a127-cf0dc29b9583"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:47:20.866286 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.866136    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istiod-ca-cert" pod "17bc44f2-4185-4ce3-a127-cf0dc29b9583" (UID: "17bc44f2-4185-4ce3-a127-cf0dc29b9583"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:47:20.866286 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.866143    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-data\") pod \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") "
Nov 03 17:47:20.866286 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.866230    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17bc44f2-4185-4ce3-a127-cf0dc29b9583-kube-api-access-jq6lq\" (UniqueName: \"kubernetes.io/projected/17bc44f2-4185-4ce3-a127-cf0dc29b9583-kube-api-access-jq6lq\") pod \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\" (UID: \"17bc44f2-4185-4ce3-a127-cf0dc29b9583\") "
Nov 03 17:47:20.866662 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.866304    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-credential-socket" pod "17bc44f2-4185-4ce3-a127-cf0dc29b9583" (UID: "17bc44f2-4185-4ce3-a127-cf0dc29b9583"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:47:20.866662 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.866316    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-workload-socket" pod "17bc44f2-4185-4ce3-a127-cf0dc29b9583" (UID: "17bc44f2-4185-4ce3-a127-cf0dc29b9583"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:47:20.866662 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.866342    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-data" pod "17bc44f2-4185-4ce3-a127-cf0dc29b9583" (UID: "17bc44f2-4185-4ce3-a127-cf0dc29b9583"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:47:20.867165 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.866876    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:20.867165 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.866905    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:20.867165 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.866921    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:20.867165 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.866936    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:20.867165 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.866949    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:20.868313 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.868290    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-token" pod "17bc44f2-4185-4ce3-a127-cf0dc29b9583" (UID: "17bc44f2-4185-4ce3-a127-cf0dc29b9583"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:47:20.868716 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.868695    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-envoy" pod "17bc44f2-4185-4ce3-a127-cf0dc29b9583" (UID: "17bc44f2-4185-4ce3-a127-cf0dc29b9583"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:47:20.868832 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.868714    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17bc44f2-4185-4ce3-a127-cf0dc29b9583-kube-api-access-jq6lq" pod "17bc44f2-4185-4ce3-a127-cf0dc29b9583" (UID: "17bc44f2-4185-4ce3-a127-cf0dc29b9583"). InnerVolumeSpecName "kube-api-access-jq6lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:47:20.868928 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.868905    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-podinfo" pod "17bc44f2-4185-4ce3-a127-cf0dc29b9583" (UID: "17bc44f2-4185-4ce3-a127-cf0dc29b9583"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:47:20.925728 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.925676    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Nov 03 17:47:20.925900 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.925759    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4" podUID="e1e64fd8-6343-4408-9303-a5b712e16186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Nov 03 17:47:20.969128 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.969089    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq6lq\" (UniqueName: \"kubernetes.io/projected/17bc44f2-4185-4ce3-a127-cf0dc29b9583-kube-api-access-jq6lq\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:20.969128 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.969132    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:20.969569 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.969149    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:20.969569 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:20.969164    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17bc44f2-4185-4ce3-a127-cf0dc29b9583-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:21.398123 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.398084    6059 generic.go:356] "Generic (PLEG): container finished" podID="17bc44f2-4185-4ce3-a127-cf0dc29b9583" containerID="30588c3d061767c44a9505418b9b67b1b0d9f5a282ff05ce4af8853a0759db21" exitCode=0
Nov 03 17:47:21.398319 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.398162    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"
Nov 03 17:47:21.398319 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.398176    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x" event={"ID":"17bc44f2-4185-4ce3-a127-cf0dc29b9583","Type":"ContainerDied","Data":"30588c3d061767c44a9505418b9b67b1b0d9f5a282ff05ce4af8853a0759db21"}
Nov 03 17:47:21.398319 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.398250    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x" event={"ID":"17bc44f2-4185-4ce3-a127-cf0dc29b9583","Type":"ContainerDied","Data":"8cfb5c457a05a5b22baa703b8dfd835fb7df23460c4600a3e0a088d6c8d08861"}
Nov 03 17:47:21.398319 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.398282    6059 scope.go:122] "RemoveContainer" containerID="30588c3d061767c44a9505418b9b67b1b0d9f5a282ff05ce4af8853a0759db21"
Nov 03 17:47:21.402997 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.402970    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"]
Nov 03 17:47:21.403948 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.403925    6059 cpu_manager.go:414] "containerMap: removing container" podUID="17bc44f2-4185-4ce3-a127-cf0dc29b9583" containerName="istio-proxy"
Nov 03 17:47:21.403948 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.403947    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17bc44f2-4185-4ce3-a127-cf0dc29b9583" containerName="istio-proxy"
Nov 03 17:47:21.404136 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.404111    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="17bc44f2-4185-4ce3-a127-cf0dc29b9583" containerName="istio-proxy"
Nov 03 17:47:21.404136 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.404136    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="17bc44f2-4185-4ce3-a127-cf0dc29b9583" containerName="istio-proxy"
Nov 03 17:47:21.408983 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.408959    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.413001 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.412984    6059 scope.go:122] "RemoveContainer" containerID="81eff07997580368b18f8080d233d0f6c3a34929a413476ebf4e935c4ec4d076"
Nov 03 17:47:21.423481 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.423464    6059 scope.go:122] "RemoveContainer" containerID="30588c3d061767c44a9505418b9b67b1b0d9f5a282ff05ce4af8853a0759db21"
Nov 03 17:47:21.423757 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:47:21.423732    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30588c3d061767c44a9505418b9b67b1b0d9f5a282ff05ce4af8853a0759db21\": container with ID starting with 30588c3d061767c44a9505418b9b67b1b0d9f5a282ff05ce4af8853a0759db21 not found: ID does not exist" containerID="30588c3d061767c44a9505418b9b67b1b0d9f5a282ff05ce4af8853a0759db21"
Nov 03 17:47:21.423859 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.423764    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30588c3d061767c44a9505418b9b67b1b0d9f5a282ff05ce4af8853a0759db21"} err="failed to get container status \"30588c3d061767c44a9505418b9b67b1b0d9f5a282ff05ce4af8853a0759db21\": rpc error: code = NotFound desc = could not find container \"30588c3d061767c44a9505418b9b67b1b0d9f5a282ff05ce4af8853a0759db21\": container with ID starting with 30588c3d061767c44a9505418b9b67b1b0d9f5a282ff05ce4af8853a0759db21 not found: ID does not exist"
Nov 03 17:47:21.423859 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.423780    6059 scope.go:122] "RemoveContainer" containerID="81eff07997580368b18f8080d233d0f6c3a34929a413476ebf4e935c4ec4d076"
Nov 03 17:47:21.424005 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:47:21.423989    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81eff07997580368b18f8080d233d0f6c3a34929a413476ebf4e935c4ec4d076\": container with ID starting with 81eff07997580368b18f8080d233d0f6c3a34929a413476ebf4e935c4ec4d076 not found: ID does not exist" containerID="81eff07997580368b18f8080d233d0f6c3a34929a413476ebf4e935c4ec4d076"
Nov 03 17:47:21.424044 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.424010    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81eff07997580368b18f8080d233d0f6c3a34929a413476ebf4e935c4ec4d076"} err="failed to get container status \"81eff07997580368b18f8080d233d0f6c3a34929a413476ebf4e935c4ec4d076\": rpc error: code = NotFound desc = could not find container \"81eff07997580368b18f8080d233d0f6c3a34929a413476ebf4e935c4ec4d076\": container with ID starting with 81eff07997580368b18f8080d233d0f6c3a34929a413476ebf4e935c4ec4d076 not found: ID does not exist"
Nov 03 17:47:21.511045 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.511014    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"]
Nov 03 17:47:21.576869 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.576838    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029774c2-7278-4d5b-ba88-055cedf047eb-istiod-ca-cert\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.577060 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.576978    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029774c2-7278-4d5b-ba88-055cedf047eb-istio-podinfo\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.577060 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.577043    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-workload-certs\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.577173 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.577121    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-workload-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.577235 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.577199    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-credential-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.577295 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.577253    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029774c2-7278-4d5b-ba88-055cedf047eb-istio-token\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.577349 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.577295    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k257\" (UniqueName: \"kubernetes.io/projected/029774c2-7278-4d5b-ba88-055cedf047eb-kube-api-access-8k257\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.577519 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.577494    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-istio-data\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.577577 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.577545    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"]
Nov 03 17:47:21.577670 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.577654    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-istio-envoy\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.581787 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.581764    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3756d5e7-openshift-default-5458c5dfd9-whr2x"]
Nov 03 17:47:21.679143 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.679058    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-istio-data\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.679299 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.679152    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-istio-envoy\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.679299 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.679171    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029774c2-7278-4d5b-ba88-055cedf047eb-istiod-ca-cert\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.679466 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.679328    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029774c2-7278-4d5b-ba88-055cedf047eb-istio-podinfo\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.679466 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.679441    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-workload-certs\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.679577 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.679484    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-workload-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.679577 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.679486    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-istio-data\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.679577 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.679513    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-credential-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.679577 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.679547    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029774c2-7278-4d5b-ba88-055cedf047eb-istio-token\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.679783 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.679591    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k257\" (UniqueName: \"kubernetes.io/projected/029774c2-7278-4d5b-ba88-055cedf047eb-kube-api-access-8k257\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.679837 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.679790    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-workload-certs\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.679837 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.679819    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-workload-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.679941 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.679857    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-credential-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.679941 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.679898    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029774c2-7278-4d5b-ba88-055cedf047eb-istiod-ca-cert\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.682114 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.682085    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-istio-envoy\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.682218 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.682192    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029774c2-7278-4d5b-ba88-055cedf047eb-istio-podinfo\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.745939 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.745905    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029774c2-7278-4d5b-ba88-055cedf047eb-istio-token\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.746533 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.746504    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k257\" (UniqueName: \"kubernetes.io/projected/029774c2-7278-4d5b-ba88-055cedf047eb-kube-api-access-8k257\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:21.930571 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.930500    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:21.930782 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.930761    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:21.931332 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:21.931312    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:47:22.025709 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:22.025674    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:22.436757 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:22.436696    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:22.461089 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:47:22.461059    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029774c2_7278_4d5b_ba88_055cedf047eb.slice/crio-537d3bb93feb470c1b5ce542ddca7cd77ac1ca2bd25e1f7317fbaa2c590928b1 WatchSource:0}: Error finding container 537d3bb93feb470c1b5ce542ddca7cd77ac1ca2bd25e1f7317fbaa2c590928b1: Status 404 returned error can't find the container with id 537d3bb93feb470c1b5ce542ddca7cd77ac1ca2bd25e1f7317fbaa2c590928b1
Nov 03 17:47:22.463960 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:22.463933    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"]
Nov 03 17:47:22.464062 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:22.463966    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:47:22.464062 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:22.464039    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:47:22.464140 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:22.464084    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:47:22.907712 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:22.907669    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:23.094611 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:23.094578    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:23.361887 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:23.361858    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17bc44f2-4185-4ce3-a127-cf0dc29b9583" path="/var/lib/kubelet/pods/17bc44f2-4185-4ce3-a127-cf0dc29b9583/volumes"
Nov 03 17:47:23.415575 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:23.415538    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w" event={"ID":"029774c2-7278-4d5b-ba88-055cedf047eb","Type":"ContainerStarted","Data":"7bad9c5394a4255e0fe64928f63376cbadadc017764609e2f30f1e6a26ad4294"}
Nov 03 17:47:23.416079 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:23.415601    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w" event={"ID":"029774c2-7278-4d5b-ba88-055cedf047eb","Type":"ContainerStarted","Data":"537d3bb93feb470c1b5ce542ddca7cd77ac1ca2bd25e1f7317fbaa2c590928b1"}
Nov 03 17:47:23.557483 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:23.557427    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w" podStartSLOduration=2.557413178 podStartE2EDuration="2.557413178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:23.554875872 +0000 UTC m=+1784.352588652" watchObservedRunningTime="2026-11-03 17:47:23.557413178 +0000 UTC m=+1784.355125960"
Nov 03 17:47:24.026745 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.026711    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:24.028228 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.028195    6059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 03 17:47:24.028390 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.028247    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w" podUID="029774c2-7278-4d5b-ba88-055cedf047eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 03 17:47:24.207117 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.207083    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:24.386663 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.386578    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:24.573152 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.573116    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:24.606744 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.606713    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8"]
Nov 03 17:47:24.607307 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.607292    6059 cpu_manager.go:414] "containerMap: removing container" podUID="17bc44f2-4185-4ce3-a127-cf0dc29b9583" containerName="istio-proxy"
Nov 03 17:47:24.607348 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.607308    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17bc44f2-4185-4ce3-a127-cf0dc29b9583" containerName="istio-proxy"
Nov 03 17:47:24.611285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.611265    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8"
Nov 03 17:47:24.665903 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.662829    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l"]
Nov 03 17:47:24.670239 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.668073    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l"
Nov 03 17:47:24.673879 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.673828    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8"]
Nov 03 17:47:24.689679 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.687214    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsz2j\" (UniqueName: \"kubernetes.io/projected/50c1cc3e-ffc9-4b51-9a95-5677f4898c02-kube-api-access-jsz2j\") pod \"payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8\" (UID: \"50c1cc3e-ffc9-4b51-9a95-5677f4898c02\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8"
Nov 03 17:47:24.689679 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.687475    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c1cc3e-ffc9-4b51-9a95-5677f4898c02-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8\" (UID: \"50c1cc3e-ffc9-4b51-9a95-5677f4898c02\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8"
Nov 03 17:47:24.713905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.713186    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-1b3c7bad\""
Nov 03 17:47:24.713905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.713559    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-1b3c7bad-dockercfg-gx6sm\""
Nov 03 17:47:24.767982 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.767947    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l"]
Nov 03 17:47:24.797381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.797230    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsz2j\" (UniqueName: \"kubernetes.io/projected/50c1cc3e-ffc9-4b51-9a95-5677f4898c02-kube-api-access-jsz2j\") pod \"payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8\" (UID: \"50c1cc3e-ffc9-4b51-9a95-5677f4898c02\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8"
Nov 03 17:47:24.797603 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.797384    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx6rk\" (UniqueName: \"kubernetes.io/projected/77951355-6d25-4504-9a27-db09e1d94236-kube-api-access-fx6rk\") pod \"payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l\" (UID: \"77951355-6d25-4504-9a27-db09e1d94236\") " pod="openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l"
Nov 03 17:47:24.797603 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.797440    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77951355-6d25-4504-9a27-db09e1d94236-plugins-config-volume\") pod \"payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l\" (UID: \"77951355-6d25-4504-9a27-db09e1d94236\") " pod="openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l"
Nov 03 17:47:24.797755 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.797645    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c1cc3e-ffc9-4b51-9a95-5677f4898c02-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8\" (UID: \"50c1cc3e-ffc9-4b51-9a95-5677f4898c02\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8"
Nov 03 17:47:24.804417 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.798857    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c1cc3e-ffc9-4b51-9a95-5677f4898c02-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8\" (UID: \"50c1cc3e-ffc9-4b51-9a95-5677f4898c02\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8"
Nov 03 17:47:24.869399 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.869352    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsz2j\" (UniqueName: \"kubernetes.io/projected/50c1cc3e-ffc9-4b51-9a95-5677f4898c02-kube-api-access-jsz2j\") pod \"payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8\" (UID: \"50c1cc3e-ffc9-4b51-9a95-5677f4898c02\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8"
Nov 03 17:47:24.899189 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.899156    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx6rk\" (UniqueName: \"kubernetes.io/projected/77951355-6d25-4504-9a27-db09e1d94236-kube-api-access-fx6rk\") pod \"payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l\" (UID: \"77951355-6d25-4504-9a27-db09e1d94236\") " pod="openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l"
Nov 03 17:47:24.899346 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.899198    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77951355-6d25-4504-9a27-db09e1d94236-plugins-config-volume\") pod \"payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l\" (UID: \"77951355-6d25-4504-9a27-db09e1d94236\") " pod="openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l"
Nov 03 17:47:24.899943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.899921    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77951355-6d25-4504-9a27-db09e1d94236-plugins-config-volume\") pod \"payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l\" (UID: \"77951355-6d25-4504-9a27-db09e1d94236\") " pod="openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l"
Nov 03 17:47:24.925886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.925816    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8"
Nov 03 17:47:24.978342 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.978306    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"]
Nov 03 17:47:24.978588 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.978561    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7" podUID="7d3b16ed-f9c0-4c11-b7de-4f472c859170" containerName="istio-proxy" containerID="cri-o://044182d8a0b4d486379b91efbcce4bc4cbfa3077d05e45804fd86c7c9cbf4f37" gracePeriod=30
Nov 03 17:47:24.980835 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:24.980804    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx6rk\" (UniqueName: \"kubernetes.io/projected/77951355-6d25-4504-9a27-db09e1d94236-kube-api-access-fx6rk\") pod \"payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l\" (UID: \"77951355-6d25-4504-9a27-db09e1d94236\") " pod="openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l"
Nov 03 17:47:25.017778 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:25.017746    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l"
Nov 03 17:47:25.026511 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:25.026486    6059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 03 17:47:25.026644 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:25.026529    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w" podUID="029774c2-7278-4d5b-ba88-055cedf047eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 03 17:47:25.066152 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:25.065867    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:25.381144 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:47:25.381106    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c1cc3e_ffc9_4b51_9a95_5677f4898c02.slice/crio-1cfeeb9551cba7cce16c6378adebfcf8ad0dfd949ffc065f0ba6cfec07d1f3a1 WatchSource:0}: Error finding container 1cfeeb9551cba7cce16c6378adebfcf8ad0dfd949ffc065f0ba6cfec07d1f3a1: Status 404 returned error can't find the container with id 1cfeeb9551cba7cce16c6378adebfcf8ad0dfd949ffc065f0ba6cfec07d1f3a1
Nov 03 17:47:25.383465 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:25.383439    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8"]
Nov 03 17:47:25.459071 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:25.459038    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8" event={"ID":"50c1cc3e-ffc9-4b51-9a95-5677f4898c02","Type":"ContainerStarted","Data":"1cfeeb9551cba7cce16c6378adebfcf8ad0dfd949ffc065f0ba6cfec07d1f3a1"}
Nov 03 17:47:25.657710 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:25.657272    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l"]
Nov 03 17:47:25.985091 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:25.985059    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:26.026182 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:26.026148    6059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 03 17:47:26.026394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:26.026199    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w" podUID="029774c2-7278-4d5b-ba88-055cedf047eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 03 17:47:26.470210 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:26.470173    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l" event={"ID":"77951355-6d25-4504-9a27-db09e1d94236","Type":"ContainerStarted","Data":"d161997d4c5d3f12ccde9dd66617e969343c1ce0480b90187b5b5f3a561ef0e8"}
Nov 03 17:47:26.470724 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:26.470245    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l" event={"ID":"77951355-6d25-4504-9a27-db09e1d94236","Type":"ContainerStarted","Data":"25cdf610eed11b95faabb4fbeda5e285e81b7c31693409451d5bcbb35de1f71a"}
Nov 03 17:47:26.470724 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:26.470277    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l"
Nov 03 17:47:26.472101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:26.472046    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8" event={"ID":"50c1cc3e-ffc9-4b51-9a95-5677f4898c02","Type":"ContainerStarted","Data":"ef1a4b620bb8dbebbbf7f2b275aafea2fad5c5118a097d4c0554d631a389c5ab"}
Nov 03 17:47:26.472176 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:26.472135    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8"
Nov 03 17:47:26.607855 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:26.607802    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l" podStartSLOduration=2.607789243 podStartE2EDuration="2.607789243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:26.603403467 +0000 UTC m=+1787.401116249" watchObservedRunningTime="2026-11-03 17:47:26.607789243 +0000 UTC m=+1787.405502024"
Nov 03 17:47:26.754440 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:26.753948    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8" podStartSLOduration=2.75392776 podStartE2EDuration="2.75392776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:26.751891735 +0000 UTC m=+1787.549604521" watchObservedRunningTime="2026-11-03 17:47:26.75392776 +0000 UTC m=+1787.551640542"
Nov 03 17:47:27.026101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:27.026018    6059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 03 17:47:27.026101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:27.026081    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w" podUID="029774c2-7278-4d5b-ba88-055cedf047eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 03 17:47:28.026652 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:28.026613    6059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 03 17:47:28.027060 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:28.026673    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w" podUID="029774c2-7278-4d5b-ba88-055cedf047eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 03 17:47:28.550093 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:28.550049    6059 patch_prober.go:29] interesting pod/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:47:28.550260 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:28.550141    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7" podUID="7d3b16ed-f9c0-4c11-b7de-4f472c859170" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:47:29.155843 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:29.155803    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:29.156563 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:29.156529    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:29.158711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:29.158678    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:47:30.239058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.239031    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:47:30.343295 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.343206    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-data\") pod \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") "
Nov 03 17:47:30.343295 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.343277    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-envoy\") pod \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") "
Nov 03 17:47:30.343545 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.343528    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-podinfo\") pod \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") "
Nov 03 17:47:30.343614 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.343552    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-data" pod "7d3b16ed-f9c0-4c11-b7de-4f472c859170" (UID: "7d3b16ed-f9c0-4c11-b7de-4f472c859170"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:47:30.343614 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.343582    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-workload-socket\") pod \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") "
Nov 03 17:47:30.343727 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.343617    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-credential-socket\") pod \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") "
Nov 03 17:47:30.343727 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.343709    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-workload-certs\") pod \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") "
Nov 03 17:47:30.343814 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.343768    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d3b16ed-f9c0-4c11-b7de-4f472c859170-kube-api-access-7h4v7\" (UniqueName: \"kubernetes.io/projected/7d3b16ed-f9c0-4c11-b7de-4f472c859170-kube-api-access-7h4v7\") pod \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") "
Nov 03 17:47:30.343881 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.343845    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-token\" (UniqueName: \"kubernetes.io/projected/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-token\") pod \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") "
Nov 03 17:47:30.343941 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.343916    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istiod-ca-cert\") pod \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\" (UID: \"7d3b16ed-f9c0-4c11-b7de-4f472c859170\") "
Nov 03 17:47:30.343941 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.343919    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-credential-socket" pod "7d3b16ed-f9c0-4c11-b7de-4f472c859170" (UID: "7d3b16ed-f9c0-4c11-b7de-4f472c859170"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:47:30.344047 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.344019    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-workload-certs" pod "7d3b16ed-f9c0-4c11-b7de-4f472c859170" (UID: "7d3b16ed-f9c0-4c11-b7de-4f472c859170"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:47:30.344281 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.344201    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-workload-socket" pod "7d3b16ed-f9c0-4c11-b7de-4f472c859170" (UID: "7d3b16ed-f9c0-4c11-b7de-4f472c859170"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:47:30.344464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.344440    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istiod-ca-cert" pod "7d3b16ed-f9c0-4c11-b7de-4f472c859170" (UID: "7d3b16ed-f9c0-4c11-b7de-4f472c859170"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:47:30.344607 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.344587    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:30.344709 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.344612    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:30.344709 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.344629    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:30.344709 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.344646    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:30.344709 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.344663    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:30.345888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.345755    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-podinfo" pod "7d3b16ed-f9c0-4c11-b7de-4f472c859170" (UID: "7d3b16ed-f9c0-4c11-b7de-4f472c859170"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:47:30.345888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.345870    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-envoy" pod "7d3b16ed-f9c0-4c11-b7de-4f472c859170" (UID: "7d3b16ed-f9c0-4c11-b7de-4f472c859170"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:47:30.346121 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.346092    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d3b16ed-f9c0-4c11-b7de-4f472c859170-kube-api-access-7h4v7" pod "7d3b16ed-f9c0-4c11-b7de-4f472c859170" (UID: "7d3b16ed-f9c0-4c11-b7de-4f472c859170"). InnerVolumeSpecName "kube-api-access-7h4v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:47:30.346346 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.346326    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-token" pod "7d3b16ed-f9c0-4c11-b7de-4f472c859170" (UID: "7d3b16ed-f9c0-4c11-b7de-4f472c859170"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:47:30.445511 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.445482    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:30.445511 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.445509    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:30.445709 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.445521    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h4v7\" (UniqueName: \"kubernetes.io/projected/7d3b16ed-f9c0-4c11-b7de-4f472c859170-kube-api-access-7h4v7\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:30.445709 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.445532    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d3b16ed-f9c0-4c11-b7de-4f472c859170-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:30.507455 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.507411    6059 generic.go:356] "Generic (PLEG): container finished" podID="7d3b16ed-f9c0-4c11-b7de-4f472c859170" containerID="044182d8a0b4d486379b91efbcce4bc4cbfa3077d05e45804fd86c7c9cbf4f37" exitCode=0
Nov 03 17:47:30.507610 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.507465    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"
Nov 03 17:47:30.507610 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.507488    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7" event={"ID":"7d3b16ed-f9c0-4c11-b7de-4f472c859170","Type":"ContainerDied","Data":"044182d8a0b4d486379b91efbcce4bc4cbfa3077d05e45804fd86c7c9cbf4f37"}
Nov 03 17:47:30.507610 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.507549    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7" event={"ID":"7d3b16ed-f9c0-4c11-b7de-4f472c859170","Type":"ContainerDied","Data":"a8ade10869a22e0d5cd2da540042b06cec5ad4284452f36eb4910b12cc552d3b"}
Nov 03 17:47:30.507610 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.507571    6059 scope.go:122] "RemoveContainer" containerID="044182d8a0b4d486379b91efbcce4bc4cbfa3077d05e45804fd86c7c9cbf4f37"
Nov 03 17:47:30.523572 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.523552    6059 scope.go:122] "RemoveContainer" containerID="34f240a2a5be507ab00360ed3768aa60e74a382a394fe9b62f5dcc0638588a85"
Nov 03 17:47:30.532893 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.532875    6059 scope.go:122] "RemoveContainer" containerID="044182d8a0b4d486379b91efbcce4bc4cbfa3077d05e45804fd86c7c9cbf4f37"
Nov 03 17:47:30.533198 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:47:30.533174    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"044182d8a0b4d486379b91efbcce4bc4cbfa3077d05e45804fd86c7c9cbf4f37\": container with ID starting with 044182d8a0b4d486379b91efbcce4bc4cbfa3077d05e45804fd86c7c9cbf4f37 not found: ID does not exist" containerID="044182d8a0b4d486379b91efbcce4bc4cbfa3077d05e45804fd86c7c9cbf4f37"
Nov 03 17:47:30.533253 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.533210    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"044182d8a0b4d486379b91efbcce4bc4cbfa3077d05e45804fd86c7c9cbf4f37"} err="failed to get container status \"044182d8a0b4d486379b91efbcce4bc4cbfa3077d05e45804fd86c7c9cbf4f37\": rpc error: code = NotFound desc = could not find container \"044182d8a0b4d486379b91efbcce4bc4cbfa3077d05e45804fd86c7c9cbf4f37\": container with ID starting with 044182d8a0b4d486379b91efbcce4bc4cbfa3077d05e45804fd86c7c9cbf4f37 not found: ID does not exist"
Nov 03 17:47:30.533253 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.533228    6059 scope.go:122] "RemoveContainer" containerID="34f240a2a5be507ab00360ed3768aa60e74a382a394fe9b62f5dcc0638588a85"
Nov 03 17:47:30.533539 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:47:30.533513    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34f240a2a5be507ab00360ed3768aa60e74a382a394fe9b62f5dcc0638588a85\": container with ID starting with 34f240a2a5be507ab00360ed3768aa60e74a382a394fe9b62f5dcc0638588a85 not found: ID does not exist" containerID="34f240a2a5be507ab00360ed3768aa60e74a382a394fe9b62f5dcc0638588a85"
Nov 03 17:47:30.533612 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.533537    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34f240a2a5be507ab00360ed3768aa60e74a382a394fe9b62f5dcc0638588a85"} err="failed to get container status \"34f240a2a5be507ab00360ed3768aa60e74a382a394fe9b62f5dcc0638588a85\": rpc error: code = NotFound desc = could not find container \"34f240a2a5be507ab00360ed3768aa60e74a382a394fe9b62f5dcc0638588a85\": container with ID starting with 34f240a2a5be507ab00360ed3768aa60e74a382a394fe9b62f5dcc0638588a85 not found: ID does not exist"
Nov 03 17:47:30.680663 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.680568    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"]
Nov 03 17:47:30.687902 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.687874    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-49c123d1-openshift-default-58457466b5-gdxk7"]
Nov 03 17:47:30.716028 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:30.715985    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:31.337740 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:31.337703    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d3b16ed-f9c0-4c11-b7de-4f472c859170" path="/var/lib/kubelet/pods/7d3b16ed-f9c0-4c11-b7de-4f472c859170/volumes"
Nov 03 17:47:31.540140 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:31.540103    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"]
Nov 03 17:47:31.540418 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:31.540389    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv" podUID="89c1509f-87be-4742-86ac-65c438f9523b" containerName="istio-proxy" containerID="cri-o://58a5837abf1a2a9450b92e5456fb6055f3c46deb8b7039fa455b963943d056ec" gracePeriod=30
Nov 03 17:47:31.928969 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:31.928938    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e258a30-m7xfw"]
Nov 03 17:47:31.929542 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:31.929528    6059 cpu_manager.go:414] "containerMap: removing container" podUID="7d3b16ed-f9c0-4c11-b7de-4f472c859170" containerName="istio-proxy"
Nov 03 17:47:31.929607 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:31.929543    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d3b16ed-f9c0-4c11-b7de-4f472c859170" containerName="istio-proxy"
Nov 03 17:47:31.929607 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:31.929558    6059 cpu_manager.go:414] "containerMap: removing container" podUID="7d3b16ed-f9c0-4c11-b7de-4f472c859170" containerName="istio-proxy"
Nov 03 17:47:31.929607 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:31.929563    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d3b16ed-f9c0-4c11-b7de-4f472c859170" containerName="istio-proxy"
Nov 03 17:47:31.929700 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:31.929672    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d3b16ed-f9c0-4c11-b7de-4f472c859170" containerName="istio-proxy"
Nov 03 17:47:31.929700 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:31.929682    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d3b16ed-f9c0-4c11-b7de-4f472c859170" containerName="istio-proxy"
Nov 03 17:47:31.937238 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:31.937198    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e258a30-m7xfw"
Nov 03 17:47:31.964273 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:31.964248    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dqs8\" (UniqueName: \"kubernetes.io/projected/b925950e-a95c-4b36-9103-2f83aff2f1ce-kube-api-access-5dqs8\") pod \"maas-api-revoke-sub-1e258a30-m7xfw\" (UID: \"b925950e-a95c-4b36-9103-2f83aff2f1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e258a30-m7xfw"
Nov 03 17:47:31.964454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:31.964283    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b925950e-a95c-4b36-9103-2f83aff2f1ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e258a30-m7xfw\" (UID: \"b925950e-a95c-4b36-9103-2f83aff2f1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e258a30-m7xfw"
Nov 03 17:47:31.964454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:31.964323    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b925950e-a95c-4b36-9103-2f83aff2f1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e258a30-m7xfw\" (UID: \"b925950e-a95c-4b36-9103-2f83aff2f1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e258a30-m7xfw"
Nov 03 17:47:31.993464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:31.993430    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:47:31.993636 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:31.993492    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:47:31.993636 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:31.993442    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:47:31.993636 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:31.993522    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:47:31.993636 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:31.993623    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e258a30-m7xfw"]
Nov 03 17:47:32.065581 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:32.065541    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dqs8\" (UniqueName: \"kubernetes.io/projected/b925950e-a95c-4b36-9103-2f83aff2f1ce-kube-api-access-5dqs8\") pod \"maas-api-revoke-sub-1e258a30-m7xfw\" (UID: \"b925950e-a95c-4b36-9103-2f83aff2f1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e258a30-m7xfw"
Nov 03 17:47:32.065726 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:32.065589    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b925950e-a95c-4b36-9103-2f83aff2f1ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e258a30-m7xfw\" (UID: \"b925950e-a95c-4b36-9103-2f83aff2f1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e258a30-m7xfw"
Nov 03 17:47:32.065726 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:32.065654    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b925950e-a95c-4b36-9103-2f83aff2f1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e258a30-m7xfw\" (UID: \"b925950e-a95c-4b36-9103-2f83aff2f1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e258a30-m7xfw"
Nov 03 17:47:32.066387 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:32.066344    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b925950e-a95c-4b36-9103-2f83aff2f1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e258a30-m7xfw\" (UID: \"b925950e-a95c-4b36-9103-2f83aff2f1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e258a30-m7xfw"
Nov 03 17:47:32.068002 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:32.067979    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b925950e-a95c-4b36-9103-2f83aff2f1ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e258a30-m7xfw\" (UID: \"b925950e-a95c-4b36-9103-2f83aff2f1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e258a30-m7xfw"
Nov 03 17:47:32.128646 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:32.128617    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dqs8\" (UniqueName: \"kubernetes.io/projected/b925950e-a95c-4b36-9103-2f83aff2f1ce-kube-api-access-5dqs8\") pod \"maas-api-revoke-sub-1e258a30-m7xfw\" (UID: \"b925950e-a95c-4b36-9103-2f83aff2f1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e258a30-m7xfw"
Nov 03 17:47:32.252467 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:32.252439    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e258a30-m7xfw"
Nov 03 17:47:32.628701 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:47:32.628658    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb925950e_a95c_4b36_9103_2f83aff2f1ce.slice/crio-6ddc111c06b794ea2b4ba6056f225b114462844c8eaad04652d4200f7c2d27cb WatchSource:0}: Error finding container 6ddc111c06b794ea2b4ba6056f225b114462844c8eaad04652d4200f7c2d27cb: Status 404 returned error can't find the container with id 6ddc111c06b794ea2b4ba6056f225b114462844c8eaad04652d4200f7c2d27cb
Nov 03 17:47:32.631254 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:32.631230    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e258a30-m7xfw"]
Nov 03 17:47:33.530523 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:33.530481    6059 generic.go:356] "Generic (PLEG): container finished" podID="b925950e-a95c-4b36-9103-2f83aff2f1ce" containerID="0633096f2f9e390380a0657ee17353f74254690fb1276a89bb048feb518a3473" exitCode=0
Nov 03 17:47:33.530668 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:33.530558    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e258a30-m7xfw" event={"ID":"b925950e-a95c-4b36-9103-2f83aff2f1ce","Type":"ContainerDied","Data":"0633096f2f9e390380a0657ee17353f74254690fb1276a89bb048feb518a3473"}
Nov 03 17:47:33.530668 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:33.530609    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e258a30-m7xfw" event={"ID":"b925950e-a95c-4b36-9103-2f83aff2f1ce","Type":"ContainerStarted","Data":"6ddc111c06b794ea2b4ba6056f225b114462844c8eaad04652d4200f7c2d27cb"}
Nov 03 17:47:34.027844 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:34.027808    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:47:34.028236 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:34.027854    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv" podUID="89c1509f-87be-4742-86ac-65c438f9523b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:47:34.689805 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:34.689773    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e258a30-m7xfw"
Nov 03 17:47:34.800434 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:34.800334    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b925950e-a95c-4b36-9103-2f83aff2f1ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b925950e-a95c-4b36-9103-2f83aff2f1ce-maas-api-openshift-ca\") pod \"b925950e-a95c-4b36-9103-2f83aff2f1ce\" (UID: \"b925950e-a95c-4b36-9103-2f83aff2f1ce\") "
Nov 03 17:47:34.800621 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:34.800442    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b925950e-a95c-4b36-9103-2f83aff2f1ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b925950e-a95c-4b36-9103-2f83aff2f1ce-maas-api-xks-ca\") pod \"b925950e-a95c-4b36-9103-2f83aff2f1ce\" (UID: \"b925950e-a95c-4b36-9103-2f83aff2f1ce\") "
Nov 03 17:47:34.800621 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:34.800593    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b925950e-a95c-4b36-9103-2f83aff2f1ce-kube-api-access-5dqs8\" (UniqueName: \"kubernetes.io/projected/b925950e-a95c-4b36-9103-2f83aff2f1ce-kube-api-access-5dqs8\") pod \"b925950e-a95c-4b36-9103-2f83aff2f1ce\" (UID: \"b925950e-a95c-4b36-9103-2f83aff2f1ce\") "
Nov 03 17:47:34.800727 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:34.800672    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b925950e-a95c-4b36-9103-2f83aff2f1ce-maas-api-openshift-ca" pod "b925950e-a95c-4b36-9103-2f83aff2f1ce" (UID: "b925950e-a95c-4b36-9103-2f83aff2f1ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:47:34.801071 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:34.801050    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b925950e-a95c-4b36-9103-2f83aff2f1ce-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:34.802482 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:34.802462    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b925950e-a95c-4b36-9103-2f83aff2f1ce-maas-api-xks-ca" pod "b925950e-a95c-4b36-9103-2f83aff2f1ce" (UID: "b925950e-a95c-4b36-9103-2f83aff2f1ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:47:34.802624 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:34.802609    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b925950e-a95c-4b36-9103-2f83aff2f1ce-kube-api-access-5dqs8" pod "b925950e-a95c-4b36-9103-2f83aff2f1ce" (UID: "b925950e-a95c-4b36-9103-2f83aff2f1ce"). InnerVolumeSpecName "kube-api-access-5dqs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:47:34.902042 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:34.902007    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dqs8\" (UniqueName: \"kubernetes.io/projected/b925950e-a95c-4b36-9103-2f83aff2f1ce-kube-api-access-5dqs8\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:34.902042 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:34.902040    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b925950e-a95c-4b36-9103-2f83aff2f1ce-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:35.357313 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:35.357279    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:35.545011 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:35.544982    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e258a30-m7xfw"
Nov 03 17:47:35.545156 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:35.544975    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e258a30-m7xfw" event={"ID":"b925950e-a95c-4b36-9103-2f83aff2f1ce","Type":"ContainerDied","Data":"6ddc111c06b794ea2b4ba6056f225b114462844c8eaad04652d4200f7c2d27cb"}
Nov 03 17:47:35.545156 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:35.545074    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ddc111c06b794ea2b4ba6056f225b114462844c8eaad04652d4200f7c2d27cb"
Nov 03 17:47:36.829157 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:36.829094    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:47:36.861899 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:47:36.861862    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:47:36.927729 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:36.927700    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-istio-data\") pod \"89c1509f-87be-4742-86ac-65c438f9523b\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") "
Nov 03 17:47:36.927729 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:36.927744    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-workload-socket\") pod \"89c1509f-87be-4742-86ac-65c438f9523b\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") "
Nov 03 17:47:36.928007 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:36.927832    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89c1509f-87be-4742-86ac-65c438f9523b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c1509f-87be-4742-86ac-65c438f9523b-istiod-ca-cert\") pod \"89c1509f-87be-4742-86ac-65c438f9523b\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") "
Nov 03 17:47:36.928007 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:36.927875    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89c1509f-87be-4742-86ac-65c438f9523b-istio-token\" (UniqueName: \"kubernetes.io/projected/89c1509f-87be-4742-86ac-65c438f9523b-istio-token\") pod \"89c1509f-87be-4742-86ac-65c438f9523b\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") "
Nov 03 17:47:36.928007 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:36.927902    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89c1509f-87be-4742-86ac-65c438f9523b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c1509f-87be-4742-86ac-65c438f9523b-istio-podinfo\") pod \"89c1509f-87be-4742-86ac-65c438f9523b\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") "
Nov 03 17:47:36.928007 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:36.927935    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89c1509f-87be-4742-86ac-65c438f9523b-kube-api-access-8wbmt\" (UniqueName: \"kubernetes.io/projected/89c1509f-87be-4742-86ac-65c438f9523b-kube-api-access-8wbmt\") pod \"89c1509f-87be-4742-86ac-65c438f9523b\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") "
Nov 03 17:47:36.928007 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:36.927973    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-istio-envoy\") pod \"89c1509f-87be-4742-86ac-65c438f9523b\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") "
Nov 03 17:47:36.928315 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:36.928014    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-workload-certs\") pod \"89c1509f-87be-4742-86ac-65c438f9523b\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") "
Nov 03 17:47:36.928315 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:36.928063    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-credential-socket\") pod \"89c1509f-87be-4742-86ac-65c438f9523b\" (UID: \"89c1509f-87be-4742-86ac-65c438f9523b\") "
Nov 03 17:47:36.928315 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:36.928086    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-workload-socket" pod "89c1509f-87be-4742-86ac-65c438f9523b" (UID: "89c1509f-87be-4742-86ac-65c438f9523b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:47:36.928315 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:36.928260    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c1509f-87be-4742-86ac-65c438f9523b-istiod-ca-cert" pod "89c1509f-87be-4742-86ac-65c438f9523b" (UID: "89c1509f-87be-4742-86ac-65c438f9523b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:47:36.928737 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:36.928712    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-workload-certs" pod "89c1509f-87be-4742-86ac-65c438f9523b" (UID: "89c1509f-87be-4742-86ac-65c438f9523b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:47:36.928848 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:36.928810    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:36.928848 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:36.928835    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c1509f-87be-4742-86ac-65c438f9523b-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:36.928990 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:36.928950    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-credential-socket" pod "89c1509f-87be-4742-86ac-65c438f9523b" (UID: "89c1509f-87be-4742-86ac-65c438f9523b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:47:36.930337 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:36.930313    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-istio-envoy" pod "89c1509f-87be-4742-86ac-65c438f9523b" (UID: "89c1509f-87be-4742-86ac-65c438f9523b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:47:36.930453 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:36.930432    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-istio-data" pod "89c1509f-87be-4742-86ac-65c438f9523b" (UID: "89c1509f-87be-4742-86ac-65c438f9523b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:47:36.930585 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:36.930558    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89c1509f-87be-4742-86ac-65c438f9523b-istio-podinfo" pod "89c1509f-87be-4742-86ac-65c438f9523b" (UID: "89c1509f-87be-4742-86ac-65c438f9523b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:47:36.930760 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:36.930714    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c1509f-87be-4742-86ac-65c438f9523b-istio-token" pod "89c1509f-87be-4742-86ac-65c438f9523b" (UID: "89c1509f-87be-4742-86ac-65c438f9523b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:47:36.930932 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:36.930909    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c1509f-87be-4742-86ac-65c438f9523b-kube-api-access-8wbmt" pod "89c1509f-87be-4742-86ac-65c438f9523b" (UID: "89c1509f-87be-4742-86ac-65c438f9523b"). InnerVolumeSpecName "kube-api-access-8wbmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:47:37.029821 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:37.029784    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c1509f-87be-4742-86ac-65c438f9523b-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:37.029821 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:37.029815    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c1509f-87be-4742-86ac-65c438f9523b-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:37.030047 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:37.029828    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wbmt\" (UniqueName: \"kubernetes.io/projected/89c1509f-87be-4742-86ac-65c438f9523b-kube-api-access-8wbmt\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:37.030047 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:37.029843    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:37.030047 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:37.029852    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:37.030047 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:37.029862    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:37.030047 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:37.029870    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c1509f-87be-4742-86ac-65c438f9523b-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:37.483748 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:37.483715    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8"
Nov 03 17:47:37.483941 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:37.483789    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l"
Nov 03 17:47:37.561884 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:37.561842    6059 generic.go:356] "Generic (PLEG): container finished" podID="89c1509f-87be-4742-86ac-65c438f9523b" containerID="58a5837abf1a2a9450b92e5456fb6055f3c46deb8b7039fa455b963943d056ec" exitCode=0
Nov 03 17:47:37.562057 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:37.561914    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"
Nov 03 17:47:37.562057 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:37.561915    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv" event={"ID":"89c1509f-87be-4742-86ac-65c438f9523b","Type":"ContainerDied","Data":"58a5837abf1a2a9450b92e5456fb6055f3c46deb8b7039fa455b963943d056ec"}
Nov 03 17:47:37.562057 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:37.561994    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv" event={"ID":"89c1509f-87be-4742-86ac-65c438f9523b","Type":"ContainerDied","Data":"6f7470474be9dbdf584fc328a7e58381b13b6644dfa23410c3b6e1e0779035d9"}
Nov 03 17:47:37.562057 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:37.562016    6059 scope.go:122] "RemoveContainer" containerID="58a5837abf1a2a9450b92e5456fb6055f3c46deb8b7039fa455b963943d056ec"
Nov 03 17:47:37.572197 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:37.572179    6059 scope.go:122] "RemoveContainer" containerID="58a5837abf1a2a9450b92e5456fb6055f3c46deb8b7039fa455b963943d056ec"
Nov 03 17:47:37.572526 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:47:37.572500    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58a5837abf1a2a9450b92e5456fb6055f3c46deb8b7039fa455b963943d056ec\": container with ID starting with 58a5837abf1a2a9450b92e5456fb6055f3c46deb8b7039fa455b963943d056ec not found: ID does not exist" containerID="58a5837abf1a2a9450b92e5456fb6055f3c46deb8b7039fa455b963943d056ec"
Nov 03 17:47:37.572623 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:37.572531    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58a5837abf1a2a9450b92e5456fb6055f3c46deb8b7039fa455b963943d056ec"} err="failed to get container status \"58a5837abf1a2a9450b92e5456fb6055f3c46deb8b7039fa455b963943d056ec\": rpc error: code = NotFound desc = could not find container \"58a5837abf1a2a9450b92e5456fb6055f3c46deb8b7039fa455b963943d056ec\": container with ID starting with 58a5837abf1a2a9450b92e5456fb6055f3c46deb8b7039fa455b963943d056ec not found: ID does not exist"
Nov 03 17:47:37.843698 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:37.843658    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"]
Nov 03 17:47:37.847800 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:37.847771    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-6x9sv"]
Nov 03 17:47:39.336822 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:39.336788    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89c1509f-87be-4742-86ac-65c438f9523b" path="/var/lib/kubelet/pods/89c1509f-87be-4742-86ac-65c438f9523b/volumes"
Nov 03 17:47:39.638629 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:39.638536    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 17:47:39.647162 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:39.647134    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 17:47:39.657117 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:39.657090    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:47:39.665097 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:39.665076    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:47:39.732679 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:39.732648    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:40.486510 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:40.486475    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:42.685091 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:42.685056    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:42.876218 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:42.876166    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:44.158784 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:44.158740    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:44.405485 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:44.405447    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:44.602056 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:44.602018    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:49.428871 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:49.428835    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:51.196454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.196423    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq"]
Nov 03 17:47:51.197026 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.197011    6059 cpu_manager.go:414] "containerMap: removing container" podUID="b925950e-a95c-4b36-9103-2f83aff2f1ce" containerName="revoke-keys"
Nov 03 17:47:51.197080 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.197027    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b925950e-a95c-4b36-9103-2f83aff2f1ce" containerName="revoke-keys"
Nov 03 17:47:51.197080 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.197043    6059 cpu_manager.go:414] "containerMap: removing container" podUID="89c1509f-87be-4742-86ac-65c438f9523b" containerName="istio-proxy"
Nov 03 17:47:51.197080 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.197048    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89c1509f-87be-4742-86ac-65c438f9523b" containerName="istio-proxy"
Nov 03 17:47:51.197180 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.197150    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b925950e-a95c-4b36-9103-2f83aff2f1ce" containerName="revoke-keys"
Nov 03 17:47:51.197180 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.197159    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="89c1509f-87be-4742-86ac-65c438f9523b" containerName="istio-proxy"
Nov 03 17:47:51.202595 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.202574    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq"
Nov 03 17:47:51.258883 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.258847    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:47:51.300089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.300057    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28714918-4efe-4c05-af87-04e1200ec028-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1e2f82f-vjxqq\" (UID: \"28714918-4efe-4c05-af87-04e1200ec028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq"
Nov 03 17:47:51.300256 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.300128    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9ptg\" (UniqueName: \"kubernetes.io/projected/28714918-4efe-4c05-af87-04e1200ec028-kube-api-access-q9ptg\") pod \"maas-api-revoke-sub-d1e2f82f-vjxqq\" (UID: \"28714918-4efe-4c05-af87-04e1200ec028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq"
Nov 03 17:47:51.300331 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.300304    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28714918-4efe-4c05-af87-04e1200ec028-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1e2f82f-vjxqq\" (UID: \"28714918-4efe-4c05-af87-04e1200ec028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq"
Nov 03 17:47:51.310524 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.310498    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:47:51.310754 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.310741    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:47:51.310800 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.310766    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:47:51.314194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.314176    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq"]
Nov 03 17:47:51.401207 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.401173    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28714918-4efe-4c05-af87-04e1200ec028-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1e2f82f-vjxqq\" (UID: \"28714918-4efe-4c05-af87-04e1200ec028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq"
Nov 03 17:47:51.401418 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.401255    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28714918-4efe-4c05-af87-04e1200ec028-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1e2f82f-vjxqq\" (UID: \"28714918-4efe-4c05-af87-04e1200ec028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq"
Nov 03 17:47:51.401418 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.401301    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9ptg\" (UniqueName: \"kubernetes.io/projected/28714918-4efe-4c05-af87-04e1200ec028-kube-api-access-q9ptg\") pod \"maas-api-revoke-sub-d1e2f82f-vjxqq\" (UID: \"28714918-4efe-4c05-af87-04e1200ec028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq"
Nov 03 17:47:51.401808 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.401786    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28714918-4efe-4c05-af87-04e1200ec028-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1e2f82f-vjxqq\" (UID: \"28714918-4efe-4c05-af87-04e1200ec028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq"
Nov 03 17:47:51.403562 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.403542    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28714918-4efe-4c05-af87-04e1200ec028-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1e2f82f-vjxqq\" (UID: \"28714918-4efe-4c05-af87-04e1200ec028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq"
Nov 03 17:47:51.464928 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.464894    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9ptg\" (UniqueName: \"kubernetes.io/projected/28714918-4efe-4c05-af87-04e1200ec028-kube-api-access-q9ptg\") pod \"maas-api-revoke-sub-d1e2f82f-vjxqq\" (UID: \"28714918-4efe-4c05-af87-04e1200ec028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq"
Nov 03 17:47:51.516082 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.516053    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq"
Nov 03 17:47:51.895736 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:47:51.895701    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28714918_4efe_4c05_af87_04e1200ec028.slice/crio-d3c19accf2cfb4f06bbb673040443a5f7da62694b86394915bdbeb9eb0f78866 WatchSource:0}: Error finding container d3c19accf2cfb4f06bbb673040443a5f7da62694b86394915bdbeb9eb0f78866: Status 404 returned error can't find the container with id d3c19accf2cfb4f06bbb673040443a5f7da62694b86394915bdbeb9eb0f78866
Nov 03 17:47:51.899772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:51.899742    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq"]
Nov 03 17:47:52.741694 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:52.741660    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq" event={"ID":"28714918-4efe-4c05-af87-04e1200ec028","Type":"ContainerStarted","Data":"080859bb97ebb0ee35adc5754688501df431f80da038a19662acb1a03aa5bb4b"}
Nov 03 17:47:52.742084 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:52.741738    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq" event={"ID":"28714918-4efe-4c05-af87-04e1200ec028","Type":"ContainerStarted","Data":"d3c19accf2cfb4f06bbb673040443a5f7da62694b86394915bdbeb9eb0f78866"}
Nov 03 17:47:52.871736 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:52.871678    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq" podStartSLOduration=1.871663769 podStartE2EDuration="1.871663769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:52.868651215 +0000 UTC m=+1813.666364007" watchObservedRunningTime="2026-11-03 17:47:52.871663769 +0000 UTC m=+1813.669376550"
Nov 03 17:47:53.753438 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:53.753401    6059 generic.go:356] "Generic (PLEG): container finished" podID="28714918-4efe-4c05-af87-04e1200ec028" containerID="080859bb97ebb0ee35adc5754688501df431f80da038a19662acb1a03aa5bb4b" exitCode=0
Nov 03 17:47:53.753917 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:53.753487    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq" event={"ID":"28714918-4efe-4c05-af87-04e1200ec028","Type":"ContainerDied","Data":"080859bb97ebb0ee35adc5754688501df431f80da038a19662acb1a03aa5bb4b"}
Nov 03 17:47:53.883039 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:53.883002    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:54.085471 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:54.085387    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:54.273377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:54.273324    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:47:54.902846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:54.902816    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq"
Nov 03 17:47:54.946333 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:54.946304    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28714918-4efe-4c05-af87-04e1200ec028-kube-api-access-q9ptg\" (UniqueName: \"kubernetes.io/projected/28714918-4efe-4c05-af87-04e1200ec028-kube-api-access-q9ptg\") pod \"28714918-4efe-4c05-af87-04e1200ec028\" (UID: \"28714918-4efe-4c05-af87-04e1200ec028\") "
Nov 03 17:47:54.946516 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:54.946410    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28714918-4efe-4c05-af87-04e1200ec028-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28714918-4efe-4c05-af87-04e1200ec028-maas-api-openshift-ca\") pod \"28714918-4efe-4c05-af87-04e1200ec028\" (UID: \"28714918-4efe-4c05-af87-04e1200ec028\") "
Nov 03 17:47:54.946597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:54.946540    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28714918-4efe-4c05-af87-04e1200ec028-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28714918-4efe-4c05-af87-04e1200ec028-maas-api-xks-ca\") pod \"28714918-4efe-4c05-af87-04e1200ec028\" (UID: \"28714918-4efe-4c05-af87-04e1200ec028\") "
Nov 03 17:47:54.946812 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:54.946783    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28714918-4efe-4c05-af87-04e1200ec028-maas-api-openshift-ca" pod "28714918-4efe-4c05-af87-04e1200ec028" (UID: "28714918-4efe-4c05-af87-04e1200ec028"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:47:54.947078 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:54.947060    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28714918-4efe-4c05-af87-04e1200ec028-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:54.948457 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:54.948434    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28714918-4efe-4c05-af87-04e1200ec028-maas-api-xks-ca" pod "28714918-4efe-4c05-af87-04e1200ec028" (UID: "28714918-4efe-4c05-af87-04e1200ec028"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:47:54.948548 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:54.948508    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28714918-4efe-4c05-af87-04e1200ec028-kube-api-access-q9ptg" pod "28714918-4efe-4c05-af87-04e1200ec028" (UID: "28714918-4efe-4c05-af87-04e1200ec028"). InnerVolumeSpecName "kube-api-access-q9ptg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:47:55.048191 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:55.048062    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28714918-4efe-4c05-af87-04e1200ec028-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:55.048191 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:55.048089    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9ptg\" (UniqueName: \"kubernetes.io/projected/28714918-4efe-4c05-af87-04e1200ec028-kube-api-access-q9ptg\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:47:55.768237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:55.768208    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq"
Nov 03 17:47:55.768457 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:55.768239    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq" event={"ID":"28714918-4efe-4c05-af87-04e1200ec028","Type":"ContainerDied","Data":"d3c19accf2cfb4f06bbb673040443a5f7da62694b86394915bdbeb9eb0f78866"}
Nov 03 17:47:55.768457 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:55.768285    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3c19accf2cfb4f06bbb673040443a5f7da62694b86394915bdbeb9eb0f78866"
Nov 03 17:47:58.768141 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:47:58.768104    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:00.347515 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:00.347475    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-h98kq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:48:00.348127 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:00.348095    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq" podUID="1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:48:00.348591 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:00.348566    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-tdr2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:48:00.348756 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:00.348734    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c" podUID="4f0c42da-8020-48d1-9e61-7dfab0d4a65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:48:01.839118 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:01.839085    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:03.657167 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:03.657125    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:06.870802 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:48:06.870765    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:48:08.442338 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:08.442300    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:08.744412 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:08.744380    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:12.460324 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:12.460296    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/ai-tenant-e2e-extmod-b-8c9b0f_payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv_3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1/payload-processing/0.log"
Nov 03 17:48:12.676475 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:12.676438    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/models-as-a-service_payload-processing-external-model-86cbdddf86-2k5kw_ebefb030-567e-47f1-bb77-f62b222d5280/payload-processing/0.log"
Nov 03 17:48:13.084717 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.084683    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f1cb49-ft257"]
Nov 03 17:48:13.085284 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.085271    6059 cpu_manager.go:414] "containerMap: removing container" podUID="28714918-4efe-4c05-af87-04e1200ec028" containerName="revoke-keys"
Nov 03 17:48:13.085328 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.085286    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28714918-4efe-4c05-af87-04e1200ec028" containerName="revoke-keys"
Nov 03 17:48:13.085417 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.085407    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="28714918-4efe-4c05-af87-04e1200ec028" containerName="revoke-keys"
Nov 03 17:48:13.089122 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.089105    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f1cb49-ft257"
Nov 03 17:48:13.145752 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.145724    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:48:13.191408 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.191372    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dwlm\" (UniqueName: \"kubernetes.io/projected/c82ee932-a22b-4780-ab90-065599279e4a-kube-api-access-6dwlm\") pod \"maas-api-revoke-sub-79f1cb49-ft257\" (UID: \"c82ee932-a22b-4780-ab90-065599279e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f1cb49-ft257"
Nov 03 17:48:13.191592 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.191536    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82ee932-a22b-4780-ab90-065599279e4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79f1cb49-ft257\" (UID: \"c82ee932-a22b-4780-ab90-065599279e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f1cb49-ft257"
Nov 03 17:48:13.191679 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.191660    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82ee932-a22b-4780-ab90-065599279e4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79f1cb49-ft257\" (UID: \"c82ee932-a22b-4780-ab90-065599279e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f1cb49-ft257"
Nov 03 17:48:13.197665 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.197640    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:48:13.197799 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.197640    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:48:13.197799 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.197708    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:48:13.202678 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.202652    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f1cb49-ft257"]
Nov 03 17:48:13.293426 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.293386    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82ee932-a22b-4780-ab90-065599279e4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79f1cb49-ft257\" (UID: \"c82ee932-a22b-4780-ab90-065599279e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f1cb49-ft257"
Nov 03 17:48:13.293633 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.293503    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82ee932-a22b-4780-ab90-065599279e4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79f1cb49-ft257\" (UID: \"c82ee932-a22b-4780-ab90-065599279e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f1cb49-ft257"
Nov 03 17:48:13.293633 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.293595    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dwlm\" (UniqueName: \"kubernetes.io/projected/c82ee932-a22b-4780-ab90-065599279e4a-kube-api-access-6dwlm\") pod \"maas-api-revoke-sub-79f1cb49-ft257\" (UID: \"c82ee932-a22b-4780-ab90-065599279e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f1cb49-ft257"
Nov 03 17:48:13.293973 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.293951    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82ee932-a22b-4780-ab90-065599279e4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79f1cb49-ft257\" (UID: \"c82ee932-a22b-4780-ab90-065599279e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f1cb49-ft257"
Nov 03 17:48:13.295932 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.295913    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82ee932-a22b-4780-ab90-065599279e4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79f1cb49-ft257\" (UID: \"c82ee932-a22b-4780-ab90-065599279e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f1cb49-ft257"
Nov 03 17:48:13.356916 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.356845    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dwlm\" (UniqueName: \"kubernetes.io/projected/c82ee932-a22b-4780-ab90-065599279e4a-kube-api-access-6dwlm\") pod \"maas-api-revoke-sub-79f1cb49-ft257\" (UID: \"c82ee932-a22b-4780-ab90-065599279e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f1cb49-ft257"
Nov 03 17:48:13.403445 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.403416    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f1cb49-ft257"
Nov 03 17:48:13.734893 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.734857    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:13.775559 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:48:13.775530    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82ee932_a22b_4780_ab90_065599279e4a.slice/crio-d067ec658a76d2ea0326c22e688214518a746dd499be0b29a796479f86ce8512 WatchSource:0}: Error finding container d067ec658a76d2ea0326c22e688214518a746dd499be0b29a796479f86ce8512: Status 404 returned error can't find the container with id d067ec658a76d2ea0326c22e688214518a746dd499be0b29a796479f86ce8512
Nov 03 17:48:13.779458 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.779434    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f1cb49-ft257"]
Nov 03 17:48:13.957013 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:13.956979    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f1cb49-ft257" event={"ID":"c82ee932-a22b-4780-ab90-065599279e4a","Type":"ContainerStarted","Data":"d067ec658a76d2ea0326c22e688214518a746dd499be0b29a796479f86ce8512"}
Nov 03 17:48:14.965036 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:14.965002    6059 generic.go:356] "Generic (PLEG): container finished" podID="c82ee932-a22b-4780-ab90-065599279e4a" containerID="6724da802912d322cee7de47450a23e54116f20e4fbb36871b4e087056aeef7e" exitCode=0
Nov 03 17:48:14.965434 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:14.965090    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f1cb49-ft257" event={"ID":"c82ee932-a22b-4780-ab90-065599279e4a","Type":"ContainerDied","Data":"6724da802912d322cee7de47450a23e54116f20e4fbb36871b4e087056aeef7e"}
Nov 03 17:48:15.675990 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:15.675953    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:15.865735 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:15.865700    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:16.117831 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:16.117806    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f1cb49-ft257"
Nov 03 17:48:16.231793 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:16.231705    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c82ee932-a22b-4780-ab90-065599279e4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82ee932-a22b-4780-ab90-065599279e4a-maas-api-xks-ca\") pod \"c82ee932-a22b-4780-ab90-065599279e4a\" (UID: \"c82ee932-a22b-4780-ab90-065599279e4a\") "
Nov 03 17:48:16.231793 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:16.231774    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c82ee932-a22b-4780-ab90-065599279e4a-kube-api-access-6dwlm\" (UniqueName: \"kubernetes.io/projected/c82ee932-a22b-4780-ab90-065599279e4a-kube-api-access-6dwlm\") pod \"c82ee932-a22b-4780-ab90-065599279e4a\" (UID: \"c82ee932-a22b-4780-ab90-065599279e4a\") "
Nov 03 17:48:16.231975 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:16.231941    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c82ee932-a22b-4780-ab90-065599279e4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82ee932-a22b-4780-ab90-065599279e4a-maas-api-openshift-ca\") pod \"c82ee932-a22b-4780-ab90-065599279e4a\" (UID: \"c82ee932-a22b-4780-ab90-065599279e4a\") "
Nov 03 17:48:16.232291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:16.232265    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82ee932-a22b-4780-ab90-065599279e4a-maas-api-openshift-ca" pod "c82ee932-a22b-4780-ab90-065599279e4a" (UID: "c82ee932-a22b-4780-ab90-065599279e4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:48:16.232550 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:16.232530    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82ee932-a22b-4780-ab90-065599279e4a-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:16.233807 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:16.233780    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c82ee932-a22b-4780-ab90-065599279e4a-maas-api-xks-ca" pod "c82ee932-a22b-4780-ab90-065599279e4a" (UID: "c82ee932-a22b-4780-ab90-065599279e4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:48:16.233882 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:16.233847    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82ee932-a22b-4780-ab90-065599279e4a-kube-api-access-6dwlm" pod "c82ee932-a22b-4780-ab90-065599279e4a" (UID: "c82ee932-a22b-4780-ab90-065599279e4a"). InnerVolumeSpecName "kube-api-access-6dwlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:48:16.333365 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:16.333332    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82ee932-a22b-4780-ab90-065599279e4a-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:16.333525 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:16.333395    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dwlm\" (UniqueName: \"kubernetes.io/projected/c82ee932-a22b-4780-ab90-065599279e4a-kube-api-access-6dwlm\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:16.980511 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:16.980479    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f1cb49-ft257"
Nov 03 17:48:16.980511 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:16.980488    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f1cb49-ft257" event={"ID":"c82ee932-a22b-4780-ab90-065599279e4a","Type":"ContainerDied","Data":"d067ec658a76d2ea0326c22e688214518a746dd499be0b29a796479f86ce8512"}
Nov 03 17:48:16.980744 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:16.980550    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d067ec658a76d2ea0326c22e688214518a746dd499be0b29a796479f86ce8512"
Nov 03 17:48:18.378859 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:18.378825    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:23.307394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:23.307318    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:25.628815 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:25.628782    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:25.818630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:25.818594    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:25.889434 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:25.889344    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7484-8e52e9"]
Nov 03 17:48:25.890616 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:25.890595    6059 cpu_manager.go:414] "containerMap: removing container" podUID="c82ee932-a22b-4780-ab90-065599279e4a" containerName="revoke-keys"
Nov 03 17:48:25.890671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:25.890619    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c82ee932-a22b-4780-ab90-065599279e4a" containerName="revoke-keys"
Nov 03 17:48:25.890818 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:25.890804    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c82ee932-a22b-4780-ab90-065599279e4a" containerName="revoke-keys"
Nov 03 17:48:25.895075 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:25.895057    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-8e52e9"
Nov 03 17:48:25.949363 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:25.949315    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7484-8e52e9"]
Nov 03 17:48:25.968587 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:25.968551    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc62b\" (UniqueName: \"kubernetes.io/projected/8aced450-56a7-4d4f-9b3c-e30c0f406bcf-kube-api-access-rc62b\") pod \"test-curl-7484-8e52e9\" (UID: \"8aced450-56a7-4d4f-9b3c-e30c0f406bcf\") " pod="opendatahub/test-curl-7484-8e52e9"
Nov 03 17:48:25.996627 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:25.996585    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hjccf\""
Nov 03 17:48:26.070712 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:26.070676    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc62b\" (UniqueName: \"kubernetes.io/projected/8aced450-56a7-4d4f-9b3c-e30c0f406bcf-kube-api-access-rc62b\") pod \"test-curl-7484-8e52e9\" (UID: \"8aced450-56a7-4d4f-9b3c-e30c0f406bcf\") " pod="opendatahub/test-curl-7484-8e52e9"
Nov 03 17:48:26.133135 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:26.133108    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc62b\" (UniqueName: \"kubernetes.io/projected/8aced450-56a7-4d4f-9b3c-e30c0f406bcf-kube-api-access-rc62b\") pod \"test-curl-7484-8e52e9\" (UID: \"8aced450-56a7-4d4f-9b3c-e30c0f406bcf\") " pod="opendatahub/test-curl-7484-8e52e9"
Nov 03 17:48:26.209756 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:26.209724    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-8e52e9"
Nov 03 17:48:26.580405 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:48:26.580351    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8aced450_56a7_4d4f_9b3c_e30c0f406bcf.slice/crio-18c23e74403e5eec5d24d7bf9ce8b3ead23b246c0f1058d3c96163504c38b293 WatchSource:0}: Error finding container 18c23e74403e5eec5d24d7bf9ce8b3ead23b246c0f1058d3c96163504c38b293: Status 404 returned error can't find the container with id 18c23e74403e5eec5d24d7bf9ce8b3ead23b246c0f1058d3c96163504c38b293
Nov 03 17:48:26.583942 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:26.583918    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7484-8e52e9"]
Nov 03 17:48:27.017420 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.017389    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53d5534c-knmgq"]
Nov 03 17:48:27.022439 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.022417    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d5534c-knmgq"
Nov 03 17:48:27.123262 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.123224    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7484-8e52e9" event={"ID":"8aced450-56a7-4d4f-9b3c-e30c0f406bcf","Type":"ContainerStarted","Data":"a120bd64c0c10e94beb4a2c69da80697332f06b87ac6fb69cb60cee4a2914a12"}
Nov 03 17:48:27.123453 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.123272    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7484-8e52e9" event={"ID":"8aced450-56a7-4d4f-9b3c-e30c0f406bcf","Type":"ContainerStarted","Data":"18c23e74403e5eec5d24d7bf9ce8b3ead23b246c0f1058d3c96163504c38b293"}
Nov 03 17:48:27.128312 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.128284    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:48:27.128450 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.128323    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:48:27.128450 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.128386    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:48:27.128450 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.128391    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:48:27.128704 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.128689    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53d5534c-knmgq"]
Nov 03 17:48:27.203377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.203183    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a770d2-0510-48b2-83d0-244d9f3d2816-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53d5534c-knmgq\" (UID: \"11a770d2-0510-48b2-83d0-244d9f3d2816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d5534c-knmgq"
Nov 03 17:48:27.203596 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.203483    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frqc4\" (UniqueName: \"kubernetes.io/projected/11a770d2-0510-48b2-83d0-244d9f3d2816-kube-api-access-frqc4\") pod \"maas-api-revoke-sub-53d5534c-knmgq\" (UID: \"11a770d2-0510-48b2-83d0-244d9f3d2816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d5534c-knmgq"
Nov 03 17:48:27.203596 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.203556    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a770d2-0510-48b2-83d0-244d9f3d2816-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53d5534c-knmgq\" (UID: \"11a770d2-0510-48b2-83d0-244d9f3d2816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d5534c-knmgq"
Nov 03 17:48:27.262737 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.262686    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7484-8e52e9" podStartSLOduration=2.262667652 podStartE2EDuration="2.262667652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:48:27.260407896 +0000 UTC m=+1848.058120673" watchObservedRunningTime="2026-11-03 17:48:27.262667652 +0000 UTC m=+1848.060380437"
Nov 03 17:48:27.304647 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.304568    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a770d2-0510-48b2-83d0-244d9f3d2816-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53d5534c-knmgq\" (UID: \"11a770d2-0510-48b2-83d0-244d9f3d2816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d5534c-knmgq"
Nov 03 17:48:27.304792 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.304707    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frqc4\" (UniqueName: \"kubernetes.io/projected/11a770d2-0510-48b2-83d0-244d9f3d2816-kube-api-access-frqc4\") pod \"maas-api-revoke-sub-53d5534c-knmgq\" (UID: \"11a770d2-0510-48b2-83d0-244d9f3d2816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d5534c-knmgq"
Nov 03 17:48:27.304792 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.304746    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a770d2-0510-48b2-83d0-244d9f3d2816-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53d5534c-knmgq\" (UID: \"11a770d2-0510-48b2-83d0-244d9f3d2816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d5534c-knmgq"
Nov 03 17:48:27.305379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.305332    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a770d2-0510-48b2-83d0-244d9f3d2816-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53d5534c-knmgq\" (UID: \"11a770d2-0510-48b2-83d0-244d9f3d2816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d5534c-knmgq"
Nov 03 17:48:27.306915 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.306886    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a770d2-0510-48b2-83d0-244d9f3d2816-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53d5534c-knmgq\" (UID: \"11a770d2-0510-48b2-83d0-244d9f3d2816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d5534c-knmgq"
Nov 03 17:48:27.367851 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.367822    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frqc4\" (UniqueName: \"kubernetes.io/projected/11a770d2-0510-48b2-83d0-244d9f3d2816-kube-api-access-frqc4\") pod \"maas-api-revoke-sub-53d5534c-knmgq\" (UID: \"11a770d2-0510-48b2-83d0-244d9f3d2816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d5534c-knmgq"
Nov 03 17:48:27.637797 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.637716    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d5534c-knmgq"
Nov 03 17:48:27.780903 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.780871    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"]
Nov 03 17:48:27.785303 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.785282    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"
Nov 03 17:48:27.845170 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.845136    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"]
Nov 03 17:48:27.912909 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.912816    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nmq4\" (UniqueName: \"kubernetes.io/projected/fc7d2c53-ef50-455f-9661-296553225d59-kube-api-access-8nmq4\") pod \"payload-processing-external-model-68478f785f-pxl4p\" (UID: \"fc7d2c53-ef50-455f-9661-296553225d59\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"
Nov 03 17:48:27.912909 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.912880    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc7d2c53-ef50-455f-9661-296553225d59-provider-credentials\") pod \"payload-processing-external-model-68478f785f-pxl4p\" (UID: \"fc7d2c53-ef50-455f-9661-296553225d59\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"
Nov 03 17:48:27.913113 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.912981    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7d2c53-ef50-455f-9661-296553225d59-plugins-config-volume\") pod \"payload-processing-external-model-68478f785f-pxl4p\" (UID: \"fc7d2c53-ef50-455f-9661-296553225d59\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"
Nov 03 17:48:27.913169 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:27.913152    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc7d2c53-ef50-455f-9661-296553225d59-routing-overlay-volume\") pod \"payload-processing-external-model-68478f785f-pxl4p\" (UID: \"fc7d2c53-ef50-455f-9661-296553225d59\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"
Nov 03 17:48:28.014452 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:28.014415    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc7d2c53-ef50-455f-9661-296553225d59-routing-overlay-volume\") pod \"payload-processing-external-model-68478f785f-pxl4p\" (UID: \"fc7d2c53-ef50-455f-9661-296553225d59\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"
Nov 03 17:48:28.014590 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:48:28.014454    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a770d2_0510_48b2_83d0_244d9f3d2816.slice/crio-9e97874950c8b91df2f4d655e22cb8b5606f397d634af5b286910f678c64d1d9 WatchSource:0}: Error finding container 9e97874950c8b91df2f4d655e22cb8b5606f397d634af5b286910f678c64d1d9: Status 404 returned error can't find the container with id 9e97874950c8b91df2f4d655e22cb8b5606f397d634af5b286910f678c64d1d9
Nov 03 17:48:28.014738 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:28.014712    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nmq4\" (UniqueName: \"kubernetes.io/projected/fc7d2c53-ef50-455f-9661-296553225d59-kube-api-access-8nmq4\") pod \"payload-processing-external-model-68478f785f-pxl4p\" (UID: \"fc7d2c53-ef50-455f-9661-296553225d59\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"
Nov 03 17:48:28.014872 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:28.014793    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc7d2c53-ef50-455f-9661-296553225d59-provider-credentials\") pod \"payload-processing-external-model-68478f785f-pxl4p\" (UID: \"fc7d2c53-ef50-455f-9661-296553225d59\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"
Nov 03 17:48:28.014947 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:28.014875    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7d2c53-ef50-455f-9661-296553225d59-plugins-config-volume\") pod \"payload-processing-external-model-68478f785f-pxl4p\" (UID: \"fc7d2c53-ef50-455f-9661-296553225d59\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"
Nov 03 17:48:28.015502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:28.015412    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc7d2c53-ef50-455f-9661-296553225d59-routing-overlay-volume\") pod \"payload-processing-external-model-68478f785f-pxl4p\" (UID: \"fc7d2c53-ef50-455f-9661-296553225d59\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"
Nov 03 17:48:28.015926 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:28.015821    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7d2c53-ef50-455f-9661-296553225d59-plugins-config-volume\") pod \"payload-processing-external-model-68478f785f-pxl4p\" (UID: \"fc7d2c53-ef50-455f-9661-296553225d59\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"
Nov 03 17:48:28.017817 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:28.017757    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc7d2c53-ef50-455f-9661-296553225d59-provider-credentials\") pod \"payload-processing-external-model-68478f785f-pxl4p\" (UID: \"fc7d2c53-ef50-455f-9661-296553225d59\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"
Nov 03 17:48:28.021553 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:28.020693    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:28.021553 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:28.021219    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53d5534c-knmgq"]
Nov 03 17:48:28.078492 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:28.078458    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nmq4\" (UniqueName: \"kubernetes.io/projected/fc7d2c53-ef50-455f-9661-296553225d59-kube-api-access-8nmq4\") pod \"payload-processing-external-model-68478f785f-pxl4p\" (UID: \"fc7d2c53-ef50-455f-9661-296553225d59\") " pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"
Nov 03 17:48:28.099341 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:28.099310    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"
Nov 03 17:48:28.133513 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:28.133476    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d5534c-knmgq" event={"ID":"11a770d2-0510-48b2-83d0-244d9f3d2816","Type":"ContainerStarted","Data":"9e97874950c8b91df2f4d655e22cb8b5606f397d634af5b286910f678c64d1d9"}
Nov 03 17:48:28.478804 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:48:28.478765    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7d2c53_ef50_455f_9661_296553225d59.slice/crio-378d867ce7a89b0698f201f267c8459119934bf07e3ee96dc1b52ca0d9be912a WatchSource:0}: Error finding container 378d867ce7a89b0698f201f267c8459119934bf07e3ee96dc1b52ca0d9be912a: Status 404 returned error can't find the container with id 378d867ce7a89b0698f201f267c8459119934bf07e3ee96dc1b52ca0d9be912a
Nov 03 17:48:28.482112 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:28.482092    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"]
Nov 03 17:48:28.752173 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:28.752073    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7484-8e52e9"]
Nov 03 17:48:28.760521 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:28.760489    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7484-8e52e9"]
Nov 03 17:48:28.906150 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:28.906112    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7484-9837a3"]
Nov 03 17:48:28.907060 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:28.907044    6059 cpu_manager.go:414] "containerMap: removing container" podUID="8aced450-56a7-4d4f-9b3c-e30c0f406bcf" containerName="test-curl-7484-8e52e9"
Nov 03 17:48:28.907115 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:28.907064    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aced450-56a7-4d4f-9b3c-e30c0f406bcf" containerName="test-curl-7484-8e52e9"
Nov 03 17:48:28.907200 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:28.907188    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aced450-56a7-4d4f-9b3c-e30c0f406bcf" containerName="test-curl-7484-8e52e9"
Nov 03 17:48:28.911091 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:28.911075    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-9837a3"
Nov 03 17:48:28.923985 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:28.923959    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xqn8\" (UniqueName: \"kubernetes.io/projected/2fe91006-9ef1-4592-bfa6-937bc4a95a83-kube-api-access-9xqn8\") pod \"test-curl-7484-9837a3\" (UID: \"2fe91006-9ef1-4592-bfa6-937bc4a95a83\") " pod="opendatahub/test-curl-7484-9837a3"
Nov 03 17:48:28.968704 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:28.968673    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7484-9837a3"]
Nov 03 17:48:29.025094 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:29.025061    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xqn8\" (UniqueName: \"kubernetes.io/projected/2fe91006-9ef1-4592-bfa6-937bc4a95a83-kube-api-access-9xqn8\") pod \"test-curl-7484-9837a3\" (UID: \"2fe91006-9ef1-4592-bfa6-937bc4a95a83\") " pod="opendatahub/test-curl-7484-9837a3"
Nov 03 17:48:29.088059 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:29.088029    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xqn8\" (UniqueName: \"kubernetes.io/projected/2fe91006-9ef1-4592-bfa6-937bc4a95a83-kube-api-access-9xqn8\") pod \"test-curl-7484-9837a3\" (UID: \"2fe91006-9ef1-4592-bfa6-937bc4a95a83\") " pod="opendatahub/test-curl-7484-9837a3"
Nov 03 17:48:29.141983 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:29.141947    6059 generic.go:356] "Generic (PLEG): container finished" podID="11a770d2-0510-48b2-83d0-244d9f3d2816" containerID="cdad340cdb7dad6c2d4d6814f5bac6f5d5a3175ae0a7385145cddb0018e3ac13" exitCode=0
Nov 03 17:48:29.142164 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:29.142036    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d5534c-knmgq" event={"ID":"11a770d2-0510-48b2-83d0-244d9f3d2816","Type":"ContainerDied","Data":"cdad340cdb7dad6c2d4d6814f5bac6f5d5a3175ae0a7385145cddb0018e3ac13"}
Nov 03 17:48:29.152483 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:29.152428    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7484-8e52e9" podUID="8aced450-56a7-4d4f-9b3c-e30c0f406bcf" containerName="test-curl-7484-8e52e9" containerID="cri-o://a120bd64c0c10e94beb4a2c69da80697332f06b87ac6fb69cb60cee4a2914a12" gracePeriod=2
Nov 03 17:48:29.152483 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:29.152456    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p" event={"ID":"fc7d2c53-ef50-455f-9661-296553225d59","Type":"ContainerStarted","Data":"08806ec4be8d786a9d89f93dc9fecaa80c2209e266f3edee8b70641059332d59"}
Nov 03 17:48:29.152711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:29.152581    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p" event={"ID":"fc7d2c53-ef50-455f-9661-296553225d59","Type":"ContainerStarted","Data":"378d867ce7a89b0698f201f267c8459119934bf07e3ee96dc1b52ca0d9be912a"}
Nov 03 17:48:29.152711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:29.152623    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"
Nov 03 17:48:29.223141 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:29.223110    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-9837a3"
Nov 03 17:48:29.314778 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:48:29.314497    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-8e52e9\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="8aced450-56a7-4d4f-9b3c-e30c0f406bcf" pod="opendatahub/test-curl-7484-8e52e9"
Nov 03 17:48:29.438965 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:29.438912    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p" podStartSLOduration=2.438898438 podStartE2EDuration="2.438898438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:48:29.43566121 +0000 UTC m=+1850.233373980" watchObservedRunningTime="2026-11-03 17:48:29.438898438 +0000 UTC m=+1850.236611263"
Nov 03 17:48:29.596726 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:48:29.596694    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fe91006_9ef1_4592_bfa6_937bc4a95a83.slice/crio-ea44e3c5096ac65e031f180a4c72611ebdd75cce9923e80bdd155bd1f5c62dd6 WatchSource:0}: Error finding container ea44e3c5096ac65e031f180a4c72611ebdd75cce9923e80bdd155bd1f5c62dd6: Status 404 returned error can't find the container with id ea44e3c5096ac65e031f180a4c72611ebdd75cce9923e80bdd155bd1f5c62dd6
Nov 03 17:48:29.599418 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:29.599396    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7484-9837a3"]
Nov 03 17:48:30.160878 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:30.160842    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7484-9837a3" event={"ID":"2fe91006-9ef1-4592-bfa6-937bc4a95a83","Type":"ContainerStarted","Data":"aea77e024a0fa02c6a297bd0fe23b671b5859c00e58f20710baf10fce831ff46"}
Nov 03 17:48:30.160878 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:30.160885    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7484-9837a3" event={"ID":"2fe91006-9ef1-4592-bfa6-937bc4a95a83","Type":"ContainerStarted","Data":"ea44e3c5096ac65e031f180a4c72611ebdd75cce9923e80bdd155bd1f5c62dd6"}
Nov 03 17:48:30.280805 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:30.280746    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7484-9837a3" podStartSLOduration=2.280731957 podStartE2EDuration="2.280731957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:48:30.279633193 +0000 UTC m=+1851.077345975" watchObservedRunningTime="2026-11-03 17:48:30.280731957 +0000 UTC m=+1851.078444738"
Nov 03 17:48:30.319847 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:30.319822    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d5534c-knmgq"
Nov 03 17:48:30.347601 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:30.347569    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a770d2-0510-48b2-83d0-244d9f3d2816-kube-api-access-frqc4\" (UniqueName: \"kubernetes.io/projected/11a770d2-0510-48b2-83d0-244d9f3d2816-kube-api-access-frqc4\") pod \"11a770d2-0510-48b2-83d0-244d9f3d2816\" (UID: \"11a770d2-0510-48b2-83d0-244d9f3d2816\") "
Nov 03 17:48:30.347779 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:30.347760    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11a770d2-0510-48b2-83d0-244d9f3d2816-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a770d2-0510-48b2-83d0-244d9f3d2816-maas-api-xks-ca\") pod \"11a770d2-0510-48b2-83d0-244d9f3d2816\" (UID: \"11a770d2-0510-48b2-83d0-244d9f3d2816\") "
Nov 03 17:48:30.347965 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:30.347946    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a770d2-0510-48b2-83d0-244d9f3d2816-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a770d2-0510-48b2-83d0-244d9f3d2816-maas-api-openshift-ca\") pod \"11a770d2-0510-48b2-83d0-244d9f3d2816\" (UID: \"11a770d2-0510-48b2-83d0-244d9f3d2816\") "
Nov 03 17:48:30.348336 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:30.348306    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a770d2-0510-48b2-83d0-244d9f3d2816-maas-api-openshift-ca" pod "11a770d2-0510-48b2-83d0-244d9f3d2816" (UID: "11a770d2-0510-48b2-83d0-244d9f3d2816"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:48:30.348507 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:30.348489    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a770d2-0510-48b2-83d0-244d9f3d2816-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:30.349636 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:30.349612    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a770d2-0510-48b2-83d0-244d9f3d2816-kube-api-access-frqc4" pod "11a770d2-0510-48b2-83d0-244d9f3d2816" (UID: "11a770d2-0510-48b2-83d0-244d9f3d2816"). InnerVolumeSpecName "kube-api-access-frqc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:48:30.349728 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:30.349683    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11a770d2-0510-48b2-83d0-244d9f3d2816-maas-api-xks-ca" pod "11a770d2-0510-48b2-83d0-244d9f3d2816" (UID: "11a770d2-0510-48b2-83d0-244d9f3d2816"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:48:30.450122 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:30.450042    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frqc4\" (UniqueName: \"kubernetes.io/projected/11a770d2-0510-48b2-83d0-244d9f3d2816-kube-api-access-frqc4\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:30.450122 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:30.450079    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a770d2-0510-48b2-83d0-244d9f3d2816-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:31.169867 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:31.169844    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d5534c-knmgq"
Nov 03 17:48:31.170224 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:31.169893    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d5534c-knmgq" event={"ID":"11a770d2-0510-48b2-83d0-244d9f3d2816","Type":"ContainerDied","Data":"9e97874950c8b91df2f4d655e22cb8b5606f397d634af5b286910f678c64d1d9"}
Nov 03 17:48:31.170224 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:31.169945    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e97874950c8b91df2f4d655e22cb8b5606f397d634af5b286910f678c64d1d9"
Nov 03 17:48:31.401315 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:31.401288    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-8e52e9"
Nov 03 17:48:31.454465 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:48:31.454349    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-8e52e9\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="8aced450-56a7-4d4f-9b3c-e30c0f406bcf" pod="opendatahub/test-curl-7484-8e52e9"
Nov 03 17:48:31.463031 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:31.463000    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aced450-56a7-4d4f-9b3c-e30c0f406bcf-kube-api-access-rc62b\" (UniqueName: \"kubernetes.io/projected/8aced450-56a7-4d4f-9b3c-e30c0f406bcf-kube-api-access-rc62b\") pod \"8aced450-56a7-4d4f-9b3c-e30c0f406bcf\" (UID: \"8aced450-56a7-4d4f-9b3c-e30c0f406bcf\") "
Nov 03 17:48:31.465418 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:31.465381    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aced450-56a7-4d4f-9b3c-e30c0f406bcf-kube-api-access-rc62b" pod "8aced450-56a7-4d4f-9b3c-e30c0f406bcf" (UID: "8aced450-56a7-4d4f-9b3c-e30c0f406bcf"). InnerVolumeSpecName "kube-api-access-rc62b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:48:31.565530 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:31.565499    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc62b\" (UniqueName: \"kubernetes.io/projected/8aced450-56a7-4d4f-9b3c-e30c0f406bcf-kube-api-access-rc62b\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:31.746637 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:31.746596    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7484-9837a3"]
Nov 03 17:48:31.751573 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:31.751540    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7484-9837a3"]
Nov 03 17:48:32.178896 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.178802    6059 generic.go:356] "Generic (PLEG): container finished" podID="8aced450-56a7-4d4f-9b3c-e30c0f406bcf" containerID="a120bd64c0c10e94beb4a2c69da80697332f06b87ac6fb69cb60cee4a2914a12" exitCode=137
Nov 03 17:48:32.178896 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.178855    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-8e52e9"
Nov 03 17:48:32.179458 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.178905    6059 scope.go:122] "RemoveContainer" containerID="a120bd64c0c10e94beb4a2c69da80697332f06b87ac6fb69cb60cee4a2914a12"
Nov 03 17:48:32.179458 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.179087    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7484-9837a3" podUID="2fe91006-9ef1-4592-bfa6-937bc4a95a83" containerName="test-curl-7484-9837a3" containerID="cri-o://aea77e024a0fa02c6a297bd0fe23b671b5859c00e58f20710baf10fce831ff46" gracePeriod=2
Nov 03 17:48:32.189452 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.189434    6059 scope.go:122] "RemoveContainer" containerID="a120bd64c0c10e94beb4a2c69da80697332f06b87ac6fb69cb60cee4a2914a12"
Nov 03 17:48:32.189763 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:48:32.189740    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a120bd64c0c10e94beb4a2c69da80697332f06b87ac6fb69cb60cee4a2914a12\": container with ID starting with a120bd64c0c10e94beb4a2c69da80697332f06b87ac6fb69cb60cee4a2914a12 not found: ID does not exist" containerID="a120bd64c0c10e94beb4a2c69da80697332f06b87ac6fb69cb60cee4a2914a12"
Nov 03 17:48:32.189816 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.189773    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a120bd64c0c10e94beb4a2c69da80697332f06b87ac6fb69cb60cee4a2914a12"} err="failed to get container status \"a120bd64c0c10e94beb4a2c69da80697332f06b87ac6fb69cb60cee4a2914a12\": rpc error: code = NotFound desc = could not find container \"a120bd64c0c10e94beb4a2c69da80697332f06b87ac6fb69cb60cee4a2914a12\": container with ID starting with a120bd64c0c10e94beb4a2c69da80697332f06b87ac6fb69cb60cee4a2914a12 not found: ID does not exist"
Nov 03 17:48:32.232268 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:48:32.232227    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-9837a3\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="2fe91006-9ef1-4592-bfa6-937bc4a95a83" pod="opendatahub/test-curl-7484-9837a3"
Nov 03 17:48:32.285293 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:48:32.285256    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-8e52e9\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="8aced450-56a7-4d4f-9b3c-e30c0f406bcf" pod="opendatahub/test-curl-7484-8e52e9"
Nov 03 17:48:32.338582 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:48:32.338541    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-9837a3\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="2fe91006-9ef1-4592-bfa6-937bc4a95a83" pod="opendatahub/test-curl-7484-9837a3"
Nov 03 17:48:32.391593 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:48:32.391551    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-8e52e9\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="8aced450-56a7-4d4f-9b3c-e30c0f406bcf" pod="opendatahub/test-curl-7484-8e52e9"
Nov 03 17:48:32.616279 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.616246    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:32.810957 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.810921    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb"]
Nov 03 17:48:32.811583 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.811567    6059 cpu_manager.go:414] "containerMap: removing container" podUID="11a770d2-0510-48b2-83d0-244d9f3d2816" containerName="revoke-keys"
Nov 03 17:48:32.811640 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.811585    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a770d2-0510-48b2-83d0-244d9f3d2816" containerName="revoke-keys"
Nov 03 17:48:32.811640 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.811595    6059 cpu_manager.go:414] "containerMap: removing container" podUID="2fe91006-9ef1-4592-bfa6-937bc4a95a83" containerName="test-curl-7484-9837a3"
Nov 03 17:48:32.811640 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.811601    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe91006-9ef1-4592-bfa6-937bc4a95a83" containerName="test-curl-7484-9837a3"
Nov 03 17:48:32.811751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.811697    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a770d2-0510-48b2-83d0-244d9f3d2816" containerName="revoke-keys"
Nov 03 17:48:32.811751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.811704    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe91006-9ef1-4592-bfa6-937bc4a95a83" containerName="test-curl-7484-9837a3"
Nov 03 17:48:32.816301 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.816285    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb"
Nov 03 17:48:32.869598 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:48:32.869502    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-9837a3\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="2fe91006-9ef1-4592-bfa6-937bc4a95a83" pod="opendatahub/test-curl-7484-9837a3"
Nov 03 17:48:32.871543 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.871513    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:48:32.871696 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.871617    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:48:32.881912 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.881877    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f639120d-e1f4-4700-a328-fec7ee568f3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-095a6192-kc4xb\" (UID: \"f639120d-e1f4-4700-a328-fec7ee568f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb"
Nov 03 17:48:32.882047 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.881925    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f639120d-e1f4-4700-a328-fec7ee568f3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-095a6192-kc4xb\" (UID: \"f639120d-e1f4-4700-a328-fec7ee568f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb"
Nov 03 17:48:32.882167 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.882131    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zqwf\" (UniqueName: \"kubernetes.io/projected/f639120d-e1f4-4700-a328-fec7ee568f3c-kube-api-access-9zqwf\") pod \"maas-api-revoke-sub-095a6192-kc4xb\" (UID: \"f639120d-e1f4-4700-a328-fec7ee568f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb"
Nov 03 17:48:32.923267 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.923237    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:48:32.923267 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.923237    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:48:32.923483 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:48:32.923273    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-8e52e9\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="8aced450-56a7-4d4f-9b3c-e30c0f406bcf" pod="opendatahub/test-curl-7484-8e52e9"
Nov 03 17:48:32.924928 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.924906    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb"]
Nov 03 17:48:32.983262 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.983227    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f639120d-e1f4-4700-a328-fec7ee568f3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-095a6192-kc4xb\" (UID: \"f639120d-e1f4-4700-a328-fec7ee568f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb"
Nov 03 17:48:32.983262 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.983265    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f639120d-e1f4-4700-a328-fec7ee568f3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-095a6192-kc4xb\" (UID: \"f639120d-e1f4-4700-a328-fec7ee568f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb"
Nov 03 17:48:32.983509 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.983334    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zqwf\" (UniqueName: \"kubernetes.io/projected/f639120d-e1f4-4700-a328-fec7ee568f3c-kube-api-access-9zqwf\") pod \"maas-api-revoke-sub-095a6192-kc4xb\" (UID: \"f639120d-e1f4-4700-a328-fec7ee568f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb"
Nov 03 17:48:32.983966 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.983935    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f639120d-e1f4-4700-a328-fec7ee568f3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-095a6192-kc4xb\" (UID: \"f639120d-e1f4-4700-a328-fec7ee568f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb"
Nov 03 17:48:32.985719 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:32.985698    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f639120d-e1f4-4700-a328-fec7ee568f3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-095a6192-kc4xb\" (UID: \"f639120d-e1f4-4700-a328-fec7ee568f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb"
Nov 03 17:48:33.049628 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:33.049591    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zqwf\" (UniqueName: \"kubernetes.io/projected/f639120d-e1f4-4700-a328-fec7ee568f3c-kube-api-access-9zqwf\") pod \"maas-api-revoke-sub-095a6192-kc4xb\" (UID: \"f639120d-e1f4-4700-a328-fec7ee568f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb"
Nov 03 17:48:33.129516 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:33.129427    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb"
Nov 03 17:48:33.336928 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:33.336895    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aced450-56a7-4d4f-9b3c-e30c0f406bcf" path="/var/lib/kubelet/pods/8aced450-56a7-4d4f-9b3c-e30c0f406bcf/volumes"
Nov 03 17:48:33.504984 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:48:33.504948    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf639120d_e1f4_4700_a328_fec7ee568f3c.slice/crio-27013ab8168d255e85b4d5da5c3e5d71907069763820f72473df5ee3af84973d WatchSource:0}: Error finding container 27013ab8168d255e85b4d5da5c3e5d71907069763820f72473df5ee3af84973d: Status 404 returned error can't find the container with id 27013ab8168d255e85b4d5da5c3e5d71907069763820f72473df5ee3af84973d
Nov 03 17:48:33.507465 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:33.507442    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb"]
Nov 03 17:48:34.199048 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:34.199018    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb" event={"ID":"f639120d-e1f4-4700-a328-fec7ee568f3c","Type":"ContainerStarted","Data":"fb34841d783d408f2a835433c534daf991a52cb62bc07b7325064463a7a678ff"}
Nov 03 17:48:34.199172 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:34.199085    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb" event={"ID":"f639120d-e1f4-4700-a328-fec7ee568f3c","Type":"ContainerStarted","Data":"27013ab8168d255e85b4d5da5c3e5d71907069763820f72473df5ee3af84973d"}
Nov 03 17:48:34.321915 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:34.321861    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb" podStartSLOduration=2.321846122 podStartE2EDuration="2.321846122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:48:34.318639397 +0000 UTC m=+1855.116352179" watchObservedRunningTime="2026-11-03 17:48:34.321846122 +0000 UTC m=+1855.119558903"
Nov 03 17:48:34.426752 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:34.426731    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-9837a3"
Nov 03 17:48:34.502719 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:34.502694    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe91006-9ef1-4592-bfa6-937bc4a95a83-kube-api-access-9xqn8\" (UniqueName: \"kubernetes.io/projected/2fe91006-9ef1-4592-bfa6-937bc4a95a83-kube-api-access-9xqn8\") pod \"2fe91006-9ef1-4592-bfa6-937bc4a95a83\" (UID: \"2fe91006-9ef1-4592-bfa6-937bc4a95a83\") "
Nov 03 17:48:34.504595 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:34.504572    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe91006-9ef1-4592-bfa6-937bc4a95a83-kube-api-access-9xqn8" pod "2fe91006-9ef1-4592-bfa6-937bc4a95a83" (UID: "2fe91006-9ef1-4592-bfa6-937bc4a95a83"). InnerVolumeSpecName "kube-api-access-9xqn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:48:34.603831 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:34.603795    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xqn8\" (UniqueName: \"kubernetes.io/projected/2fe91006-9ef1-4592-bfa6-937bc4a95a83-kube-api-access-9xqn8\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:35.207927 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:35.207895    6059 generic.go:356] "Generic (PLEG): container finished" podID="f639120d-e1f4-4700-a328-fec7ee568f3c" containerID="fb34841d783d408f2a835433c534daf991a52cb62bc07b7325064463a7a678ff" exitCode=0
Nov 03 17:48:35.208101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:35.207991    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb" event={"ID":"f639120d-e1f4-4700-a328-fec7ee568f3c","Type":"ContainerDied","Data":"fb34841d783d408f2a835433c534daf991a52cb62bc07b7325064463a7a678ff"}
Nov 03 17:48:35.209344 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:35.209319    6059 generic.go:356] "Generic (PLEG): container finished" podID="2fe91006-9ef1-4592-bfa6-937bc4a95a83" containerID="aea77e024a0fa02c6a297bd0fe23b671b5859c00e58f20710baf10fce831ff46" exitCode=137
Nov 03 17:48:35.209465 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:35.209368    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-9837a3"
Nov 03 17:48:35.209465 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:35.209403    6059 scope.go:122] "RemoveContainer" containerID="aea77e024a0fa02c6a297bd0fe23b671b5859c00e58f20710baf10fce831ff46"
Nov 03 17:48:35.219807 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:35.219792    6059 scope.go:122] "RemoveContainer" containerID="aea77e024a0fa02c6a297bd0fe23b671b5859c00e58f20710baf10fce831ff46"
Nov 03 17:48:35.220035 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:48:35.220011    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aea77e024a0fa02c6a297bd0fe23b671b5859c00e58f20710baf10fce831ff46\": container with ID starting with aea77e024a0fa02c6a297bd0fe23b671b5859c00e58f20710baf10fce831ff46 not found: ID does not exist" containerID="aea77e024a0fa02c6a297bd0fe23b671b5859c00e58f20710baf10fce831ff46"
Nov 03 17:48:35.220080 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:35.220050    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea77e024a0fa02c6a297bd0fe23b671b5859c00e58f20710baf10fce831ff46"} err="failed to get container status \"aea77e024a0fa02c6a297bd0fe23b671b5859c00e58f20710baf10fce831ff46\": rpc error: code = NotFound desc = could not find container \"aea77e024a0fa02c6a297bd0fe23b671b5859c00e58f20710baf10fce831ff46\": container with ID starting with aea77e024a0fa02c6a297bd0fe23b671b5859c00e58f20710baf10fce831ff46 not found: ID does not exist"
Nov 03 17:48:35.337792 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:35.337718    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fe91006-9ef1-4592-bfa6-937bc4a95a83" path="/var/lib/kubelet/pods/2fe91006-9ef1-4592-bfa6-937bc4a95a83/volumes"
Nov 03 17:48:35.382906 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:48:35.382865    6059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7484-9837a3\" is forbidden: User \"system:node:ip-10-0-143-117.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-117.us-west-2.compute.internal' and this object" podUID="2fe91006-9ef1-4592-bfa6-937bc4a95a83" pod="opendatahub/test-curl-7484-9837a3"
Nov 03 17:48:36.454152 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:36.454129    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb"
Nov 03 17:48:36.526976 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:36.526889    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f639120d-e1f4-4700-a328-fec7ee568f3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f639120d-e1f4-4700-a328-fec7ee568f3c-maas-api-xks-ca\") pod \"f639120d-e1f4-4700-a328-fec7ee568f3c\" (UID: \"f639120d-e1f4-4700-a328-fec7ee568f3c\") "
Nov 03 17:48:36.527137 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:36.526984    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f639120d-e1f4-4700-a328-fec7ee568f3c-kube-api-access-9zqwf\" (UniqueName: \"kubernetes.io/projected/f639120d-e1f4-4700-a328-fec7ee568f3c-kube-api-access-9zqwf\") pod \"f639120d-e1f4-4700-a328-fec7ee568f3c\" (UID: \"f639120d-e1f4-4700-a328-fec7ee568f3c\") "
Nov 03 17:48:36.527137 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:36.527022    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f639120d-e1f4-4700-a328-fec7ee568f3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f639120d-e1f4-4700-a328-fec7ee568f3c-maas-api-openshift-ca\") pod \"f639120d-e1f4-4700-a328-fec7ee568f3c\" (UID: \"f639120d-e1f4-4700-a328-fec7ee568f3c\") "
Nov 03 17:48:36.527464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:36.527439    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f639120d-e1f4-4700-a328-fec7ee568f3c-maas-api-openshift-ca" pod "f639120d-e1f4-4700-a328-fec7ee568f3c" (UID: "f639120d-e1f4-4700-a328-fec7ee568f3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:48:36.527578 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:36.527562    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f639120d-e1f4-4700-a328-fec7ee568f3c-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:36.529098 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:36.529069    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f639120d-e1f4-4700-a328-fec7ee568f3c-maas-api-xks-ca" pod "f639120d-e1f4-4700-a328-fec7ee568f3c" (UID: "f639120d-e1f4-4700-a328-fec7ee568f3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:48:36.529217 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:36.529112    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f639120d-e1f4-4700-a328-fec7ee568f3c-kube-api-access-9zqwf" pod "f639120d-e1f4-4700-a328-fec7ee568f3c" (UID: "f639120d-e1f4-4700-a328-fec7ee568f3c"). InnerVolumeSpecName "kube-api-access-9zqwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:48:36.628967 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:36.628931    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f639120d-e1f4-4700-a328-fec7ee568f3c-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:36.628967 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:36.628961    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zqwf\" (UniqueName: \"kubernetes.io/projected/f639120d-e1f4-4700-a328-fec7ee568f3c-kube-api-access-9zqwf\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:36.861778 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:48:36.861692    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:48:37.134627 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:37.134538    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:37.226174 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:37.226140    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb" event={"ID":"f639120d-e1f4-4700-a328-fec7ee568f3c","Type":"ContainerDied","Data":"27013ab8168d255e85b4d5da5c3e5d71907069763820f72473df5ee3af84973d"}
Nov 03 17:48:37.226367 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:37.226177    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb"
Nov 03 17:48:37.226367 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:37.226229    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27013ab8168d255e85b4d5da5c3e5d71907069763820f72473df5ee3af84973d"
Nov 03 17:48:39.364258 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:39.364214    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:39.583501 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:39.583467    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:40.164834 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:40.164794    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p" podUID="fc7d2c53-ef50-455f-9661-296553225d59" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 17:48:40.569137 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:40.569103    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:40.844613 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:40.844524    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:40.868865 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:40.868832    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"]
Nov 03 17:48:40.870197 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:40.870163    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p" podUID="fc7d2c53-ef50-455f-9661-296553225d59" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 17:48:40.920795 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:40.920766    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"]
Nov 03 17:48:41.058763 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:41.058731    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:41.953528 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:41.953490    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5"]
Nov 03 17:48:41.954455 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:41.954435    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f639120d-e1f4-4700-a328-fec7ee568f3c" containerName="revoke-keys"
Nov 03 17:48:41.954513 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:41.954459    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f639120d-e1f4-4700-a328-fec7ee568f3c" containerName="revoke-keys"
Nov 03 17:48:41.954653 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:41.954640    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f639120d-e1f4-4700-a328-fec7ee568f3c" containerName="revoke-keys"
Nov 03 17:48:41.959367 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:41.959331    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5"
Nov 03 17:48:42.016294 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:42.016266    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5"]
Nov 03 17:48:42.063662 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:42.063620    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:48:42.063662 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:42.063662    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:48:42.063931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:42.063776    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:48:42.065214 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:42.065194    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:48:42.105093 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:42.105049    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09273357-8630-4ae8-a45e-d90373bcb506-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7fac61c-8f5s5\" (UID: \"09273357-8630-4ae8-a45e-d90373bcb506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5"
Nov 03 17:48:42.105260 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:42.105159    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27gxb\" (UniqueName: \"kubernetes.io/projected/09273357-8630-4ae8-a45e-d90373bcb506-kube-api-access-27gxb\") pod \"maas-api-revoke-sub-a7fac61c-8f5s5\" (UID: \"09273357-8630-4ae8-a45e-d90373bcb506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5"
Nov 03 17:48:42.105260 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:42.105183    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09273357-8630-4ae8-a45e-d90373bcb506-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7fac61c-8f5s5\" (UID: \"09273357-8630-4ae8-a45e-d90373bcb506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5"
Nov 03 17:48:42.206654 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:42.206624    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09273357-8630-4ae8-a45e-d90373bcb506-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7fac61c-8f5s5\" (UID: \"09273357-8630-4ae8-a45e-d90373bcb506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5"
Nov 03 17:48:42.206824 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:42.206792    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27gxb\" (UniqueName: \"kubernetes.io/projected/09273357-8630-4ae8-a45e-d90373bcb506-kube-api-access-27gxb\") pod \"maas-api-revoke-sub-a7fac61c-8f5s5\" (UID: \"09273357-8630-4ae8-a45e-d90373bcb506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5"
Nov 03 17:48:42.206886 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:42.206837    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09273357-8630-4ae8-a45e-d90373bcb506-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7fac61c-8f5s5\" (UID: \"09273357-8630-4ae8-a45e-d90373bcb506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5"
Nov 03 17:48:42.207515 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:42.207481    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09273357-8630-4ae8-a45e-d90373bcb506-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7fac61c-8f5s5\" (UID: \"09273357-8630-4ae8-a45e-d90373bcb506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5"
Nov 03 17:48:42.209373 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:42.209330    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09273357-8630-4ae8-a45e-d90373bcb506-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7fac61c-8f5s5\" (UID: \"09273357-8630-4ae8-a45e-d90373bcb506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5"
Nov 03 17:48:42.270063 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:42.270034    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27gxb\" (UniqueName: \"kubernetes.io/projected/09273357-8630-4ae8-a45e-d90373bcb506-kube-api-access-27gxb\") pod \"maas-api-revoke-sub-a7fac61c-8f5s5\" (UID: \"09273357-8630-4ae8-a45e-d90373bcb506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5"
Nov 03 17:48:42.282555 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:42.279304    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5"
Nov 03 17:48:42.443741 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:42.443702    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:42.665392 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:42.665332    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5"]
Nov 03 17:48:42.672225 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:48:42.672201    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09273357_8630_4ae8_a45e_d90373bcb506.slice/crio-93cf8156cfda26c83f86a7ba236dd98616e2032d496e86475cd5d243c9671134 WatchSource:0}: Error finding container 93cf8156cfda26c83f86a7ba236dd98616e2032d496e86475cd5d243c9671134: Status 404 returned error can't find the container with id 93cf8156cfda26c83f86a7ba236dd98616e2032d496e86475cd5d243c9671134
Nov 03 17:48:43.310223 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:43.310180    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5" event={"ID":"09273357-8630-4ae8-a45e-d90373bcb506","Type":"ContainerStarted","Data":"c8d24fd9bee0d9ea5ae37594f37e65672741329e05040df561b2bdff04b3f9bc"}
Nov 03 17:48:43.310948 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:43.310918    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5" event={"ID":"09273357-8630-4ae8-a45e-d90373bcb506","Type":"ContainerStarted","Data":"93cf8156cfda26c83f86a7ba236dd98616e2032d496e86475cd5d243c9671134"}
Nov 03 17:48:43.790379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:43.790278    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5" podStartSLOduration=2.790257212 podStartE2EDuration="2.790257212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:48:43.456419466 +0000 UTC m=+1864.254132249" watchObservedRunningTime="2026-11-03 17:48:43.790257212 +0000 UTC m=+1864.587969990"
Nov 03 17:48:43.802619 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:43.802585    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8"]
Nov 03 17:48:43.808455 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:43.808424    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8"
Nov 03 17:48:43.810706 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:43.810465    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c800894-kzjg8\" (UID: \"1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8"
Nov 03 17:48:43.810706 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:43.810519    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vqg9\" (UniqueName: \"kubernetes.io/projected/1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd-kube-api-access-5vqg9\") pod \"maas-api-revoke-sub-5c800894-kzjg8\" (UID: \"1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8"
Nov 03 17:48:43.810706 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:43.810582    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c800894-kzjg8\" (UID: \"1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8"
Nov 03 17:48:43.876390 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:43.875950    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8"]
Nov 03 17:48:43.919413 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:43.917580    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c800894-kzjg8\" (UID: \"1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8"
Nov 03 17:48:43.919413 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:43.917629    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vqg9\" (UniqueName: \"kubernetes.io/projected/1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd-kube-api-access-5vqg9\") pod \"maas-api-revoke-sub-5c800894-kzjg8\" (UID: \"1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8"
Nov 03 17:48:43.919413 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:43.917690    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c800894-kzjg8\" (UID: \"1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8"
Nov 03 17:48:43.919413 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:43.919267    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c800894-kzjg8\" (UID: \"1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8"
Nov 03 17:48:43.923320 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:43.923293    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c800894-kzjg8\" (UID: \"1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8"
Nov 03 17:48:44.006320 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:44.005816    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vqg9\" (UniqueName: \"kubernetes.io/projected/1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd-kube-api-access-5vqg9\") pod \"maas-api-revoke-sub-5c800894-kzjg8\" (UID: \"1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8"
Nov 03 17:48:44.126090 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:44.126004    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8"
Nov 03 17:48:44.324705 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:44.324675    6059 generic.go:356] "Generic (PLEG): container finished" podID="09273357-8630-4ae8-a45e-d90373bcb506" containerID="c8d24fd9bee0d9ea5ae37594f37e65672741329e05040df561b2bdff04b3f9bc" exitCode=0
Nov 03 17:48:44.325947 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:44.325346    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5" event={"ID":"09273357-8630-4ae8-a45e-d90373bcb506","Type":"ContainerDied","Data":"c8d24fd9bee0d9ea5ae37594f37e65672741329e05040df561b2bdff04b3f9bc"}
Nov 03 17:48:44.611413 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:44.602697    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8"]
Nov 03 17:48:44.624581 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:48:44.624533    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1594d9c1_d6b7_44b0_9ef6_d87fd83e36dd.slice/crio-69969b8e9ea7ce04d067bd6befb0307fe509187aec8cb89345aa32e86ae5b058 WatchSource:0}: Error finding container 69969b8e9ea7ce04d067bd6befb0307fe509187aec8cb89345aa32e86ae5b058: Status 404 returned error can't find the container with id 69969b8e9ea7ce04d067bd6befb0307fe509187aec8cb89345aa32e86ae5b058
Nov 03 17:48:45.345149 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:45.345119    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-h98kq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:48:45.345751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:45.345703    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq" podUID="1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:48:45.350751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:45.350517    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-tdr2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:48:45.350751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:45.350571    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c" podUID="4f0c42da-8020-48d1-9e61-7dfab0d4a65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:48:45.376524 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:45.376476    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8" event={"ID":"1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd","Type":"ContainerStarted","Data":"1aabe24a5c39a41ed6ce88c92445548236df1d06ac1e7dc173f48ec49ae18f68"}
Nov 03 17:48:45.378061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:45.378032    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8" event={"ID":"1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd","Type":"ContainerStarted","Data":"69969b8e9ea7ce04d067bd6befb0307fe509187aec8cb89345aa32e86ae5b058"}
Nov 03 17:48:45.519672 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:45.519599    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8" podStartSLOduration=2.5195780819999998 podStartE2EDuration="2.519578082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:48:45.471024939 +0000 UTC m=+1866.268737726" watchObservedRunningTime="2026-11-03 17:48:45.519578082 +0000 UTC m=+1866.317290864"
Nov 03 17:48:45.522560 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:45.522528    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"]
Nov 03 17:48:45.573152 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:45.573124    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5"
Nov 03 17:48:45.661785 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:45.661699    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09273357-8630-4ae8-a45e-d90373bcb506-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09273357-8630-4ae8-a45e-d90373bcb506-maas-api-openshift-ca\") pod \"09273357-8630-4ae8-a45e-d90373bcb506\" (UID: \"09273357-8630-4ae8-a45e-d90373bcb506\") "
Nov 03 17:48:45.661785 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:45.661743    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09273357-8630-4ae8-a45e-d90373bcb506-kube-api-access-27gxb\" (UniqueName: \"kubernetes.io/projected/09273357-8630-4ae8-a45e-d90373bcb506-kube-api-access-27gxb\") pod \"09273357-8630-4ae8-a45e-d90373bcb506\" (UID: \"09273357-8630-4ae8-a45e-d90373bcb506\") "
Nov 03 17:48:45.662043 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:45.661921    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09273357-8630-4ae8-a45e-d90373bcb506-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09273357-8630-4ae8-a45e-d90373bcb506-maas-api-xks-ca\") pod \"09273357-8630-4ae8-a45e-d90373bcb506\" (UID: \"09273357-8630-4ae8-a45e-d90373bcb506\") "
Nov 03 17:48:45.662154 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:45.662131    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09273357-8630-4ae8-a45e-d90373bcb506-maas-api-openshift-ca" pod "09273357-8630-4ae8-a45e-d90373bcb506" (UID: "09273357-8630-4ae8-a45e-d90373bcb506"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:48:45.662478 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:45.662462    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09273357-8630-4ae8-a45e-d90373bcb506-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:45.663945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:45.663916    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09273357-8630-4ae8-a45e-d90373bcb506-maas-api-xks-ca" pod "09273357-8630-4ae8-a45e-d90373bcb506" (UID: "09273357-8630-4ae8-a45e-d90373bcb506"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:48:45.663945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:45.663923    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09273357-8630-4ae8-a45e-d90373bcb506-kube-api-access-27gxb" pod "09273357-8630-4ae8-a45e-d90373bcb506" (UID: "09273357-8630-4ae8-a45e-d90373bcb506"). InnerVolumeSpecName "kube-api-access-27gxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:48:45.684569 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:45.684526    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:45.763768 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:45.763725    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27gxb\" (UniqueName: \"kubernetes.io/projected/09273357-8630-4ae8-a45e-d90373bcb506-kube-api-access-27gxb\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:45.763768 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:45.763761    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09273357-8630-4ae8-a45e-d90373bcb506-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:45.870135 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:45.870093    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p" podUID="fc7d2c53-ef50-455f-9661-296553225d59" containerName="payload-processing" containerID="cri-o://08806ec4be8d786a9d89f93dc9fecaa80c2209e266f3edee8b70641059332d59" gracePeriod=25
Nov 03 17:48:45.921868 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:45.921786    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw" podUID="ebefb030-567e-47f1-bb77-f62b222d5280" containerName="payload-processing" containerID="cri-o://7d193d49070c96a5bd0dcee8f5d0f38ca04f9df6f4d34a6fb30ee60cb36b3bf6" gracePeriod=25
Nov 03 17:48:46.197273 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.197244    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"
Nov 03 17:48:46.211172 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.211142    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"
Nov 03 17:48:46.276524 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.276486    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc7d2c53-ef50-455f-9661-296553225d59-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc7d2c53-ef50-455f-9661-296553225d59-provider-credentials\") pod \"fc7d2c53-ef50-455f-9661-296553225d59\" (UID: \"fc7d2c53-ef50-455f-9661-296553225d59\") "
Nov 03 17:48:46.276675 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.276607    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebefb030-567e-47f1-bb77-f62b222d5280-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ebefb030-567e-47f1-bb77-f62b222d5280-provider-credentials\") pod \"ebefb030-567e-47f1-bb77-f62b222d5280\" (UID: \"ebefb030-567e-47f1-bb77-f62b222d5280\") "
Nov 03 17:48:46.276675 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.276652    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc7d2c53-ef50-455f-9661-296553225d59-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc7d2c53-ef50-455f-9661-296553225d59-routing-overlay-volume\") pod \"fc7d2c53-ef50-455f-9661-296553225d59\" (UID: \"fc7d2c53-ef50-455f-9661-296553225d59\") "
Nov 03 17:48:46.276742 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.276695    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebefb030-567e-47f1-bb77-f62b222d5280-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebefb030-567e-47f1-bb77-f62b222d5280-plugins-config-volume\") pod \"ebefb030-567e-47f1-bb77-f62b222d5280\" (UID: \"ebefb030-567e-47f1-bb77-f62b222d5280\") "
Nov 03 17:48:46.276794 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.276768    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebefb030-567e-47f1-bb77-f62b222d5280-kube-api-access-82qkm\" (UniqueName: \"kubernetes.io/projected/ebefb030-567e-47f1-bb77-f62b222d5280-kube-api-access-82qkm\") pod \"ebefb030-567e-47f1-bb77-f62b222d5280\" (UID: \"ebefb030-567e-47f1-bb77-f62b222d5280\") "
Nov 03 17:48:46.276840 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.276799    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebefb030-567e-47f1-bb77-f62b222d5280-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ebefb030-567e-47f1-bb77-f62b222d5280-routing-overlay-volume\") pod \"ebefb030-567e-47f1-bb77-f62b222d5280\" (UID: \"ebefb030-567e-47f1-bb77-f62b222d5280\") "
Nov 03 17:48:46.276897 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.276869    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc7d2c53-ef50-455f-9661-296553225d59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7d2c53-ef50-455f-9661-296553225d59-plugins-config-volume\") pod \"fc7d2c53-ef50-455f-9661-296553225d59\" (UID: \"fc7d2c53-ef50-455f-9661-296553225d59\") "
Nov 03 17:48:46.276984 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.276948    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc7d2c53-ef50-455f-9661-296553225d59-kube-api-access-8nmq4\" (UniqueName: \"kubernetes.io/projected/fc7d2c53-ef50-455f-9661-296553225d59-kube-api-access-8nmq4\") pod \"fc7d2c53-ef50-455f-9661-296553225d59\" (UID: \"fc7d2c53-ef50-455f-9661-296553225d59\") "
Nov 03 17:48:46.277174 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.277129    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc7d2c53-ef50-455f-9661-296553225d59-routing-overlay-volume" pod "fc7d2c53-ef50-455f-9661-296553225d59" (UID: "fc7d2c53-ef50-455f-9661-296553225d59"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:48:46.277280 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.277204    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebefb030-567e-47f1-bb77-f62b222d5280-plugins-config-volume" pod "ebefb030-567e-47f1-bb77-f62b222d5280" (UID: "ebefb030-567e-47f1-bb77-f62b222d5280"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:48:46.277280 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.277214    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebefb030-567e-47f1-bb77-f62b222d5280-routing-overlay-volume" pod "ebefb030-567e-47f1-bb77-f62b222d5280" (UID: "ebefb030-567e-47f1-bb77-f62b222d5280"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:48:46.277575 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.277547    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc7d2c53-ef50-455f-9661-296553225d59-plugins-config-volume" pod "fc7d2c53-ef50-455f-9661-296553225d59" (UID: "fc7d2c53-ef50-455f-9661-296553225d59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:48:46.277575 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.277566    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc7d2c53-ef50-455f-9661-296553225d59-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:46.277726 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.277590    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebefb030-567e-47f1-bb77-f62b222d5280-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:46.277726 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.277607    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ebefb030-567e-47f1-bb77-f62b222d5280-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:46.278880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.278857    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7d2c53-ef50-455f-9661-296553225d59-provider-credentials" pod "fc7d2c53-ef50-455f-9661-296553225d59" (UID: "fc7d2c53-ef50-455f-9661-296553225d59"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:48:46.279017 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.278995    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebefb030-567e-47f1-bb77-f62b222d5280-provider-credentials" pod "ebefb030-567e-47f1-bb77-f62b222d5280" (UID: "ebefb030-567e-47f1-bb77-f62b222d5280"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:48:46.279324 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.279298    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebefb030-567e-47f1-bb77-f62b222d5280-kube-api-access-82qkm" pod "ebefb030-567e-47f1-bb77-f62b222d5280" (UID: "ebefb030-567e-47f1-bb77-f62b222d5280"). InnerVolumeSpecName "kube-api-access-82qkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:48:46.279464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.279337    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7d2c53-ef50-455f-9661-296553225d59-kube-api-access-8nmq4" pod "fc7d2c53-ef50-455f-9661-296553225d59" (UID: "fc7d2c53-ef50-455f-9661-296553225d59"). InnerVolumeSpecName "kube-api-access-8nmq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:48:46.352281 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.352238    6059 generic.go:356] "Generic (PLEG): container finished" podID="fc7d2c53-ef50-455f-9661-296553225d59" containerID="08806ec4be8d786a9d89f93dc9fecaa80c2209e266f3edee8b70641059332d59" exitCode=0
Nov 03 17:48:46.352791 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.352289    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"
Nov 03 17:48:46.352791 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.352323    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p" event={"ID":"fc7d2c53-ef50-455f-9661-296553225d59","Type":"ContainerDied","Data":"08806ec4be8d786a9d89f93dc9fecaa80c2209e266f3edee8b70641059332d59"}
Nov 03 17:48:46.352791 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.352405    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68478f785f-pxl4p" event={"ID":"fc7d2c53-ef50-455f-9661-296553225d59","Type":"ContainerDied","Data":"378d867ce7a89b0698f201f267c8459119934bf07e3ee96dc1b52ca0d9be912a"}
Nov 03 17:48:46.352791 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.352440    6059 scope.go:122] "RemoveContainer" containerID="08806ec4be8d786a9d89f93dc9fecaa80c2209e266f3edee8b70641059332d59"
Nov 03 17:48:46.354554 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.354526    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5" event={"ID":"09273357-8630-4ae8-a45e-d90373bcb506","Type":"ContainerDied","Data":"93cf8156cfda26c83f86a7ba236dd98616e2032d496e86475cd5d243c9671134"}
Nov 03 17:48:46.354677 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.354583    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93cf8156cfda26c83f86a7ba236dd98616e2032d496e86475cd5d243c9671134"
Nov 03 17:48:46.354677 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.354521    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5"
Nov 03 17:48:46.356584 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.356560    6059 generic.go:356] "Generic (PLEG): container finished" podID="1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd" containerID="1aabe24a5c39a41ed6ce88c92445548236df1d06ac1e7dc173f48ec49ae18f68" exitCode=0
Nov 03 17:48:46.356699 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.356649    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8" event={"ID":"1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd","Type":"ContainerDied","Data":"1aabe24a5c39a41ed6ce88c92445548236df1d06ac1e7dc173f48ec49ae18f68"}
Nov 03 17:48:46.358875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.358320    6059 generic.go:356] "Generic (PLEG): container finished" podID="ebefb030-567e-47f1-bb77-f62b222d5280" containerID="7d193d49070c96a5bd0dcee8f5d0f38ca04f9df6f4d34a6fb30ee60cb36b3bf6" exitCode=0
Nov 03 17:48:46.358875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.358445    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw" event={"ID":"ebefb030-567e-47f1-bb77-f62b222d5280","Type":"ContainerDied","Data":"7d193d49070c96a5bd0dcee8f5d0f38ca04f9df6f4d34a6fb30ee60cb36b3bf6"}
Nov 03 17:48:46.358875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.358484    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw" event={"ID":"ebefb030-567e-47f1-bb77-f62b222d5280","Type":"ContainerDied","Data":"902fab72ac2ae16614b81f2d06ccdf86891b955f70f20e57e414553a741590ac"}
Nov 03 17:48:46.358875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.358503    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"
Nov 03 17:48:46.379628 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.379598    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7d2c53-ef50-455f-9661-296553225d59-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:46.379766 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.379638    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nmq4\" (UniqueName: \"kubernetes.io/projected/fc7d2c53-ef50-455f-9661-296553225d59-kube-api-access-8nmq4\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:46.379766 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.379663    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc7d2c53-ef50-455f-9661-296553225d59-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:46.379766 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.379682    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ebefb030-567e-47f1-bb77-f62b222d5280-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:46.379766 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.379701    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82qkm\" (UniqueName: \"kubernetes.io/projected/ebefb030-567e-47f1-bb77-f62b222d5280-kube-api-access-82qkm\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:46.379997 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.379823    6059 scope.go:122] "RemoveContainer" containerID="08806ec4be8d786a9d89f93dc9fecaa80c2209e266f3edee8b70641059332d59"
Nov 03 17:48:46.381072 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:48:46.381045    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08806ec4be8d786a9d89f93dc9fecaa80c2209e266f3edee8b70641059332d59\": container with ID starting with 08806ec4be8d786a9d89f93dc9fecaa80c2209e266f3edee8b70641059332d59 not found: ID does not exist" containerID="08806ec4be8d786a9d89f93dc9fecaa80c2209e266f3edee8b70641059332d59"
Nov 03 17:48:46.381167 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.381081    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08806ec4be8d786a9d89f93dc9fecaa80c2209e266f3edee8b70641059332d59"} err="failed to get container status \"08806ec4be8d786a9d89f93dc9fecaa80c2209e266f3edee8b70641059332d59\": rpc error: code = NotFound desc = could not find container \"08806ec4be8d786a9d89f93dc9fecaa80c2209e266f3edee8b70641059332d59\": container with ID starting with 08806ec4be8d786a9d89f93dc9fecaa80c2209e266f3edee8b70641059332d59 not found: ID does not exist"
Nov 03 17:48:46.381167 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.381101    6059 scope.go:122] "RemoveContainer" containerID="7d193d49070c96a5bd0dcee8f5d0f38ca04f9df6f4d34a6fb30ee60cb36b3bf6"
Nov 03 17:48:46.390481 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.390463    6059 scope.go:122] "RemoveContainer" containerID="7d193d49070c96a5bd0dcee8f5d0f38ca04f9df6f4d34a6fb30ee60cb36b3bf6"
Nov 03 17:48:46.390774 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:48:46.390753    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d193d49070c96a5bd0dcee8f5d0f38ca04f9df6f4d34a6fb30ee60cb36b3bf6\": container with ID starting with 7d193d49070c96a5bd0dcee8f5d0f38ca04f9df6f4d34a6fb30ee60cb36b3bf6 not found: ID does not exist" containerID="7d193d49070c96a5bd0dcee8f5d0f38ca04f9df6f4d34a6fb30ee60cb36b3bf6"
Nov 03 17:48:46.390825 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.390782    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d193d49070c96a5bd0dcee8f5d0f38ca04f9df6f4d34a6fb30ee60cb36b3bf6"} err="failed to get container status \"7d193d49070c96a5bd0dcee8f5d0f38ca04f9df6f4d34a6fb30ee60cb36b3bf6\": rpc error: code = NotFound desc = could not find container \"7d193d49070c96a5bd0dcee8f5d0f38ca04f9df6f4d34a6fb30ee60cb36b3bf6\": container with ID starting with 7d193d49070c96a5bd0dcee8f5d0f38ca04f9df6f4d34a6fb30ee60cb36b3bf6 not found: ID does not exist"
Nov 03 17:48:46.558942 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.558914    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"]
Nov 03 17:48:46.563944 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.563903    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-86cbdddf86-2k5kw"]
Nov 03 17:48:46.677701 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.677665    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"]
Nov 03 17:48:46.683446 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:46.683422    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-68478f785f-pxl4p"]
Nov 03 17:48:47.173704 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.173668    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"]
Nov 03 17:48:47.174288 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.174273    6059 cpu_manager.go:414] "containerMap: removing container" podUID="09273357-8630-4ae8-a45e-d90373bcb506" containerName="revoke-keys"
Nov 03 17:48:47.174335 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.174290    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09273357-8630-4ae8-a45e-d90373bcb506" containerName="revoke-keys"
Nov 03 17:48:47.174335 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.174331    6059 cpu_manager.go:414] "containerMap: removing container" podUID="ebefb030-567e-47f1-bb77-f62b222d5280" containerName="payload-processing"
Nov 03 17:48:47.174413 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.174336    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebefb030-567e-47f1-bb77-f62b222d5280" containerName="payload-processing"
Nov 03 17:48:47.174413 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.174346    6059 cpu_manager.go:414] "containerMap: removing container" podUID="fc7d2c53-ef50-455f-9661-296553225d59" containerName="payload-processing"
Nov 03 17:48:47.174413 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.174351    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc7d2c53-ef50-455f-9661-296553225d59" containerName="payload-processing"
Nov 03 17:48:47.174508 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.174485    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebefb030-567e-47f1-bb77-f62b222d5280" containerName="payload-processing"
Nov 03 17:48:47.174508 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.174495    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc7d2c53-ef50-455f-9661-296553225d59" containerName="payload-processing"
Nov 03 17:48:47.174508 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.174505    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="09273357-8630-4ae8-a45e-d90373bcb506" containerName="revoke-keys"
Nov 03 17:48:47.179340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.179322    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.243537 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.242831    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"]
Nov 03 17:48:47.291696 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.291650    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk2t7\" (UniqueName: \"kubernetes.io/projected/4983714f-6635-4557-9e25-80cf0c914d9d-kube-api-access-lk2t7\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.292033 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.291998    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4983714f-6635-4557-9e25-80cf0c914d9d-istio-token\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.292200 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.292180    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4983714f-6635-4557-9e25-80cf0c914d9d-istio-podinfo\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.292308 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.292223    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-workload-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.292308 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.292255    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-credential-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.292454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.292392    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-istio-data\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.292454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.292444    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4983714f-6635-4557-9e25-80cf0c914d9d-istiod-ca-cert\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.292575 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.292501    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-workload-certs\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.292575 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.292525    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-istio-envoy\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.336620 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.336589    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebefb030-567e-47f1-bb77-f62b222d5280" path="/var/lib/kubelet/pods/ebefb030-567e-47f1-bb77-f62b222d5280/volumes"
Nov 03 17:48:47.337019 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.337006    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc7d2c53-ef50-455f-9661-296553225d59" path="/var/lib/kubelet/pods/fc7d2c53-ef50-455f-9661-296553225d59/volumes"
Nov 03 17:48:47.394274 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.394237    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4983714f-6635-4557-9e25-80cf0c914d9d-istio-token\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.394794 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.394338    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4983714f-6635-4557-9e25-80cf0c914d9d-istio-podinfo\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.394794 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.394381    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-workload-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.394794 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.394407    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-credential-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.394794 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.394429    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-istio-data\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.394794 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.394474    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4983714f-6635-4557-9e25-80cf0c914d9d-istiod-ca-cert\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.394794 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.394544    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-workload-certs\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.394794 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.394578    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-istio-envoy\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.394794 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.394618    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk2t7\" (UniqueName: \"kubernetes.io/projected/4983714f-6635-4557-9e25-80cf0c914d9d-kube-api-access-lk2t7\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.395331 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.394859    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-credential-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.395331 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.394894    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-workload-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.395331 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.395135    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-workload-certs\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.395331 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.395233    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-istio-data\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.395331 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.395305    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4983714f-6635-4557-9e25-80cf0c914d9d-istiod-ca-cert\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.397013 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.396988    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-istio-envoy\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.397143 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.397030    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4983714f-6635-4557-9e25-80cf0c914d9d-istio-podinfo\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.457991 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.457960    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4983714f-6635-4557-9e25-80cf0c914d9d-istio-token\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.458160 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.458009    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk2t7\" (UniqueName: \"kubernetes.io/projected/4983714f-6635-4557-9e25-80cf0c914d9d-kube-api-access-lk2t7\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.495596 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.495560    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:47.505113 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.505084    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8"
Nov 03 17:48:47.596591 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.596554    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd-maas-api-xks-ca\") pod \"1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd\" (UID: \"1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd\") "
Nov 03 17:48:47.596776 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.596616    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd-kube-api-access-5vqg9\" (UniqueName: \"kubernetes.io/projected/1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd-kube-api-access-5vqg9\") pod \"1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd\" (UID: \"1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd\") "
Nov 03 17:48:47.596776 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.596708    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd-maas-api-openshift-ca\") pod \"1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd\" (UID: \"1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd\") "
Nov 03 17:48:47.597688 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.597660    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd-maas-api-openshift-ca" pod "1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd" (UID: "1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:48:47.599576 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.599553    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd-maas-api-xks-ca" pod "1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd" (UID: "1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:48:47.600071 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.600052    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd-kube-api-access-5vqg9" pod "1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd" (UID: "1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd"). InnerVolumeSpecName "kube-api-access-5vqg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:48:47.698864 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.698776    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:47.698864 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.698809    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vqg9\" (UniqueName: \"kubernetes.io/projected/1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd-kube-api-access-5vqg9\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:47.698864 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.698819    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:47.897223 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.897188    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:47.931154 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.931099    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"]
Nov 03 17:48:47.935488 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.935444    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:48:47.935630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.935527    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:48:47.935630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:47.935573    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:48:48.379902 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:48.379820    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8"
Nov 03 17:48:48.380060 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:48.379819    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8" event={"ID":"1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd","Type":"ContainerDied","Data":"69969b8e9ea7ce04d067bd6befb0307fe509187aec8cb89345aa32e86ae5b058"}
Nov 03 17:48:48.380060 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:48.379960    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69969b8e9ea7ce04d067bd6befb0307fe509187aec8cb89345aa32e86ae5b058"
Nov 03 17:48:48.381829 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:48.381800    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn" event={"ID":"4983714f-6635-4557-9e25-80cf0c914d9d","Type":"ContainerStarted","Data":"96f71e58b485513665faa6f2e6feb84f72ca5c305dc7592ecd68f1d5b8998e43"}
Nov 03 17:48:48.381942 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:48.381847    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn" event={"ID":"4983714f-6635-4557-9e25-80cf0c914d9d","Type":"ContainerStarted","Data":"f1c6b0183901434531802d1a87e6fac1ec1a98b4401bf8788c807f61fcf4db9c"}
Nov 03 17:48:48.496624 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:48.496596    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:48.508101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:48.508056    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn" podStartSLOduration=1.508044966 podStartE2EDuration="1.508044966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:48:48.50613964 +0000 UTC m=+1869.303852421" watchObservedRunningTime="2026-11-03 17:48:48.508044966 +0000 UTC m=+1869.305757741"
Nov 03 17:48:49.497223 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:49.497190    6059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 17:48:49.497613 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:49.497239    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn" podUID="4983714f-6635-4557-9e25-80cf0c914d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 17:48:50.496547 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:50.496513    6059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 17:48:50.496740 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:50.496569    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn" podUID="4983714f-6635-4557-9e25-80cf0c914d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 17:48:51.496589 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:51.496553    6059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 17:48:51.497097 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:51.496613    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn" podUID="4983714f-6635-4557-9e25-80cf0c914d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 17:48:52.496334 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:52.496281    6059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 17:48:52.496608 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:52.496339    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn" podUID="4983714f-6635-4557-9e25-80cf0c914d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 17:48:52.499588 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:52.499454    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:52.687252 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:52.687222    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:53.496228 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:53.496189    6059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 17:48:53.496413 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:53.496256    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn" podUID="4983714f-6635-4557-9e25-80cf0c914d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 17:48:54.125517 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:54.125478    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l"]
Nov 03 17:48:54.235144 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:54.235113    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8"]
Nov 03 17:48:54.498487 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:54.498453    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:54.682317 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:54.682283    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:54.744926 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:54.744889    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:54.746159 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:54.746135    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:54.747132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:54.747108    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:48:55.523388 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:55.516726    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9becb93d-f24kn"]
Nov 03 17:48:55.523388 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:55.517669    6059 cpu_manager.go:414] "containerMap: removing container" podUID="1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd" containerName="revoke-keys"
Nov 03 17:48:55.523388 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:55.517688    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd" containerName="revoke-keys"
Nov 03 17:48:55.523388 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:55.517814    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd" containerName="revoke-keys"
Nov 03 17:48:55.539501 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:55.536734    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9becb93d-f24kn"
Nov 03 17:48:55.589672 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:55.589505    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9becb93d-f24kn"]
Nov 03 17:48:55.592208 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:55.592174    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:48:55.592368 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:55.592229    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:48:55.592708 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:55.592690    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:48:55.592819 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:55.592798    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:48:55.630872 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:55.630837    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a601f158-37db-48fb-936d-e3c4f9ce235f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9becb93d-f24kn\" (UID: \"a601f158-37db-48fb-936d-e3c4f9ce235f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9becb93d-f24kn"
Nov 03 17:48:55.631041 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:55.631016    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a601f158-37db-48fb-936d-e3c4f9ce235f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9becb93d-f24kn\" (UID: \"a601f158-37db-48fb-936d-e3c4f9ce235f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9becb93d-f24kn"
Nov 03 17:48:55.631150 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:55.631134    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njwfh\" (UniqueName: \"kubernetes.io/projected/a601f158-37db-48fb-936d-e3c4f9ce235f-kube-api-access-njwfh\") pod \"maas-api-revoke-sub-9becb93d-f24kn\" (UID: \"a601f158-37db-48fb-936d-e3c4f9ce235f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9becb93d-f24kn"
Nov 03 17:48:55.732913 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:55.732874    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a601f158-37db-48fb-936d-e3c4f9ce235f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9becb93d-f24kn\" (UID: \"a601f158-37db-48fb-936d-e3c4f9ce235f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9becb93d-f24kn"
Nov 03 17:48:55.733076 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:55.732968    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njwfh\" (UniqueName: \"kubernetes.io/projected/a601f158-37db-48fb-936d-e3c4f9ce235f-kube-api-access-njwfh\") pod \"maas-api-revoke-sub-9becb93d-f24kn\" (UID: \"a601f158-37db-48fb-936d-e3c4f9ce235f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9becb93d-f24kn"
Nov 03 17:48:55.733161 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:55.733073    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a601f158-37db-48fb-936d-e3c4f9ce235f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9becb93d-f24kn\" (UID: \"a601f158-37db-48fb-936d-e3c4f9ce235f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9becb93d-f24kn"
Nov 03 17:48:55.733586 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:55.733560    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a601f158-37db-48fb-936d-e3c4f9ce235f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9becb93d-f24kn\" (UID: \"a601f158-37db-48fb-936d-e3c4f9ce235f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9becb93d-f24kn"
Nov 03 17:48:55.735552 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:55.735524    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a601f158-37db-48fb-936d-e3c4f9ce235f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9becb93d-f24kn\" (UID: \"a601f158-37db-48fb-936d-e3c4f9ce235f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9becb93d-f24kn"
Nov 03 17:48:55.796513 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:55.796442    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njwfh\" (UniqueName: \"kubernetes.io/projected/a601f158-37db-48fb-936d-e3c4f9ce235f-kube-api-access-njwfh\") pod \"maas-api-revoke-sub-9becb93d-f24kn\" (UID: \"a601f158-37db-48fb-936d-e3c4f9ce235f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9becb93d-f24kn"
Nov 03 17:48:55.870625 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:55.870585    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9becb93d-f24kn"
Nov 03 17:48:56.052451 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:56.052351    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa5e2fd-pjg2p"]
Nov 03 17:48:56.055440 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:56.055418    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa5e2fd-pjg2p"]
Nov 03 17:48:56.253647 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:48:56.253611    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda601f158_37db_48fb_936d_e3c4f9ce235f.slice/crio-e45e78655eab160691663f3e0039dd902b386ca4760047766bd6c294291a557a WatchSource:0}: Error finding container e45e78655eab160691663f3e0039dd902b386ca4760047766bd6c294291a557a: Status 404 returned error can't find the container with id e45e78655eab160691663f3e0039dd902b386ca4760047766bd6c294291a557a
Nov 03 17:48:56.256080 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:56.256057    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9becb93d-f24kn"]
Nov 03 17:48:56.451006 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:56.450976    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9becb93d-f24kn" event={"ID":"a601f158-37db-48fb-936d-e3c4f9ce235f","Type":"ContainerStarted","Data":"e45e78655eab160691663f3e0039dd902b386ca4760047766bd6c294291a557a"}
Nov 03 17:48:57.231328 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:57.231284    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:57.337157 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:57.337122    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4789e44b-2cec-46b9-95fe-68fab84f0cfb" path="/var/lib/kubelet/pods/4789e44b-2cec-46b9-95fe-68fab84f0cfb/volumes"
Nov 03 17:48:57.460447 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:57.460414    6059 generic.go:356] "Generic (PLEG): container finished" podID="a601f158-37db-48fb-936d-e3c4f9ce235f" containerID="8bd87385e6dca90f145dd684e5c11489d790a25997ab7cfb2338d9f7dc723c04" exitCode=0
Nov 03 17:48:57.460614 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:57.460492    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9becb93d-f24kn" event={"ID":"a601f158-37db-48fb-936d-e3c4f9ce235f","Type":"ContainerDied","Data":"8bd87385e6dca90f145dd684e5c11489d790a25997ab7cfb2338d9f7dc723c04"}
Nov 03 17:48:58.614267 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:58.614237    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9becb93d-f24kn"
Nov 03 17:48:58.775186 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:58.775095    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a601f158-37db-48fb-936d-e3c4f9ce235f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a601f158-37db-48fb-936d-e3c4f9ce235f-maas-api-xks-ca\") pod \"a601f158-37db-48fb-936d-e3c4f9ce235f\" (UID: \"a601f158-37db-48fb-936d-e3c4f9ce235f\") "
Nov 03 17:48:58.775346 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:58.775248    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a601f158-37db-48fb-936d-e3c4f9ce235f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a601f158-37db-48fb-936d-e3c4f9ce235f-maas-api-openshift-ca\") pod \"a601f158-37db-48fb-936d-e3c4f9ce235f\" (UID: \"a601f158-37db-48fb-936d-e3c4f9ce235f\") "
Nov 03 17:48:58.775346 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:58.775330    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a601f158-37db-48fb-936d-e3c4f9ce235f-kube-api-access-njwfh\" (UniqueName: \"kubernetes.io/projected/a601f158-37db-48fb-936d-e3c4f9ce235f-kube-api-access-njwfh\") pod \"a601f158-37db-48fb-936d-e3c4f9ce235f\" (UID: \"a601f158-37db-48fb-936d-e3c4f9ce235f\") "
Nov 03 17:48:58.775595 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:58.775569    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a601f158-37db-48fb-936d-e3c4f9ce235f-maas-api-openshift-ca" pod "a601f158-37db-48fb-936d-e3c4f9ce235f" (UID: "a601f158-37db-48fb-936d-e3c4f9ce235f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:48:58.776138 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:58.776056    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a601f158-37db-48fb-936d-e3c4f9ce235f-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:58.777437 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:58.777410    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a601f158-37db-48fb-936d-e3c4f9ce235f-maas-api-xks-ca" pod "a601f158-37db-48fb-936d-e3c4f9ce235f" (UID: "a601f158-37db-48fb-936d-e3c4f9ce235f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:48:58.777531 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:58.777485    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a601f158-37db-48fb-936d-e3c4f9ce235f-kube-api-access-njwfh" pod "a601f158-37db-48fb-936d-e3c4f9ce235f" (UID: "a601f158-37db-48fb-936d-e3c4f9ce235f"). InnerVolumeSpecName "kube-api-access-njwfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:48:58.877873 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:58.877841    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njwfh\" (UniqueName: \"kubernetes.io/projected/a601f158-37db-48fb-936d-e3c4f9ce235f-kube-api-access-njwfh\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:58.877873 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:58.877869    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a601f158-37db-48fb-936d-e3c4f9ce235f-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:59.125982 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.125878    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l" podUID="77951355-6d25-4504-9a27-db09e1d94236" containerName="payload-processing" containerID="cri-o://d161997d4c5d3f12ccde9dd66617e969343c1ce0480b90187b5b5f3a561ef0e8" gracePeriod=25
Nov 03 17:48:59.235576 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.235532    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8" podUID="50c1cc3e-ffc9-4b51-9a95-5677f4898c02" containerName="payload-pre-processing" containerID="cri-o://ef1a4b620bb8dbebbbf7f2b275aafea2fad5c5118a097d4c0554d631a389c5ab" gracePeriod=25
Nov 03 17:48:59.382407 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.382333    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l"
Nov 03 17:48:59.409873 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.409797    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:48:59.477176 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.477135    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9becb93d-f24kn"
Nov 03 17:48:59.477176 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.477158    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9becb93d-f24kn" event={"ID":"a601f158-37db-48fb-936d-e3c4f9ce235f","Type":"ContainerDied","Data":"e45e78655eab160691663f3e0039dd902b386ca4760047766bd6c294291a557a"}
Nov 03 17:48:59.477448 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.477211    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e45e78655eab160691663f3e0039dd902b386ca4760047766bd6c294291a557a"
Nov 03 17:48:59.479040 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.479013    6059 generic.go:356] "Generic (PLEG): container finished" podID="50c1cc3e-ffc9-4b51-9a95-5677f4898c02" containerID="ef1a4b620bb8dbebbbf7f2b275aafea2fad5c5118a097d4c0554d631a389c5ab" exitCode=0
Nov 03 17:48:59.479129 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.479090    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8" event={"ID":"50c1cc3e-ffc9-4b51-9a95-5677f4898c02","Type":"ContainerDied","Data":"ef1a4b620bb8dbebbbf7f2b275aafea2fad5c5118a097d4c0554d631a389c5ab"}
Nov 03 17:48:59.480721 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.480702    6059 generic.go:356] "Generic (PLEG): container finished" podID="77951355-6d25-4504-9a27-db09e1d94236" containerID="d161997d4c5d3f12ccde9dd66617e969343c1ce0480b90187b5b5f3a561ef0e8" exitCode=0
Nov 03 17:48:59.480819 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.480741    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l"
Nov 03 17:48:59.480819 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.480781    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l" event={"ID":"77951355-6d25-4504-9a27-db09e1d94236","Type":"ContainerDied","Data":"d161997d4c5d3f12ccde9dd66617e969343c1ce0480b90187b5b5f3a561ef0e8"}
Nov 03 17:48:59.480897 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.480829    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l" event={"ID":"77951355-6d25-4504-9a27-db09e1d94236","Type":"ContainerDied","Data":"25cdf610eed11b95faabb4fbeda5e285e81b7c31693409451d5bcbb35de1f71a"}
Nov 03 17:48:59.480897 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.480853    6059 scope.go:122] "RemoveContainer" containerID="d161997d4c5d3f12ccde9dd66617e969343c1ce0480b90187b5b5f3a561ef0e8"
Nov 03 17:48:59.487264 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.487244    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77951355-6d25-4504-9a27-db09e1d94236-kube-api-access-fx6rk\" (UniqueName: \"kubernetes.io/projected/77951355-6d25-4504-9a27-db09e1d94236-kube-api-access-fx6rk\") pod \"77951355-6d25-4504-9a27-db09e1d94236\" (UID: \"77951355-6d25-4504-9a27-db09e1d94236\") "
Nov 03 17:48:59.487370 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.487338    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77951355-6d25-4504-9a27-db09e1d94236-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77951355-6d25-4504-9a27-db09e1d94236-plugins-config-volume\") pod \"77951355-6d25-4504-9a27-db09e1d94236\" (UID: \"77951355-6d25-4504-9a27-db09e1d94236\") "
Nov 03 17:48:59.487751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.487726    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77951355-6d25-4504-9a27-db09e1d94236-plugins-config-volume" pod "77951355-6d25-4504-9a27-db09e1d94236" (UID: "77951355-6d25-4504-9a27-db09e1d94236"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:48:59.488043 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.488020    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77951355-6d25-4504-9a27-db09e1d94236-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:59.489399 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.489372    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77951355-6d25-4504-9a27-db09e1d94236-kube-api-access-fx6rk" pod "77951355-6d25-4504-9a27-db09e1d94236" (UID: "77951355-6d25-4504-9a27-db09e1d94236"). InnerVolumeSpecName "kube-api-access-fx6rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:48:59.493334 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.493301    6059 scope.go:122] "RemoveContainer" containerID="d161997d4c5d3f12ccde9dd66617e969343c1ce0480b90187b5b5f3a561ef0e8"
Nov 03 17:48:59.495791 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:48:59.495765    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d161997d4c5d3f12ccde9dd66617e969343c1ce0480b90187b5b5f3a561ef0e8\": container with ID starting with d161997d4c5d3f12ccde9dd66617e969343c1ce0480b90187b5b5f3a561ef0e8 not found: ID does not exist" containerID="d161997d4c5d3f12ccde9dd66617e969343c1ce0480b90187b5b5f3a561ef0e8"
Nov 03 17:48:59.495891 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.495801    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d161997d4c5d3f12ccde9dd66617e969343c1ce0480b90187b5b5f3a561ef0e8"} err="failed to get container status \"d161997d4c5d3f12ccde9dd66617e969343c1ce0480b90187b5b5f3a561ef0e8\": rpc error: code = NotFound desc = could not find container \"d161997d4c5d3f12ccde9dd66617e969343c1ce0480b90187b5b5f3a561ef0e8\": container with ID starting with d161997d4c5d3f12ccde9dd66617e969343c1ce0480b90187b5b5f3a561ef0e8 not found: ID does not exist"
Nov 03 17:48:59.507937 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.507916    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8"
Nov 03 17:48:59.589858 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.589820    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx6rk\" (UniqueName: \"kubernetes.io/projected/77951355-6d25-4504-9a27-db09e1d94236-kube-api-access-fx6rk\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:59.687414 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.687327    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10021fbc-6q4s6"]
Nov 03 17:48:59.687945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.687929    6059 cpu_manager.go:414] "containerMap: removing container" podUID="50c1cc3e-ffc9-4b51-9a95-5677f4898c02" containerName="payload-pre-processing"
Nov 03 17:48:59.688003 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.687946    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c1cc3e-ffc9-4b51-9a95-5677f4898c02" containerName="payload-pre-processing"
Nov 03 17:48:59.688003 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.687973    6059 cpu_manager.go:414] "containerMap: removing container" podUID="a601f158-37db-48fb-936d-e3c4f9ce235f" containerName="revoke-keys"
Nov 03 17:48:59.688003 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.687978    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a601f158-37db-48fb-936d-e3c4f9ce235f" containerName="revoke-keys"
Nov 03 17:48:59.688003 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.687991    6059 cpu_manager.go:414] "containerMap: removing container" podUID="77951355-6d25-4504-9a27-db09e1d94236" containerName="payload-processing"
Nov 03 17:48:59.688003 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.687996    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77951355-6d25-4504-9a27-db09e1d94236" containerName="payload-processing"
Nov 03 17:48:59.688164 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.688083    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c1cc3e-ffc9-4b51-9a95-5677f4898c02" containerName="payload-pre-processing"
Nov 03 17:48:59.688164 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.688097    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a601f158-37db-48fb-936d-e3c4f9ce235f" containerName="revoke-keys"
Nov 03 17:48:59.688164 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.688105    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="77951355-6d25-4504-9a27-db09e1d94236" containerName="payload-processing"
Nov 03 17:48:59.690576 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.690516    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50c1cc3e-ffc9-4b51-9a95-5677f4898c02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c1cc3e-ffc9-4b51-9a95-5677f4898c02-plugins-config-volume\") pod \"50c1cc3e-ffc9-4b51-9a95-5677f4898c02\" (UID: \"50c1cc3e-ffc9-4b51-9a95-5677f4898c02\") "
Nov 03 17:48:59.690887 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.690860    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c1cc3e-ffc9-4b51-9a95-5677f4898c02-plugins-config-volume" pod "50c1cc3e-ffc9-4b51-9a95-5677f4898c02" (UID: "50c1cc3e-ffc9-4b51-9a95-5677f4898c02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:48:59.691007 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.690886    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c1cc3e-ffc9-4b51-9a95-5677f4898c02-kube-api-access-jsz2j\" (UniqueName: \"kubernetes.io/projected/50c1cc3e-ffc9-4b51-9a95-5677f4898c02-kube-api-access-jsz2j\") pod \"50c1cc3e-ffc9-4b51-9a95-5677f4898c02\" (UID: \"50c1cc3e-ffc9-4b51-9a95-5677f4898c02\") "
Nov 03 17:48:59.691449 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.691429    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10021fbc-6q4s6"
Nov 03 17:48:59.691522 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.691501    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c1cc3e-ffc9-4b51-9a95-5677f4898c02-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:59.692971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.692947    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c1cc3e-ffc9-4b51-9a95-5677f4898c02-kube-api-access-jsz2j" pod "50c1cc3e-ffc9-4b51-9a95-5677f4898c02" (UID: "50c1cc3e-ffc9-4b51-9a95-5677f4898c02"). InnerVolumeSpecName "kube-api-access-jsz2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:48:59.753699 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.753663    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10021fbc-6q4s6"]
Nov 03 17:48:59.754068 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.754046    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:48:59.754427 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.754403    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:48:59.754526 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.754436    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:48:59.754526 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.754493    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:48:59.792427 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.792401    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319eee54-1366-4f28-a395-23f1c57a0b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10021fbc-6q4s6\" (UID: \"319eee54-1366-4f28-a395-23f1c57a0b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10021fbc-6q4s6"
Nov 03 17:48:59.792555 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.792442    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlz4m\" (UniqueName: \"kubernetes.io/projected/319eee54-1366-4f28-a395-23f1c57a0b7d-kube-api-access-vlz4m\") pod \"maas-api-revoke-sub-10021fbc-6q4s6\" (UID: \"319eee54-1366-4f28-a395-23f1c57a0b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10021fbc-6q4s6"
Nov 03 17:48:59.792729 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.792710    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319eee54-1366-4f28-a395-23f1c57a0b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10021fbc-6q4s6\" (UID: \"319eee54-1366-4f28-a395-23f1c57a0b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10021fbc-6q4s6"
Nov 03 17:48:59.792825 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.792811    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsz2j\" (UniqueName: \"kubernetes.io/projected/50c1cc3e-ffc9-4b51-9a95-5677f4898c02-kube-api-access-jsz2j\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:48:59.893891 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.893857    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319eee54-1366-4f28-a395-23f1c57a0b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10021fbc-6q4s6\" (UID: \"319eee54-1366-4f28-a395-23f1c57a0b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10021fbc-6q4s6"
Nov 03 17:48:59.894095 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.893937    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319eee54-1366-4f28-a395-23f1c57a0b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10021fbc-6q4s6\" (UID: \"319eee54-1366-4f28-a395-23f1c57a0b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10021fbc-6q4s6"
Nov 03 17:48:59.894095 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.893961    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlz4m\" (UniqueName: \"kubernetes.io/projected/319eee54-1366-4f28-a395-23f1c57a0b7d-kube-api-access-vlz4m\") pod \"maas-api-revoke-sub-10021fbc-6q4s6\" (UID: \"319eee54-1366-4f28-a395-23f1c57a0b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10021fbc-6q4s6"
Nov 03 17:48:59.894574 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.894548    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319eee54-1366-4f28-a395-23f1c57a0b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10021fbc-6q4s6\" (UID: \"319eee54-1366-4f28-a395-23f1c57a0b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10021fbc-6q4s6"
Nov 03 17:48:59.896318 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.896295    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319eee54-1366-4f28-a395-23f1c57a0b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10021fbc-6q4s6\" (UID: \"319eee54-1366-4f28-a395-23f1c57a0b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10021fbc-6q4s6"
Nov 03 17:48:59.957649 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.957619    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlz4m\" (UniqueName: \"kubernetes.io/projected/319eee54-1366-4f28-a395-23f1c57a0b7d-kube-api-access-vlz4m\") pod \"maas-api-revoke-sub-10021fbc-6q4s6\" (UID: \"319eee54-1366-4f28-a395-23f1c57a0b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10021fbc-6q4s6"
Nov 03 17:48:59.968637 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.968321    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l"]
Nov 03 17:48:59.972599 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:48:59.972574    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1b3c7bad-ddffc7b9c-r7w8l"]
Nov 03 17:49:00.020032 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:00.019997    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10021fbc-6q4s6"
Nov 03 17:49:00.399251 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:49:00.399211    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319eee54_1366_4f28_a395_23f1c57a0b7d.slice/crio-ab1268e275453fe5c9e234b2f2641ffabb97d4557c765a1e6c4a7abc2b6df9b7 WatchSource:0}: Error finding container ab1268e275453fe5c9e234b2f2641ffabb97d4557c765a1e6c4a7abc2b6df9b7: Status 404 returned error can't find the container with id ab1268e275453fe5c9e234b2f2641ffabb97d4557c765a1e6c4a7abc2b6df9b7
Nov 03 17:49:00.409327 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:00.404788    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10021fbc-6q4s6"]
Nov 03 17:49:00.489927 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:00.489888    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8" event={"ID":"50c1cc3e-ffc9-4b51-9a95-5677f4898c02","Type":"ContainerDied","Data":"1cfeeb9551cba7cce16c6378adebfcf8ad0dfd949ffc065f0ba6cfec07d1f3a1"}
Nov 03 17:49:00.490100 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:00.489965    6059 scope.go:122] "RemoveContainer" containerID="ef1a4b620bb8dbebbbf7f2b275aafea2fad5c5118a097d4c0554d631a389c5ab"
Nov 03 17:49:00.490100 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:00.489901    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8"
Nov 03 17:49:00.491507 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:00.491200    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10021fbc-6q4s6" event={"ID":"319eee54-1366-4f28-a395-23f1c57a0b7d","Type":"ContainerStarted","Data":"ab1268e275453fe5c9e234b2f2641ffabb97d4557c765a1e6c4a7abc2b6df9b7"}
Nov 03 17:49:00.536622 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:00.536592    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:00.559912 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:00.559853    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v" podUID="2bbc9082-b2da-4198-8a8a-b6ab26b5011e" containerName="main" containerID="cri-o://865b337eb340e3f06fc26671774e0097dd03638c98f2892b7b2d6fc6d36543b5" gracePeriod=45
Nov 03 17:49:00.621859 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:00.621819    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v" podUID="2bbc9082-b2da-4198-8a8a-b6ab26b5011e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": dial tcp 10.133.0.50:8000: connect: connection refused"
Nov 03 17:49:00.664412 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:00.664350    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8"]
Nov 03 17:49:00.670603 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:00.670573    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1b3c7bad-9d9669c5f-m4sq8"]
Nov 03 17:49:00.716458 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:00.716422    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:00.867805 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:00.867779    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:49:01.008646 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.008614    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-tls-certs\" (UniqueName: \"kubernetes.io/secret/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-tls-certs\") pod \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") "
Nov 03 17:49:01.008861 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.008669    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-dshm\") pod \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") "
Nov 03 17:49:01.008861 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.008717    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-model-cache\") pod \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") "
Nov 03 17:49:01.009066 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.009037    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-model-cache" pod "2bbc9082-b2da-4198-8a8a-b6ab26b5011e" (UID: "2bbc9082-b2da-4198-8a8a-b6ab26b5011e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:01.009399 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.009248    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-kube-api-access-twbx8\" (UniqueName: \"kubernetes.io/projected/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-kube-api-access-twbx8\") pod \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") "
Nov 03 17:49:01.009763 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.009597    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-home\" (UniqueName: \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-home\") pod \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") "
Nov 03 17:49:01.009901 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.009835    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-home" pod "2bbc9082-b2da-4198-8a8a-b6ab26b5011e" (UID: "2bbc9082-b2da-4198-8a8a-b6ab26b5011e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:01.010094 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.010068    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-tmp-dir\") pod \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\" (UID: \"2bbc9082-b2da-4198-8a8a-b6ab26b5011e\") "
Nov 03 17:49:01.010488 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.010459    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-tmp-dir" pod "2bbc9082-b2da-4198-8a8a-b6ab26b5011e" (UID: "2bbc9082-b2da-4198-8a8a-b6ab26b5011e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:01.010715 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.010691    6059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-tmp-dir\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:01.010810 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.010722    6059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-model-cache\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:01.010810 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.010738    6059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-home\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:01.011480 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.011453    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-tls-certs" pod "2bbc9082-b2da-4198-8a8a-b6ab26b5011e" (UID: "2bbc9082-b2da-4198-8a8a-b6ab26b5011e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:49:01.012466 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.012445    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-dshm" pod "2bbc9082-b2da-4198-8a8a-b6ab26b5011e" (UID: "2bbc9082-b2da-4198-8a8a-b6ab26b5011e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:01.014517 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.014495    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-kube-api-access-twbx8" pod "2bbc9082-b2da-4198-8a8a-b6ab26b5011e" (UID: "2bbc9082-b2da-4198-8a8a-b6ab26b5011e"). InnerVolumeSpecName "kube-api-access-twbx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:49:01.111541 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.111513    6059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-tls-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:01.111541 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.111538    6059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-dshm\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:01.111541 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.111549    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twbx8\" (UniqueName: \"kubernetes.io/projected/2bbc9082-b2da-4198-8a8a-b6ab26b5011e-kube-api-access-twbx8\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:01.337975 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.337945    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50c1cc3e-ffc9-4b51-9a95-5677f4898c02" path="/var/lib/kubelet/pods/50c1cc3e-ffc9-4b51-9a95-5677f4898c02/volumes"
Nov 03 17:49:01.338417 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.338401    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77951355-6d25-4504-9a27-db09e1d94236" path="/var/lib/kubelet/pods/77951355-6d25-4504-9a27-db09e1d94236/volumes"
Nov 03 17:49:01.501490 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.501455    6059 generic.go:356] "Generic (PLEG): container finished" podID="319eee54-1366-4f28-a395-23f1c57a0b7d" containerID="2523de2fd34c4ef3e674a9fd2c9f4a40e62c5d5abb0d9e85af8c4e184030aafb" exitCode=0
Nov 03 17:49:01.501706 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.501546    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10021fbc-6q4s6" event={"ID":"319eee54-1366-4f28-a395-23f1c57a0b7d","Type":"ContainerDied","Data":"2523de2fd34c4ef3e674a9fd2c9f4a40e62c5d5abb0d9e85af8c4e184030aafb"}
Nov 03 17:49:01.502839 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.502817    6059 generic.go:356] "Generic (PLEG): container finished" podID="2bbc9082-b2da-4198-8a8a-b6ab26b5011e" containerID="865b337eb340e3f06fc26671774e0097dd03638c98f2892b7b2d6fc6d36543b5" exitCode=0
Nov 03 17:49:01.502930 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.502855    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"
Nov 03 17:49:01.502969 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.502855    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v" event={"ID":"2bbc9082-b2da-4198-8a8a-b6ab26b5011e","Type":"ContainerDied","Data":"865b337eb340e3f06fc26671774e0097dd03638c98f2892b7b2d6fc6d36543b5"}
Nov 03 17:49:01.503029 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.502973    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v" event={"ID":"2bbc9082-b2da-4198-8a8a-b6ab26b5011e","Type":"ContainerDied","Data":"691fb0173b09fcdf872c09c92d421e120068248534d74fda34fac94dba387815"}
Nov 03 17:49:01.503029 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.502996    6059 scope.go:122] "RemoveContainer" containerID="865b337eb340e3f06fc26671774e0097dd03638c98f2892b7b2d6fc6d36543b5"
Nov 03 17:49:01.515966 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.515938    6059 scope.go:122] "RemoveContainer" containerID="865b337eb340e3f06fc26671774e0097dd03638c98f2892b7b2d6fc6d36543b5"
Nov 03 17:49:01.516595 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:49:01.516561    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"865b337eb340e3f06fc26671774e0097dd03638c98f2892b7b2d6fc6d36543b5\": container with ID starting with 865b337eb340e3f06fc26671774e0097dd03638c98f2892b7b2d6fc6d36543b5 not found: ID does not exist" containerID="865b337eb340e3f06fc26671774e0097dd03638c98f2892b7b2d6fc6d36543b5"
Nov 03 17:49:01.516710 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.516609    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865b337eb340e3f06fc26671774e0097dd03638c98f2892b7b2d6fc6d36543b5"} err="failed to get container status \"865b337eb340e3f06fc26671774e0097dd03638c98f2892b7b2d6fc6d36543b5\": rpc error: code = NotFound desc = could not find container \"865b337eb340e3f06fc26671774e0097dd03638c98f2892b7b2d6fc6d36543b5\": container with ID starting with 865b337eb340e3f06fc26671774e0097dd03638c98f2892b7b2d6fc6d36543b5 not found: ID does not exist"
Nov 03 17:49:01.781031 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.780996    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"]
Nov 03 17:49:01.784028 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:01.783980    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/simulator-w0-687fb3-kserve-5999dd6c95-ftv7v"]
Nov 03 17:49:02.306233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:02.306192    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:02.663473 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:02.663449    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10021fbc-6q4s6"
Nov 03 17:49:02.834561 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:02.834467    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/319eee54-1366-4f28-a395-23f1c57a0b7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319eee54-1366-4f28-a395-23f1c57a0b7d-maas-api-xks-ca\") pod \"319eee54-1366-4f28-a395-23f1c57a0b7d\" (UID: \"319eee54-1366-4f28-a395-23f1c57a0b7d\") "
Nov 03 17:49:02.835023 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:02.834644    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/319eee54-1366-4f28-a395-23f1c57a0b7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319eee54-1366-4f28-a395-23f1c57a0b7d-maas-api-openshift-ca\") pod \"319eee54-1366-4f28-a395-23f1c57a0b7d\" (UID: \"319eee54-1366-4f28-a395-23f1c57a0b7d\") "
Nov 03 17:49:02.835023 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:02.834706    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/319eee54-1366-4f28-a395-23f1c57a0b7d-kube-api-access-vlz4m\" (UniqueName: \"kubernetes.io/projected/319eee54-1366-4f28-a395-23f1c57a0b7d-kube-api-access-vlz4m\") pod \"319eee54-1366-4f28-a395-23f1c57a0b7d\" (UID: \"319eee54-1366-4f28-a395-23f1c57a0b7d\") "
Nov 03 17:49:02.835151 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:02.835037    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/319eee54-1366-4f28-a395-23f1c57a0b7d-maas-api-openshift-ca" pod "319eee54-1366-4f28-a395-23f1c57a0b7d" (UID: "319eee54-1366-4f28-a395-23f1c57a0b7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:49:02.835151 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:02.835120    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319eee54-1366-4f28-a395-23f1c57a0b7d-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:02.836737 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:02.836705    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/319eee54-1366-4f28-a395-23f1c57a0b7d-maas-api-xks-ca" pod "319eee54-1366-4f28-a395-23f1c57a0b7d" (UID: "319eee54-1366-4f28-a395-23f1c57a0b7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:49:02.836861 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:02.836787    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319eee54-1366-4f28-a395-23f1c57a0b7d-kube-api-access-vlz4m" pod "319eee54-1366-4f28-a395-23f1c57a0b7d" (UID: "319eee54-1366-4f28-a395-23f1c57a0b7d"). InnerVolumeSpecName "kube-api-access-vlz4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:49:02.936190 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:02.936152    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlz4m\" (UniqueName: \"kubernetes.io/projected/319eee54-1366-4f28-a395-23f1c57a0b7d-kube-api-access-vlz4m\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:02.936190 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:02.936183    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319eee54-1366-4f28-a395-23f1c57a0b7d-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:03.377391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:03.377248    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bbc9082-b2da-4198-8a8a-b6ab26b5011e" path="/var/lib/kubelet/pods/2bbc9082-b2da-4198-8a8a-b6ab26b5011e/volumes"
Nov 03 17:49:03.554384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:03.550336    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10021fbc-6q4s6" event={"ID":"319eee54-1366-4f28-a395-23f1c57a0b7d","Type":"ContainerDied","Data":"ab1268e275453fe5c9e234b2f2641ffabb97d4557c765a1e6c4a7abc2b6df9b7"}
Nov 03 17:49:03.554384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:03.550426    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab1268e275453fe5c9e234b2f2641ffabb97d4557c765a1e6c4a7abc2b6df9b7"
Nov 03 17:49:03.554384 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:03.550504    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10021fbc-6q4s6"
Nov 03 17:49:05.449381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:05.448964    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:06.508378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.507961    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-270e09a7-wcwsw"]
Nov 03 17:49:06.510380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.509110    6059 cpu_manager.go:414] "containerMap: removing container" podUID="2bbc9082-b2da-4198-8a8a-b6ab26b5011e" containerName="main"
Nov 03 17:49:06.510380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.509150    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bbc9082-b2da-4198-8a8a-b6ab26b5011e" containerName="main"
Nov 03 17:49:06.510380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.509170    6059 cpu_manager.go:414] "containerMap: removing container" podUID="319eee54-1366-4f28-a395-23f1c57a0b7d" containerName="revoke-keys"
Nov 03 17:49:06.510380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.509179    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="319eee54-1366-4f28-a395-23f1c57a0b7d" containerName="revoke-keys"
Nov 03 17:49:06.510380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.509456    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bbc9082-b2da-4198-8a8a-b6ab26b5011e" containerName="main"
Nov 03 17:49:06.510380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.509475    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="319eee54-1366-4f28-a395-23f1c57a0b7d" containerName="revoke-keys"
Nov 03 17:49:06.515375 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.515314    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270e09a7-wcwsw"
Nov 03 17:49:06.618000 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.617726    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:49:06.618000 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.617816    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:49:06.618000 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.617870    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:49:06.618814 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.618529    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-270e09a7-wcwsw\" (UID: \"8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270e09a7-wcwsw"
Nov 03 17:49:06.618814 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.618715    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmvdq\" (UniqueName: \"kubernetes.io/projected/8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723-kube-api-access-gmvdq\") pod \"maas-api-revoke-sub-270e09a7-wcwsw\" (UID: \"8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270e09a7-wcwsw"
Nov 03 17:49:06.618986 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.618932    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-270e09a7-wcwsw\" (UID: \"8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270e09a7-wcwsw"
Nov 03 17:49:06.621267 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.621244    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-270e09a7-wcwsw"]
Nov 03 17:49:06.641444 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.639534    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:49:06.719924 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.719890    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-270e09a7-wcwsw\" (UID: \"8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270e09a7-wcwsw"
Nov 03 17:49:06.720317 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.720292    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-270e09a7-wcwsw\" (UID: \"8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270e09a7-wcwsw"
Nov 03 17:49:06.720577 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.720557    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmvdq\" (UniqueName: \"kubernetes.io/projected/8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723-kube-api-access-gmvdq\") pod \"maas-api-revoke-sub-270e09a7-wcwsw\" (UID: \"8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270e09a7-wcwsw"
Nov 03 17:49:06.721048 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.721023    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-270e09a7-wcwsw\" (UID: \"8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270e09a7-wcwsw"
Nov 03 17:49:06.722848 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.722829    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-270e09a7-wcwsw\" (UID: \"8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270e09a7-wcwsw"
Nov 03 17:49:06.809460 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.809341    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmvdq\" (UniqueName: \"kubernetes.io/projected/8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723-kube-api-access-gmvdq\") pod \"maas-api-revoke-sub-270e09a7-wcwsw\" (UID: \"8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270e09a7-wcwsw"
Nov 03 17:49:06.860406 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:06.848558    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270e09a7-wcwsw"
Nov 03 17:49:07.025711 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:49:07.025673    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:49:07.578030 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:07.577994    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:07.640404 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:07.637081    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-270e09a7-wcwsw"]
Nov 03 17:49:07.953071 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:07.953040    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:08.665369 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:08.665325    6059 generic.go:356] "Generic (PLEG): container finished" podID="8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723" containerID="650d4f1508ddcfd3cfc641eaa19e754642fcc526f30944cf12c25939f5badf1e" exitCode=0
Nov 03 17:49:08.665794 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:08.665405    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270e09a7-wcwsw" event={"ID":"8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723","Type":"ContainerDied","Data":"650d4f1508ddcfd3cfc641eaa19e754642fcc526f30944cf12c25939f5badf1e"}
Nov 03 17:49:08.665794 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:08.665462    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270e09a7-wcwsw" event={"ID":"8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723","Type":"ContainerStarted","Data":"d9d1768174d2f5ac884a6822e0fcfed460998108c473d95076f1bc31fc1d68a7"}
Nov 03 17:49:09.811744 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:09.811718    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270e09a7-wcwsw"
Nov 03 17:49:09.857825 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:09.857796    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723-maas-api-xks-ca\") pod \"8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723\" (UID: \"8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723\") "
Nov 03 17:49:09.857971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:09.857940    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723-kube-api-access-gmvdq\" (UniqueName: \"kubernetes.io/projected/8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723-kube-api-access-gmvdq\") pod \"8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723\" (UID: \"8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723\") "
Nov 03 17:49:09.858037 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:09.858012    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723-maas-api-openshift-ca\") pod \"8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723\" (UID: \"8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723\") "
Nov 03 17:49:09.858429 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:09.858397    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723-maas-api-openshift-ca" pod "8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723" (UID: "8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:49:09.858569 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:09.858545    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:09.860425 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:09.860400    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723-kube-api-access-gmvdq" pod "8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723" (UID: "8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723"). InnerVolumeSpecName "kube-api-access-gmvdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:49:09.860529 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:09.860484    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723-maas-api-xks-ca" pod "8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723" (UID: "8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:49:09.960299 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:09.960272    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmvdq\" (UniqueName: \"kubernetes.io/projected/8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723-kube-api-access-gmvdq\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:09.960443 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:09.960301    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:10.680853 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:10.680825    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270e09a7-wcwsw"
Nov 03 17:49:10.681035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:10.680822    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270e09a7-wcwsw" event={"ID":"8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723","Type":"ContainerDied","Data":"d9d1768174d2f5ac884a6822e0fcfed460998108c473d95076f1bc31fc1d68a7"}
Nov 03 17:49:10.681035 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:10.680951    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9d1768174d2f5ac884a6822e0fcfed460998108c473d95076f1bc31fc1d68a7"
Nov 03 17:49:11.556675 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:11.556646    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:11.976657 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:11.976624    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:12.697174 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:12.697142    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:13.067718 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:13.067682    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:13.360118 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:13.359989    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:13.820584 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:13.820547    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:14.485716 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:14.485677    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:14.701896 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:14.701862    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:15.351789 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:15.351751    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:15.640789 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:15.640692    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:18.332246 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:18.332171    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:18.628300 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:18.628209    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"]
Nov 03 17:49:18.750862 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:18.750831    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:19.013598 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:19.013563    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:19.392040 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:19.391926    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:19.595824 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:19.595790    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:19.818546 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:19.818507    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:22.570850 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:22.570816    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:23.723572 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:23.723539    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:23.911101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:23.911064    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:24.151850 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.151769    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:24.337151 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.337117    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:24.382286 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.382243    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c"]
Nov 03 17:49:24.382899 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.382877    6059 cpu_manager.go:414] "containerMap: removing container" podUID="8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723" containerName="revoke-keys"
Nov 03 17:49:24.383003 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.382901    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723" containerName="revoke-keys"
Nov 03 17:49:24.383104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.383091    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723" containerName="revoke-keys"
Nov 03 17:49:24.387891 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.387869    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c"
Nov 03 17:49:24.452208 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.452108    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c"]
Nov 03 17:49:24.505379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.501727    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-382697-dockercfg-hcx28\""
Nov 03 17:49:24.505379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.501797    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-382697\""
Nov 03 17:49:24.531379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.526743    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:24.553478 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.553441    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb"]
Nov 03 17:49:24.557930 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.557904    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb"
Nov 03 17:49:24.605510 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.605470    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm226\" (UniqueName: \"kubernetes.io/projected/c4f61445-3d05-4900-9eb7-5d8a55eadf66-kube-api-access-jm226\") pod \"payload-processing-e2e-api-a-382697-7957766c97-6gt6c\" (UID: \"c4f61445-3d05-4900-9eb7-5d8a55eadf66\") " pod="openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c"
Nov 03 17:49:24.605717 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.605522    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c5fac3-1291-4c88-b6f4-1cf0829e33f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb\" (UID: \"53c5fac3-1291-4c88-b6f4-1cf0829e33f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb"
Nov 03 17:49:24.605834 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.605802    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f61445-3d05-4900-9eb7-5d8a55eadf66-plugins-config-volume\") pod \"payload-processing-e2e-api-a-382697-7957766c97-6gt6c\" (UID: \"c4f61445-3d05-4900-9eb7-5d8a55eadf66\") " pod="openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c"
Nov 03 17:49:24.605905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.605870    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxzw5\" (UniqueName: \"kubernetes.io/projected/53c5fac3-1291-4c88-b6f4-1cf0829e33f7-kube-api-access-cxzw5\") pod \"payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb\" (UID: \"53c5fac3-1291-4c88-b6f4-1cf0829e33f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb"
Nov 03 17:49:24.617390 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.617337    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb"]
Nov 03 17:49:24.706600 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.706569    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f61445-3d05-4900-9eb7-5d8a55eadf66-plugins-config-volume\") pod \"payload-processing-e2e-api-a-382697-7957766c97-6gt6c\" (UID: \"c4f61445-3d05-4900-9eb7-5d8a55eadf66\") " pod="openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c"
Nov 03 17:49:24.706764 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.706628    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxzw5\" (UniqueName: \"kubernetes.io/projected/53c5fac3-1291-4c88-b6f4-1cf0829e33f7-kube-api-access-cxzw5\") pod \"payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb\" (UID: \"53c5fac3-1291-4c88-b6f4-1cf0829e33f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb"
Nov 03 17:49:24.706764 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.706697    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm226\" (UniqueName: \"kubernetes.io/projected/c4f61445-3d05-4900-9eb7-5d8a55eadf66-kube-api-access-jm226\") pod \"payload-processing-e2e-api-a-382697-7957766c97-6gt6c\" (UID: \"c4f61445-3d05-4900-9eb7-5d8a55eadf66\") " pod="openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c"
Nov 03 17:49:24.706764 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.706749    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c5fac3-1291-4c88-b6f4-1cf0829e33f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb\" (UID: \"53c5fac3-1291-4c88-b6f4-1cf0829e33f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb"
Nov 03 17:49:24.707506 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.707482    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f61445-3d05-4900-9eb7-5d8a55eadf66-plugins-config-volume\") pod \"payload-processing-e2e-api-a-382697-7957766c97-6gt6c\" (UID: \"c4f61445-3d05-4900-9eb7-5d8a55eadf66\") " pod="openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c"
Nov 03 17:49:24.707631 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.707541    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c5fac3-1291-4c88-b6f4-1cf0829e33f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb\" (UID: \"53c5fac3-1291-4c88-b6f4-1cf0829e33f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb"
Nov 03 17:49:24.769974 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.769941    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxzw5\" (UniqueName: \"kubernetes.io/projected/53c5fac3-1291-4c88-b6f4-1cf0829e33f7-kube-api-access-cxzw5\") pod \"payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb\" (UID: \"53c5fac3-1291-4c88-b6f4-1cf0829e33f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb"
Nov 03 17:49:24.770333 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.769995    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm226\" (UniqueName: \"kubernetes.io/projected/c4f61445-3d05-4900-9eb7-5d8a55eadf66-kube-api-access-jm226\") pod \"payload-processing-e2e-api-a-382697-7957766c97-6gt6c\" (UID: \"c4f61445-3d05-4900-9eb7-5d8a55eadf66\") " pod="openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c"
Nov 03 17:49:24.876844 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:24.876809    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb"
Nov 03 17:49:25.001510 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:25.001424    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c"
Nov 03 17:49:25.255724 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:49:25.255681    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c5fac3_1291_4c88_b6f4_1cf0829e33f7.slice/crio-5624957d004a3ee97ae20c5646ecea28ee264eb5c7b2263f9de02cec9ef2705e WatchSource:0}: Error finding container 5624957d004a3ee97ae20c5646ecea28ee264eb5c7b2263f9de02cec9ef2705e: Status 404 returned error can't find the container with id 5624957d004a3ee97ae20c5646ecea28ee264eb5c7b2263f9de02cec9ef2705e
Nov 03 17:49:25.258249 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:25.258225    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb"]
Nov 03 17:49:25.387815 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:25.387792    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c"]
Nov 03 17:49:25.793074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:25.793040    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c" event={"ID":"c4f61445-3d05-4900-9eb7-5d8a55eadf66","Type":"ContainerStarted","Data":"0dfab80b6f5cac141a6d4ecfa7aa1ff358b5de927296e3b2f80d937dfe6973a1"}
Nov 03 17:49:25.793524 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:25.793103    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c"
Nov 03 17:49:25.793524 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:25.793129    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c" event={"ID":"c4f61445-3d05-4900-9eb7-5d8a55eadf66","Type":"ContainerStarted","Data":"df59efc8598c2b6eacdcaaf4fd54cc78d8046167bad8d308783ca8e70a038a0f"}
Nov 03 17:49:25.794821 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:25.794797    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb" event={"ID":"53c5fac3-1291-4c88-b6f4-1cf0829e33f7","Type":"ContainerStarted","Data":"23193fe06d97a0da09bf3097dcdc3190809b5af8c59ef6671949b0206431fad1"}
Nov 03 17:49:25.794962 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:25.794847    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb"
Nov 03 17:49:25.794962 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:25.794871    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb" event={"ID":"53c5fac3-1291-4c88-b6f4-1cf0829e33f7","Type":"ContainerStarted","Data":"5624957d004a3ee97ae20c5646ecea28ee264eb5c7b2263f9de02cec9ef2705e"}
Nov 03 17:49:25.913192 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:25.913127    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c" podStartSLOduration=1.9131102260000001 podStartE2EDuration="1.913110226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:25.912938896 +0000 UTC m=+1906.710651689" watchObservedRunningTime="2026-11-03 17:49:25.913110226 +0000 UTC m=+1906.710823008"
Nov 03 17:49:26.035076 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:26.035017    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb" podStartSLOduration=2.035000794 podStartE2EDuration="2.035000794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:26.032836891 +0000 UTC m=+1906.830549673" watchObservedRunningTime="2026-11-03 17:49:26.035000794 +0000 UTC m=+1906.832713576"
Nov 03 17:49:26.436163 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:26.436111    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:26.894007 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:26.893971    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"]
Nov 03 17:49:26.894548 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:26.894232    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c" podUID="84690202-b89e-4135-aebb-aa53c0e18463" containerName="istio-proxy" containerID="cri-o://2ae27f4cfca9606459d75e02e45f3162805a5e58f43d5339e5dd6c65ffd91371" gracePeriod=30
Nov 03 17:49:26.912781 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:26.912742    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"]
Nov 03 17:49:26.912993 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:26.912969    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj" podUID="cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73" containerName="istio-proxy" containerID="cri-o://7348867872b9d5e2a56f7bbb4d97654e43557219048d510d805fdc191e1792ac" gracePeriod=30
Nov 03 17:49:26.936000 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:26.935955    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:28.906668 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:28.906632    6059 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:49:28.907150 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:28.906698    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c" podUID="84690202-b89e-4135-aebb-aa53c0e18463" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:49:29.113678 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:29.113645    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:30.870832 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:30.870794    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:32.195378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.195338    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:49:32.213949 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.213924    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:49:32.225926 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.225903    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84690202-b89e-4135-aebb-aa53c0e18463-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84690202-b89e-4135-aebb-aa53c0e18463-istiod-ca-cert\") pod \"84690202-b89e-4135-aebb-aa53c0e18463\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") "
Nov 03 17:49:32.226048 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.225946    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-envoy\") pod \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") "
Nov 03 17:49:32.226048 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.225975    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istiod-ca-cert\") pod \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") "
Nov 03 17:49:32.226048 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.225999    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-workload-socket\") pod \"84690202-b89e-4135-aebb-aa53c0e18463\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") "
Nov 03 17:49:32.226048 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.226034    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-data\") pod \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") "
Nov 03 17:49:32.226283 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.226175    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-credential-socket\") pod \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") "
Nov 03 17:49:32.226283 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.226242    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-token\" (UniqueName: \"kubernetes.io/projected/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-token\") pod \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") "
Nov 03 17:49:32.226283 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.226274    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-podinfo\") pod \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") "
Nov 03 17:49:32.226463 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.226333    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-workload-certs\") pod \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") "
Nov 03 17:49:32.226463 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.226334    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-data" pod "cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73" (UID: "cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:32.226463 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.226379    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-workload-certs\") pod \"84690202-b89e-4135-aebb-aa53c0e18463\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") "
Nov 03 17:49:32.226463 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.226412    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84690202-b89e-4135-aebb-aa53c0e18463-kube-api-access-4hktx\" (UniqueName: \"kubernetes.io/projected/84690202-b89e-4135-aebb-aa53c0e18463-kube-api-access-4hktx\") pod \"84690202-b89e-4135-aebb-aa53c0e18463\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") "
Nov 03 17:49:32.226463 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.226458    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-credential-socket\") pod \"84690202-b89e-4135-aebb-aa53c0e18463\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") "
Nov 03 17:49:32.226731 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.226484    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-kube-api-access-xwktw\" (UniqueName: \"kubernetes.io/projected/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-kube-api-access-xwktw\") pod \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") "
Nov 03 17:49:32.226731 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.226522    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-istio-envoy\") pod \"84690202-b89e-4135-aebb-aa53c0e18463\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") "
Nov 03 17:49:32.226731 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.226522    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-credential-socket" pod "cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73" (UID: "cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:32.226731 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.226543    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84690202-b89e-4135-aebb-aa53c0e18463-istiod-ca-cert" pod "84690202-b89e-4135-aebb-aa53c0e18463" (UID: "84690202-b89e-4135-aebb-aa53c0e18463"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:49:32.226731 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.226633    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-workload-socket\") pod \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\" (UID: \"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73\") "
Nov 03 17:49:32.226731 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.226676    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84690202-b89e-4135-aebb-aa53c0e18463-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84690202-b89e-4135-aebb-aa53c0e18463-istio-podinfo\") pod \"84690202-b89e-4135-aebb-aa53c0e18463\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") "
Nov 03 17:49:32.226731 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.226673    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-workload-socket" pod "84690202-b89e-4135-aebb-aa53c0e18463" (UID: "84690202-b89e-4135-aebb-aa53c0e18463"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:32.227115 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.226736    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-istio-data\") pod \"84690202-b89e-4135-aebb-aa53c0e18463\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") "
Nov 03 17:49:32.227115 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.226768    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84690202-b89e-4135-aebb-aa53c0e18463-istio-token\" (UniqueName: \"kubernetes.io/projected/84690202-b89e-4135-aebb-aa53c0e18463-istio-token\") pod \"84690202-b89e-4135-aebb-aa53c0e18463\" (UID: \"84690202-b89e-4135-aebb-aa53c0e18463\") "
Nov 03 17:49:32.227115 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.226921    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-workload-socket" pod "cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73" (UID: "cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:32.227115 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.227103    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-workload-certs" pod "cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73" (UID: "cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:32.228019 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.227462    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:32.228019 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.227498    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:32.228019 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.227514    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:32.228019 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.227529    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84690202-b89e-4135-aebb-aa53c0e18463-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:32.228019 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.227543    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:32.228019 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.227556    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:32.228019 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.227774    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-credential-socket" pod "84690202-b89e-4135-aebb-aa53c0e18463" (UID: "84690202-b89e-4135-aebb-aa53c0e18463"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:32.228521 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.228334    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istiod-ca-cert" pod "cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73" (UID: "cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:49:32.228935 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.228600    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-envoy" pod "cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73" (UID: "cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:32.228935 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.228817    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-istio-data" pod "84690202-b89e-4135-aebb-aa53c0e18463" (UID: "84690202-b89e-4135-aebb-aa53c0e18463"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:32.228935 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.228837    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84690202-b89e-4135-aebb-aa53c0e18463-kube-api-access-4hktx" pod "84690202-b89e-4135-aebb-aa53c0e18463" (UID: "84690202-b89e-4135-aebb-aa53c0e18463"). InnerVolumeSpecName "kube-api-access-4hktx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:49:32.228935 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.228892    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-podinfo" pod "cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73" (UID: "cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:49:32.228935 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.228922    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-istio-envoy" pod "84690202-b89e-4135-aebb-aa53c0e18463" (UID: "84690202-b89e-4135-aebb-aa53c0e18463"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:32.228935 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.228921    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-token" pod "cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73" (UID: "cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:49:32.229410 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.229044    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-workload-certs" pod "84690202-b89e-4135-aebb-aa53c0e18463" (UID: "84690202-b89e-4135-aebb-aa53c0e18463"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:32.229410 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.229275    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84690202-b89e-4135-aebb-aa53c0e18463-istio-token" pod "84690202-b89e-4135-aebb-aa53c0e18463" (UID: "84690202-b89e-4135-aebb-aa53c0e18463"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:49:32.230278 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.230256    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84690202-b89e-4135-aebb-aa53c0e18463-istio-podinfo" pod "84690202-b89e-4135-aebb-aa53c0e18463" (UID: "84690202-b89e-4135-aebb-aa53c0e18463"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:49:32.230392 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.230351    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-kube-api-access-xwktw" pod "cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73" (UID: "cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73"). InnerVolumeSpecName "kube-api-access-xwktw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:49:32.329097 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.329064    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:32.329097 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.329093    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:32.329374 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.329106    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:32.329374 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.329119    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:32.329374 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.329129    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hktx\" (UniqueName: \"kubernetes.io/projected/84690202-b89e-4135-aebb-aa53c0e18463-kube-api-access-4hktx\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:32.329374 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.329141    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:32.329374 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.329149    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwktw\" (UniqueName: \"kubernetes.io/projected/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-kube-api-access-xwktw\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:32.329374 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.329159    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:32.329374 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.329168    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84690202-b89e-4135-aebb-aa53c0e18463-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:32.329374 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.329176    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84690202-b89e-4135-aebb-aa53c0e18463-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:32.329374 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.329193    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84690202-b89e-4135-aebb-aa53c0e18463-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:32.329374 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.329204    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:32.742852 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.742811    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn"]
Nov 03 17:49:32.743472 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.743451    6059 cpu_manager.go:414] "containerMap: removing container" podUID="cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73" containerName="istio-proxy"
Nov 03 17:49:32.743472 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.743472    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73" containerName="istio-proxy"
Nov 03 17:49:32.743638 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.743486    6059 cpu_manager.go:414] "containerMap: removing container" podUID="84690202-b89e-4135-aebb-aa53c0e18463" containerName="istio-proxy"
Nov 03 17:49:32.743638 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.743493    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84690202-b89e-4135-aebb-aa53c0e18463" containerName="istio-proxy"
Nov 03 17:49:32.743638 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.743525    6059 cpu_manager.go:414] "containerMap: removing container" podUID="84690202-b89e-4135-aebb-aa53c0e18463" containerName="istio-proxy"
Nov 03 17:49:32.743638 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.743534    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84690202-b89e-4135-aebb-aa53c0e18463" containerName="istio-proxy"
Nov 03 17:49:32.743861 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.743697    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73" containerName="istio-proxy"
Nov 03 17:49:32.743861 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.743710    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="84690202-b89e-4135-aebb-aa53c0e18463" containerName="istio-proxy"
Nov 03 17:49:32.743861 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.743728    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="84690202-b89e-4135-aebb-aa53c0e18463" containerName="istio-proxy"
Nov 03 17:49:32.748437 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.748417    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn"
Nov 03 17:49:32.803410 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.803286    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:49:32.803410 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.803311    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:49:32.803410 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.803378    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:49:32.803634 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.803621    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:49:32.808194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.808168    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn"]
Nov 03 17:49:32.835708 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.835677    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/833d9419-150a-4400-a138-037f3604f62b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecf52bc6-ghpzn\" (UID: \"833d9419-150a-4400-a138-037f3604f62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn"
Nov 03 17:49:32.835871 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.835849    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbn2r\" (UniqueName: \"kubernetes.io/projected/833d9419-150a-4400-a138-037f3604f62b-kube-api-access-fbn2r\") pod \"maas-api-revoke-sub-ecf52bc6-ghpzn\" (UID: \"833d9419-150a-4400-a138-037f3604f62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn"
Nov 03 17:49:32.836036 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.836014    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/833d9419-150a-4400-a138-037f3604f62b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecf52bc6-ghpzn\" (UID: \"833d9419-150a-4400-a138-037f3604f62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn"
Nov 03 17:49:32.847595 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.847563    6059 generic.go:356] "Generic (PLEG): container finished" podID="84690202-b89e-4135-aebb-aa53c0e18463" containerID="2ae27f4cfca9606459d75e02e45f3162805a5e58f43d5339e5dd6c65ffd91371" exitCode=0
Nov 03 17:49:32.847756 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.847618    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"
Nov 03 17:49:32.847756 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.847643    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c" event={"ID":"84690202-b89e-4135-aebb-aa53c0e18463","Type":"ContainerDied","Data":"2ae27f4cfca9606459d75e02e45f3162805a5e58f43d5339e5dd6c65ffd91371"}
Nov 03 17:49:32.847756 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.847705    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c" event={"ID":"84690202-b89e-4135-aebb-aa53c0e18463","Type":"ContainerDied","Data":"6f5419454a6e1264fbce682faf7ff010696c1a998a5c6444eda01bbaa51fc712"}
Nov 03 17:49:32.847756 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.847728    6059 scope.go:122] "RemoveContainer" containerID="2ae27f4cfca9606459d75e02e45f3162805a5e58f43d5339e5dd6c65ffd91371"
Nov 03 17:49:32.849500 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.849461    6059 generic.go:356] "Generic (PLEG): container finished" podID="cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73" containerID="7348867872b9d5e2a56f7bbb4d97654e43557219048d510d805fdc191e1792ac" exitCode=0
Nov 03 17:49:32.849610 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.849504    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj" event={"ID":"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73","Type":"ContainerDied","Data":"7348867872b9d5e2a56f7bbb4d97654e43557219048d510d805fdc191e1792ac"}
Nov 03 17:49:32.849610 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.849544    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj" event={"ID":"cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73","Type":"ContainerDied","Data":"2574dc16995247032318478d0109046ba28458ec892f6df8bbb1e52c6192cc56"}
Nov 03 17:49:32.849610 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.849509    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"
Nov 03 17:49:32.858578 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.858442    6059 scope.go:122] "RemoveContainer" containerID="0582a647eadd190de49d8ba2d75e981eb1985247445fab56d13a22aa2c1975c9"
Nov 03 17:49:32.868069 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.868047    6059 scope.go:122] "RemoveContainer" containerID="2ae27f4cfca9606459d75e02e45f3162805a5e58f43d5339e5dd6c65ffd91371"
Nov 03 17:49:32.868346 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:49:32.868324    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ae27f4cfca9606459d75e02e45f3162805a5e58f43d5339e5dd6c65ffd91371\": container with ID starting with 2ae27f4cfca9606459d75e02e45f3162805a5e58f43d5339e5dd6c65ffd91371 not found: ID does not exist" containerID="2ae27f4cfca9606459d75e02e45f3162805a5e58f43d5339e5dd6c65ffd91371"
Nov 03 17:49:32.868438 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.868371    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ae27f4cfca9606459d75e02e45f3162805a5e58f43d5339e5dd6c65ffd91371"} err="failed to get container status \"2ae27f4cfca9606459d75e02e45f3162805a5e58f43d5339e5dd6c65ffd91371\": rpc error: code = NotFound desc = could not find container \"2ae27f4cfca9606459d75e02e45f3162805a5e58f43d5339e5dd6c65ffd91371\": container with ID starting with 2ae27f4cfca9606459d75e02e45f3162805a5e58f43d5339e5dd6c65ffd91371 not found: ID does not exist"
Nov 03 17:49:32.868438 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.868397    6059 scope.go:122] "RemoveContainer" containerID="0582a647eadd190de49d8ba2d75e981eb1985247445fab56d13a22aa2c1975c9"
Nov 03 17:49:32.868679 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:49:32.868658    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0582a647eadd190de49d8ba2d75e981eb1985247445fab56d13a22aa2c1975c9\": container with ID starting with 0582a647eadd190de49d8ba2d75e981eb1985247445fab56d13a22aa2c1975c9 not found: ID does not exist" containerID="0582a647eadd190de49d8ba2d75e981eb1985247445fab56d13a22aa2c1975c9"
Nov 03 17:49:32.868720 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.868685    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0582a647eadd190de49d8ba2d75e981eb1985247445fab56d13a22aa2c1975c9"} err="failed to get container status \"0582a647eadd190de49d8ba2d75e981eb1985247445fab56d13a22aa2c1975c9\": rpc error: code = NotFound desc = could not find container \"0582a647eadd190de49d8ba2d75e981eb1985247445fab56d13a22aa2c1975c9\": container with ID starting with 0582a647eadd190de49d8ba2d75e981eb1985247445fab56d13a22aa2c1975c9 not found: ID does not exist"
Nov 03 17:49:32.868720 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.868700    6059 scope.go:122] "RemoveContainer" containerID="7348867872b9d5e2a56f7bbb4d97654e43557219048d510d805fdc191e1792ac"
Nov 03 17:49:32.878110 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.878087    6059 scope.go:122] "RemoveContainer" containerID="7348867872b9d5e2a56f7bbb4d97654e43557219048d510d805fdc191e1792ac"
Nov 03 17:49:32.878389 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:49:32.878343    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7348867872b9d5e2a56f7bbb4d97654e43557219048d510d805fdc191e1792ac\": container with ID starting with 7348867872b9d5e2a56f7bbb4d97654e43557219048d510d805fdc191e1792ac not found: ID does not exist" containerID="7348867872b9d5e2a56f7bbb4d97654e43557219048d510d805fdc191e1792ac"
Nov 03 17:49:32.878453 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.878383    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7348867872b9d5e2a56f7bbb4d97654e43557219048d510d805fdc191e1792ac"} err="failed to get container status \"7348867872b9d5e2a56f7bbb4d97654e43557219048d510d805fdc191e1792ac\": rpc error: code = NotFound desc = could not find container \"7348867872b9d5e2a56f7bbb4d97654e43557219048d510d805fdc191e1792ac\": container with ID starting with 7348867872b9d5e2a56f7bbb4d97654e43557219048d510d805fdc191e1792ac not found: ID does not exist"
Nov 03 17:49:32.937533 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.937501    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/833d9419-150a-4400-a138-037f3604f62b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecf52bc6-ghpzn\" (UID: \"833d9419-150a-4400-a138-037f3604f62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn"
Nov 03 17:49:32.937669 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.937555    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/833d9419-150a-4400-a138-037f3604f62b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecf52bc6-ghpzn\" (UID: \"833d9419-150a-4400-a138-037f3604f62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn"
Nov 03 17:49:32.937669 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.937626    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbn2r\" (UniqueName: \"kubernetes.io/projected/833d9419-150a-4400-a138-037f3604f62b-kube-api-access-fbn2r\") pod \"maas-api-revoke-sub-ecf52bc6-ghpzn\" (UID: \"833d9419-150a-4400-a138-037f3604f62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn"
Nov 03 17:49:32.938149 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.938128    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/833d9419-150a-4400-a138-037f3604f62b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecf52bc6-ghpzn\" (UID: \"833d9419-150a-4400-a138-037f3604f62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn"
Nov 03 17:49:32.939893 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.939869    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/833d9419-150a-4400-a138-037f3604f62b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecf52bc6-ghpzn\" (UID: \"833d9419-150a-4400-a138-037f3604f62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn"
Nov 03 17:49:32.986069 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.986030    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"]
Nov 03 17:49:32.998447 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:32.995221    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-hzs8c"]
Nov 03 17:49:33.001175 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.001145    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbn2r\" (UniqueName: \"kubernetes.io/projected/833d9419-150a-4400-a138-037f3604f62b-kube-api-access-fbn2r\") pod \"maas-api-revoke-sub-ecf52bc6-ghpzn\" (UID: \"833d9419-150a-4400-a138-037f3604f62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn"
Nov 03 17:49:33.062196 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.062167    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn"
Nov 03 17:49:33.108013 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.107978    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"]
Nov 03 17:49:33.112019 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.111993    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-kfrwj"]
Nov 03 17:49:33.336445 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.336371    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84690202-b89e-4135-aebb-aa53c0e18463" path="/var/lib/kubelet/pods/84690202-b89e-4135-aebb-aa53c0e18463/volumes"
Nov 03 17:49:33.336858 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.336844    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73" path="/var/lib/kubelet/pods/cc6c7b1f-0d7d-447f-9d3a-5626b83c4b73/volumes"
Nov 03 17:49:33.441956 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:49:33.441920    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833d9419_150a_4400_a138_037f3604f62b.slice/crio-4d8657cc54d456526b2fabf4eb1b69c0f73a62dd124f2a73d35c48b5437ba9cd WatchSource:0}: Error finding container 4d8657cc54d456526b2fabf4eb1b69c0f73a62dd124f2a73d35c48b5437ba9cd: Status 404 returned error can't find the container with id 4d8657cc54d456526b2fabf4eb1b69c0f73a62dd124f2a73d35c48b5437ba9cd
Nov 03 17:49:33.444920 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.444897    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn"]
Nov 03 17:49:33.648260 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.648183    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt" podUID="708810ff-e592-4d83-a6a3-1f2c313499a0" containerName="main" containerID="cri-o://874c51cdc04ee6c22b1588135c49c533183b3bb17c4730bb87bd7fae9ec70167" gracePeriod=45
Nov 03 17:49:33.846206 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.845974    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:33.875513 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.873712    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn" event={"ID":"833d9419-150a-4400-a138-037f3604f62b","Type":"ContainerStarted","Data":"b105c1f4217cc9ab61f03c538d1194b73c74047f6639ac55e43baffe363121f6"}
Nov 03 17:49:33.875513 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.873814    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn" event={"ID":"833d9419-150a-4400-a138-037f3604f62b","Type":"ContainerStarted","Data":"4d8657cc54d456526b2fabf4eb1b69c0f73a62dd124f2a73d35c48b5437ba9cd"}
Nov 03 17:49:33.877677 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.877641    6059 generic.go:356] "Generic (PLEG): container finished" podID="708810ff-e592-4d83-a6a3-1f2c313499a0" containerID="874c51cdc04ee6c22b1588135c49c533183b3bb17c4730bb87bd7fae9ec70167" exitCode=0
Nov 03 17:49:33.877788 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.877747    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt" event={"ID":"708810ff-e592-4d83-a6a3-1f2c313499a0","Type":"ContainerDied","Data":"874c51cdc04ee6c22b1588135c49c533183b3bb17c4730bb87bd7fae9ec70167"}
Nov 03 17:49:33.907862 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.907838    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:49:33.949699 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.949674    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-dshm\") pod \"708810ff-e592-4d83-a6a3-1f2c313499a0\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") "
Nov 03 17:49:33.949833 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.949791    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-home\" (UniqueName: \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-home\") pod \"708810ff-e592-4d83-a6a3-1f2c313499a0\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") "
Nov 03 17:49:33.949879 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.949868    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-model-cache\") pod \"708810ff-e592-4d83-a6a3-1f2c313499a0\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") "
Nov 03 17:49:33.949997 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.949981    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-tmp-dir\") pod \"708810ff-e592-4d83-a6a3-1f2c313499a0\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") "
Nov 03 17:49:33.950043 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.950023    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-home" pod "708810ff-e592-4d83-a6a3-1f2c313499a0" (UID: "708810ff-e592-4d83-a6a3-1f2c313499a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:33.950090 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.950042    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708810ff-e592-4d83-a6a3-1f2c313499a0-tls-certs\" (UniqueName: \"kubernetes.io/secret/708810ff-e592-4d83-a6a3-1f2c313499a0-tls-certs\") pod \"708810ff-e592-4d83-a6a3-1f2c313499a0\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") "
Nov 03 17:49:33.950152 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.950097    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708810ff-e592-4d83-a6a3-1f2c313499a0-kube-api-access-tngcz\" (UniqueName: \"kubernetes.io/projected/708810ff-e592-4d83-a6a3-1f2c313499a0-kube-api-access-tngcz\") pod \"708810ff-e592-4d83-a6a3-1f2c313499a0\" (UID: \"708810ff-e592-4d83-a6a3-1f2c313499a0\") "
Nov 03 17:49:33.950152 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.950130    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-model-cache" pod "708810ff-e592-4d83-a6a3-1f2c313499a0" (UID: "708810ff-e592-4d83-a6a3-1f2c313499a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:33.950265 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.950244    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-tmp-dir" pod "708810ff-e592-4d83-a6a3-1f2c313499a0" (UID: "708810ff-e592-4d83-a6a3-1f2c313499a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:33.950555 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.950533    6059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-home\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:33.950693 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.950558    6059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-model-cache\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:33.950693 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.950572    6059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-tmp-dir\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:33.951875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.951851    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-dshm" pod "708810ff-e592-4d83-a6a3-1f2c313499a0" (UID: "708810ff-e592-4d83-a6a3-1f2c313499a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:33.952127 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.952107    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708810ff-e592-4d83-a6a3-1f2c313499a0-tls-certs" pod "708810ff-e592-4d83-a6a3-1f2c313499a0" (UID: "708810ff-e592-4d83-a6a3-1f2c313499a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:49:33.952214 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.952170    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708810ff-e592-4d83-a6a3-1f2c313499a0-kube-api-access-tngcz" pod "708810ff-e592-4d83-a6a3-1f2c313499a0" (UID: "708810ff-e592-4d83-a6a3-1f2c313499a0"). InnerVolumeSpecName "kube-api-access-tngcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:49:33.999526 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:33.999466    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn" podStartSLOduration=1.999447881 podStartE2EDuration="1.999447881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:33.99609304 +0000 UTC m=+1914.793805820" watchObservedRunningTime="2026-11-03 17:49:33.999447881 +0000 UTC m=+1914.797160664"
Nov 03 17:49:34.051457 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:34.051425    6059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/708810ff-e592-4d83-a6a3-1f2c313499a0-tls-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:34.051457 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:34.051452    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tngcz\" (UniqueName: \"kubernetes.io/projected/708810ff-e592-4d83-a6a3-1f2c313499a0-kube-api-access-tngcz\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:34.051457 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:34.051462    6059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/708810ff-e592-4d83-a6a3-1f2c313499a0-dshm\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:34.890294 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:34.890251    6059 generic.go:356] "Generic (PLEG): container finished" podID="833d9419-150a-4400-a138-037f3604f62b" containerID="b105c1f4217cc9ab61f03c538d1194b73c74047f6639ac55e43baffe363121f6" exitCode=0
Nov 03 17:49:34.890753 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:34.890330    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn" event={"ID":"833d9419-150a-4400-a138-037f3604f62b","Type":"ContainerDied","Data":"b105c1f4217cc9ab61f03c538d1194b73c74047f6639ac55e43baffe363121f6"}
Nov 03 17:49:34.892094 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:34.892062    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt" event={"ID":"708810ff-e592-4d83-a6a3-1f2c313499a0","Type":"ContainerDied","Data":"6c3461bc3dc044501e1c5d7fbdf2bb28f8e57898cba9f7e8fa942913fbc00ac7"}
Nov 03 17:49:34.892211 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:34.892112    6059 scope.go:122] "RemoveContainer" containerID="874c51cdc04ee6c22b1588135c49c533183b3bb17c4730bb87bd7fae9ec70167"
Nov 03 17:49:34.892211 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:34.892073    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"
Nov 03 17:49:35.130626 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:35.130591    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"]
Nov 03 17:49:35.133702 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:35.133676    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af9e3c/sub-test-model-af9e3c-kserve-65d9f465d7-p82jt"]
Nov 03 17:49:35.336339 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:35.336311    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708810ff-e592-4d83-a6a3-1f2c313499a0" path="/var/lib/kubelet/pods/708810ff-e592-4d83-a6a3-1f2c313499a0/volumes"
Nov 03 17:49:35.401851 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:35.401799    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:36.055430 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.055405    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn"
Nov 03 17:49:36.071468 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.071426    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:36.174630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.174549    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/833d9419-150a-4400-a138-037f3604f62b-kube-api-access-fbn2r\" (UniqueName: \"kubernetes.io/projected/833d9419-150a-4400-a138-037f3604f62b-kube-api-access-fbn2r\") pod \"833d9419-150a-4400-a138-037f3604f62b\" (UID: \"833d9419-150a-4400-a138-037f3604f62b\") "
Nov 03 17:49:36.174630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.174595    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/833d9419-150a-4400-a138-037f3604f62b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/833d9419-150a-4400-a138-037f3604f62b-maas-api-openshift-ca\") pod \"833d9419-150a-4400-a138-037f3604f62b\" (UID: \"833d9419-150a-4400-a138-037f3604f62b\") "
Nov 03 17:49:36.174838 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.174723    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/833d9419-150a-4400-a138-037f3604f62b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/833d9419-150a-4400-a138-037f3604f62b-maas-api-xks-ca\") pod \"833d9419-150a-4400-a138-037f3604f62b\" (UID: \"833d9419-150a-4400-a138-037f3604f62b\") "
Nov 03 17:49:36.174948 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.174925    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/833d9419-150a-4400-a138-037f3604f62b-maas-api-openshift-ca" pod "833d9419-150a-4400-a138-037f3604f62b" (UID: "833d9419-150a-4400-a138-037f3604f62b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:49:36.175211 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.175191    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/833d9419-150a-4400-a138-037f3604f62b-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:36.176740 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.176708    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833d9419-150a-4400-a138-037f3604f62b-kube-api-access-fbn2r" pod "833d9419-150a-4400-a138-037f3604f62b" (UID: "833d9419-150a-4400-a138-037f3604f62b"). InnerVolumeSpecName "kube-api-access-fbn2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:49:36.176828 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.176758    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/833d9419-150a-4400-a138-037f3604f62b-maas-api-xks-ca" pod "833d9419-150a-4400-a138-037f3604f62b" (UID: "833d9419-150a-4400-a138-037f3604f62b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:49:36.276452 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.276419    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbn2r\" (UniqueName: \"kubernetes.io/projected/833d9419-150a-4400-a138-037f3604f62b-kube-api-access-fbn2r\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:36.276452 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.276450    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/833d9419-150a-4400-a138-037f3604f62b-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:36.664588 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.664557    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"]
Nov 03 17:49:36.665124 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.665110    6059 cpu_manager.go:414] "containerMap: removing container" podUID="833d9419-150a-4400-a138-037f3604f62b" containerName="revoke-keys"
Nov 03 17:49:36.665169 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.665125    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="833d9419-150a-4400-a138-037f3604f62b" containerName="revoke-keys"
Nov 03 17:49:36.665169 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.665155    6059 cpu_manager.go:414] "containerMap: removing container" podUID="708810ff-e592-4d83-a6a3-1f2c313499a0" containerName="main"
Nov 03 17:49:36.665169 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.665161    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708810ff-e592-4d83-a6a3-1f2c313499a0" containerName="main"
Nov 03 17:49:36.665272 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.665248    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="708810ff-e592-4d83-a6a3-1f2c313499a0" containerName="main"
Nov 03 17:49:36.665272 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.665263    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="833d9419-150a-4400-a138-037f3604f62b" containerName="revoke-keys"
Nov 03 17:49:36.668702 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.668684    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.745409 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.743418    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-1f6be0-openshift-default-dockercfg-5g2hb\""
Nov 03 17:49:36.784648 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.782334    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-envoy\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.784648 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.782414    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-credential-socket\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.784648 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.782453    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-workload-certs\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.784648 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.782476    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-podinfo\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.784648 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.782563    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5789\" (UniqueName: \"kubernetes.io/projected/dfcde397-4b17-48a2-bd2b-ad43fd73be72-kube-api-access-w5789\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.784648 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.782591    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-data\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.784648 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.782609    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-token\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.784648 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.782661    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-workload-socket\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.784648 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.782698    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istiod-ca-cert\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.798627 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.798602    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"]
Nov 03 17:49:36.799186 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.799158    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:36.804631 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.804609    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb"
Nov 03 17:49:36.804742 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.804689    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c"
Nov 03 17:49:36.861727 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:49:36.861691    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:49:36.884374 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.884327    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-credential-socket\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.884539 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.884404    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-workload-certs\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.884539 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.884446    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-podinfo\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.885711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.884759    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-credential-socket\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.885711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.884875    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5789\" (UniqueName: \"kubernetes.io/projected/dfcde397-4b17-48a2-bd2b-ad43fd73be72-kube-api-access-w5789\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.885711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.884942    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-data\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.885711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.884972    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-token\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.885711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.885063    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-workload-socket\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.885711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.885125    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istiod-ca-cert\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.885711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.885190    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-envoy\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.885711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.885284    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-workload-certs\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.885711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.885619    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-workload-socket\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.885711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.885630    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-data\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.886202 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.886050    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istiod-ca-cert\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.888432 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.888047    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-podinfo\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.894630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.894601    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-envoy\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.912328 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.912294    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn" event={"ID":"833d9419-150a-4400-a138-037f3604f62b","Type":"ContainerDied","Data":"4d8657cc54d456526b2fabf4eb1b69c0f73a62dd124f2a73d35c48b5437ba9cd"}
Nov 03 17:49:36.912538 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.912347    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d8657cc54d456526b2fabf4eb1b69c0f73a62dd124f2a73d35c48b5437ba9cd"
Nov 03 17:49:36.912538 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.912392    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn"
Nov 03 17:49:36.964381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.953078    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5789\" (UniqueName: \"kubernetes.io/projected/dfcde397-4b17-48a2-bd2b-ad43fd73be72-kube-api-access-w5789\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.964381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.953566    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-token\") pod \"e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") " pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:36.984328 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:36.984289    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:37.009972 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:37.009942    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:37.451618 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:49:37.451584    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfcde397_4b17_48a2_bd2b_ad43fd73be72.slice/crio-fd866ffdff88095f2556d379a19b092ac228d54a3e4b0a7d7dd28d57fc50cd4e WatchSource:0}: Error finding container fd866ffdff88095f2556d379a19b092ac228d54a3e4b0a7d7dd28d57fc50cd4e: Status 404 returned error can't find the container with id fd866ffdff88095f2556d379a19b092ac228d54a3e4b0a7d7dd28d57fc50cd4e
Nov 03 17:49:37.454543 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:37.454521    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"]
Nov 03 17:49:37.454874 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:37.454840    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:49:37.454941 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:37.454908    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:49:37.454996 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:37.454948    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:49:37.922135 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:37.922093    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz" event={"ID":"dfcde397-4b17-48a2-bd2b-ad43fd73be72","Type":"ContainerStarted","Data":"cf5c553e8a38eee6abd5a79482b43d62d5af0e6cff4f2b308c39a055e249dee3"}
Nov 03 17:49:37.922327 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:37.922188    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz" event={"ID":"dfcde397-4b17-48a2-bd2b-ad43fd73be72","Type":"ContainerStarted","Data":"fd866ffdff88095f2556d379a19b092ac228d54a3e4b0a7d7dd28d57fc50cd4e"}
Nov 03 17:49:38.010658 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:38.010629    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:38.016010 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:38.015986    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:38.052449 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:38.052407    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz" podStartSLOduration=2.052392596 podStartE2EDuration="2.052392596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:38.047554817 +0000 UTC m=+1918.845267598" watchObservedRunningTime="2026-11-03 17:49:38.052392596 +0000 UTC m=+1918.850105379"
Nov 03 17:49:38.429132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:38.429098    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:38.929496 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:38.929464    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:38.930235 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:38.930216    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:49:40.984084 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:40.984053    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:41.186038 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:41.185998    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:41.377808 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:41.377728    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:41.999773 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:41.999743    6059 scope.go:122] "RemoveContainer" containerID="183c6e29633e9715feb25b87f542a115f615c63c2ba6f8fe999007baeaf3eddf"
Nov 03 17:49:43.166742 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:43.166700    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:43.467574 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:43.467543    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:44.069793 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.069758    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:44.248809 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.248772    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:44.507592 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.507548    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q"]
Nov 03 17:49:44.513949 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.513929    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q"
Nov 03 17:49:44.572313 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.572279    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q"]
Nov 03 17:49:44.615074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.615030    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137627b9-a30d-4ff1-a8ca-5118bed82be1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q\" (UID: \"137627b9-a30d-4ff1-a8ca-5118bed82be1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q"
Nov 03 17:49:44.615293 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.615107    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw8gv\" (UniqueName: \"kubernetes.io/projected/137627b9-a30d-4ff1-a8ca-5118bed82be1-kube-api-access-tw8gv\") pod \"payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q\" (UID: \"137627b9-a30d-4ff1-a8ca-5118bed82be1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q"
Nov 03 17:49:44.621754 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.621725    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-1f6be0-dockercfg-p2zlk\""
Nov 03 17:49:44.624455 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.624398    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-1f6be0\""
Nov 03 17:49:44.630737 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.630708    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh"]
Nov 03 17:49:44.651436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.650244    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh"
Nov 03 17:49:44.698577 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.697851    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh"]
Nov 03 17:49:44.716292 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.716251    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137627b9-a30d-4ff1-a8ca-5118bed82be1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q\" (UID: \"137627b9-a30d-4ff1-a8ca-5118bed82be1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q"
Nov 03 17:49:44.716476 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.716326    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw8gv\" (UniqueName: \"kubernetes.io/projected/137627b9-a30d-4ff1-a8ca-5118bed82be1-kube-api-access-tw8gv\") pod \"payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q\" (UID: \"137627b9-a30d-4ff1-a8ca-5118bed82be1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q"
Nov 03 17:49:44.716995 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.716975    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137627b9-a30d-4ff1-a8ca-5118bed82be1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q\" (UID: \"137627b9-a30d-4ff1-a8ca-5118bed82be1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q"
Nov 03 17:49:44.779904 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.779835    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw8gv\" (UniqueName: \"kubernetes.io/projected/137627b9-a30d-4ff1-a8ca-5118bed82be1-kube-api-access-tw8gv\") pod \"payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q\" (UID: \"137627b9-a30d-4ff1-a8ca-5118bed82be1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q"
Nov 03 17:49:44.817237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.817206    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc016fa5-62ce-4281-9e0b-488aab664638-plugins-config-volume\") pod \"payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh\" (UID: \"cc016fa5-62ce-4281-9e0b-488aab664638\") " pod="openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh"
Nov 03 17:49:44.817411 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.817259    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpczl\" (UniqueName: \"kubernetes.io/projected/cc016fa5-62ce-4281-9e0b-488aab664638-kube-api-access-gpczl\") pod \"payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh\" (UID: \"cc016fa5-62ce-4281-9e0b-488aab664638\") " pod="openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh"
Nov 03 17:49:44.827908 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.827886    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q"
Nov 03 17:49:44.919053 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.918148    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpczl\" (UniqueName: \"kubernetes.io/projected/cc016fa5-62ce-4281-9e0b-488aab664638-kube-api-access-gpczl\") pod \"payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh\" (UID: \"cc016fa5-62ce-4281-9e0b-488aab664638\") " pod="openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh"
Nov 03 17:49:44.919053 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.918314    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc016fa5-62ce-4281-9e0b-488aab664638-plugins-config-volume\") pod \"payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh\" (UID: \"cc016fa5-62ce-4281-9e0b-488aab664638\") " pod="openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh"
Nov 03 17:49:44.919053 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.918981    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc016fa5-62ce-4281-9e0b-488aab664638-plugins-config-volume\") pod \"payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh\" (UID: \"cc016fa5-62ce-4281-9e0b-488aab664638\") " pod="openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh"
Nov 03 17:49:44.982287 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:44.982254    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpczl\" (UniqueName: \"kubernetes.io/projected/cc016fa5-62ce-4281-9e0b-488aab664638-kube-api-access-gpczl\") pod \"payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh\" (UID: \"cc016fa5-62ce-4281-9e0b-488aab664638\") " pod="openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh"
Nov 03 17:49:45.202696 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:49:45.202653    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137627b9_a30d_4ff1_a8ca_5118bed82be1.slice/crio-1470d3cc92479042c3157ec0054bbd0554e1b1abf9eddd9b5f6170cea6ed1a73 WatchSource:0}: Error finding container 1470d3cc92479042c3157ec0054bbd0554e1b1abf9eddd9b5f6170cea6ed1a73: Status 404 returned error can't find the container with id 1470d3cc92479042c3157ec0054bbd0554e1b1abf9eddd9b5f6170cea6ed1a73
Nov 03 17:49:45.205211 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:45.205184    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q"]
Nov 03 17:49:45.276692 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:45.276664    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh"
Nov 03 17:49:45.655476 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:49:45.655441    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc016fa5_62ce_4281_9e0b_488aab664638.slice/crio-efca46464edc8fae3f41e44a06c580cb7295df03e3590f8fddb0ca131551ea8a WatchSource:0}: Error finding container efca46464edc8fae3f41e44a06c580cb7295df03e3590f8fddb0ca131551ea8a: Status 404 returned error can't find the container with id efca46464edc8fae3f41e44a06c580cb7295df03e3590f8fddb0ca131551ea8a
Nov 03 17:49:45.660960 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:45.660931    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh"]
Nov 03 17:49:45.980485 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:45.980448    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q" event={"ID":"137627b9-a30d-4ff1-a8ca-5118bed82be1","Type":"ContainerStarted","Data":"321c01d68d840729997e2c9c7f146a1c636c9154b218471c3bc1431306d984f3"}
Nov 03 17:49:45.980691 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:45.980511    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q" event={"ID":"137627b9-a30d-4ff1-a8ca-5118bed82be1","Type":"ContainerStarted","Data":"1470d3cc92479042c3157ec0054bbd0554e1b1abf9eddd9b5f6170cea6ed1a73"}
Nov 03 17:49:45.980691 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:45.980667    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q"
Nov 03 17:49:45.982138 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:45.982110    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh" event={"ID":"cc016fa5-62ce-4281-9e0b-488aab664638","Type":"ContainerStarted","Data":"4a7f57275dfef1e74f8aa8c8a714ec46389970ec97e0c0d484928b2bae1aa578"}
Nov 03 17:49:45.982233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:45.982158    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh"
Nov 03 17:49:45.982233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:45.982178    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh" event={"ID":"cc016fa5-62ce-4281-9e0b-488aab664638","Type":"ContainerStarted","Data":"efca46464edc8fae3f41e44a06c580cb7295df03e3590f8fddb0ca131551ea8a"}
Nov 03 17:49:46.106170 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:46.106117    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q" podStartSLOduration=2.106103458 podStartE2EDuration="2.106103458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:46.103273191 +0000 UTC m=+1926.900985971" watchObservedRunningTime="2026-11-03 17:49:46.106103458 +0000 UTC m=+1926.903816239"
Nov 03 17:49:46.230127 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:46.230067    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh" podStartSLOduration=2.230050651 podStartE2EDuration="2.230050651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:46.224790482 +0000 UTC m=+1927.022503279" watchObservedRunningTime="2026-11-03 17:49:46.230050651 +0000 UTC m=+1927.027763431"
Nov 03 17:49:47.186666 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:47.186633    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:47.392100 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:47.392063    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:48.222829 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:48.222797    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:48.407379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:48.407332    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:48.598251 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:48.598162    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:50.121592 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:50.121538    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:53.519398 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:53.519351    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:49:56.992541 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:56.992512    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh"
Nov 03 17:49:56.992942 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:56.992931    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q"
Nov 03 17:49:58.138885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:49:58.138843    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:02.742532 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:02.742495    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:05.253659 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:05.253621    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:06.861904 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:50:06.861870    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:50:07.342623 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:07.342591    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:11.850725 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:11.850686    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:16.412927 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:16.412873    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:16.620146 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:16.620106    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:21.214569 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:21.214533    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:24.661305 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:24.661272    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:24.835543 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:24.835509    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:25.823859 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:25.823823    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:30.569459 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:30.569308    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:35.333630 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:35.333590    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:35.888802 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:35.888757    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:36.799155 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:36.799103    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:36.861309 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:50:36.861263    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:50:37.365586 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.365546    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"]
Nov 03 17:50:37.370857 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.370829    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.432437 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.427291    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e095fe-openshift-default-dockercfg-lcjc7\""
Nov 03 17:50:37.440056 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.440014    6059 generic.go:356] "Generic (PLEG): container finished" podID="dfcde397-4b17-48a2-bd2b-ad43fd73be72" containerID="cf5c553e8a38eee6abd5a79482b43d62d5af0e6cff4f2b308c39a055e249dee3" exitCode=0
Nov 03 17:50:37.440258 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.440132    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz" event={"ID":"dfcde397-4b17-48a2-bd2b-ad43fd73be72","Type":"ContainerDied","Data":"cf5c553e8a38eee6abd5a79482b43d62d5af0e6cff4f2b308c39a055e249dee3"}
Nov 03 17:50:37.440699 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.440674    6059 scope.go:122] "RemoveContainer" containerID="cf5c553e8a38eee6abd5a79482b43d62d5af0e6cff4f2b308c39a055e249dee3"
Nov 03 17:50:37.442510 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.442478    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:50:37.442657 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.442569    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:50:37.442657 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.442617    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:50:37.454667 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.454626    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c9e388-cf4f-4f23-98df-b36b39615609-istiod-ca-cert\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.455001 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.454978    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-envoy\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.455175 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.455160    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-workload-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.455372 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.455344    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-token\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.455633 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.455610    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-data\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.455829 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.455806    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-workload-certs\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.455962 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.455841    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-podinfo\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.455962 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.455912    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c72bq\" (UniqueName: \"kubernetes.io/projected/f7c9e388-cf4f-4f23-98df-b36b39615609-kube-api-access-c72bq\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.455962 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.455954    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-credential-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.486615 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.486407    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"]
Nov 03 17:50:37.501463 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.501340    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:37.558686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.556674    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-token\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.558686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.556761    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-data\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.558686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.556791    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-workload-certs\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.558686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.556812    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-podinfo\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.558686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.556852    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c72bq\" (UniqueName: \"kubernetes.io/projected/f7c9e388-cf4f-4f23-98df-b36b39615609-kube-api-access-c72bq\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.558686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.556884    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-credential-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.558686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.556928    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c9e388-cf4f-4f23-98df-b36b39615609-istiod-ca-cert\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.558686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.556975    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-envoy\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.558686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.556996    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-workload-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.558686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.557842    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-credential-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.558686 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.558276    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c9e388-cf4f-4f23-98df-b36b39615609-istiod-ca-cert\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.560131 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.558963    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-workload-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.560131 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.559748    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-data\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.560131 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.559981    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-workload-certs\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.564236 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.564201    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-podinfo\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.564980 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.564935    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-envoy\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.631789 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.625265    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c72bq\" (UniqueName: \"kubernetes.io/projected/f7c9e388-cf4f-4f23-98df-b36b39615609-kube-api-access-c72bq\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.647959 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.647922    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-token\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.717237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.717204    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:37.718383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:37.718338    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:38.191261 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:50:38.191223    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c9e388_cf4f_4f23_98df_b36b39615609.slice/crio-5f73f1db7ab92417ce12092b30c4ce6e3cbe1e104b359e930fff353e361a71af WatchSource:0}: Error finding container 5f73f1db7ab92417ce12092b30c4ce6e3cbe1e104b359e930fff353e361a71af: Status 404 returned error can't find the container with id 5f73f1db7ab92417ce12092b30c4ce6e3cbe1e104b359e930fff353e361a71af
Nov 03 17:50:38.194192 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:38.194153    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:50:38.194300 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:38.194241    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:50:38.194300 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:38.194284    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:50:38.195864 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:38.195841    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"]
Nov 03 17:50:38.452914 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:38.452801    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf" event={"ID":"f7c9e388-cf4f-4f23-98df-b36b39615609","Type":"ContainerStarted","Data":"a487c363356bd9016270163279b78a2674899f8b1cfb595e014595c2b2b37682"}
Nov 03 17:50:38.452914 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:38.452868    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf" event={"ID":"f7c9e388-cf4f-4f23-98df-b36b39615609","Type":"ContainerStarted","Data":"5f73f1db7ab92417ce12092b30c4ce6e3cbe1e104b359e930fff353e361a71af"}
Nov 03 17:50:38.455155 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:38.455130    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz" event={"ID":"dfcde397-4b17-48a2-bd2b-ad43fd73be72","Type":"ContainerStarted","Data":"e36dff831e947032b996235a502c0eb80a523879c7fc534ff698b40bea77654a"}
Nov 03 17:50:38.585780 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:38.582970    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf" podStartSLOduration=1.582952021 podStartE2EDuration="1.582952021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:38.576768798 +0000 UTC m=+1979.374481570" watchObservedRunningTime="2026-11-03 17:50:38.582952021 +0000 UTC m=+1979.380664803"
Nov 03 17:50:38.719525 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:38.719488    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:38.951381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:38.935465    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:50:39.011097 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:39.011017    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:50:39.012383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:39.012332    6059 patch_prober.go:29] interesting pod/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 03 17:50:39.012506 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:39.012436    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz" podUID="dfcde397-4b17-48a2-bd2b-ad43fd73be72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 03 17:50:39.723893 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:39.723850    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:39.965233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:39.965199    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:40.011166 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:40.011075    6059 patch_prober.go:29] interesting pod/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 03 17:50:40.011166 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:40.011139    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz" podUID="dfcde397-4b17-48a2-bd2b-ad43fd73be72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 03 17:50:40.471082 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:40.471045    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:40.471914 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:40.471894    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:50:41.010456 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:41.010420    6059 patch_prober.go:29] interesting pod/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 03 17:50:41.010962 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:41.010480    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz" podUID="dfcde397-4b17-48a2-bd2b-ad43fd73be72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 03 17:50:41.477761 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:41.477726    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:41.694811 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:41.694771    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:42.011021 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:42.010990    6059 patch_prober.go:29] interesting pod/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 03 17:50:42.011436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:42.011045    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz" podUID="dfcde397-4b17-48a2-bd2b-ad43fd73be72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 03 17:50:42.070839 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:42.070806    6059 scope.go:122] "RemoveContainer" containerID="0525af719d6fce866705e8e7a281407d02fb324b0b25ff07a1ddf7d0202cac0c"
Nov 03 17:50:42.081488 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:42.081468    6059 scope.go:122] "RemoveContainer" containerID="806f25f785ba4c98c7f914fc385fee5612d7cba55bc765363229fd99d58883ff"
Nov 03 17:50:43.010542 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:43.010502    6059 patch_prober.go:29] interesting pod/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 03 17:50:43.010730 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:43.010563    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz" podUID="dfcde397-4b17-48a2-bd2b-ad43fd73be72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 03 17:50:44.011095 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:44.011058    6059 patch_prober.go:29] interesting pod/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 03 17:50:44.011504 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:44.011132    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz" podUID="dfcde397-4b17-48a2-bd2b-ad43fd73be72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 03 17:50:44.550576 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:44.550542    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:44.726085 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:44.726047    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:44.922596 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:44.922522    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc"]
Nov 03 17:50:44.928911 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:44.928890    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc"
Nov 03 17:50:44.988610 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:44.988578    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc"]
Nov 03 17:50:45.015423 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.015389    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:50:45.016754 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.016727    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:50:45.020139 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.020110    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:45.037924 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.037891    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e095fe\""
Nov 03 17:50:45.038250 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.038231    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e095fe-dockercfg-d7rv9\""
Nov 03 17:50:45.097792 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.097684    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nxcj\" (UniqueName: \"kubernetes.io/projected/fa25bbe8-398d-4b88-a513-a1db1a39ace9-kube-api-access-5nxcj\") pod \"payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc\" (UID: \"fa25bbe8-398d-4b88-a513-a1db1a39ace9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc"
Nov 03 17:50:45.098197 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.098174    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa25bbe8-398d-4b88-a513-a1db1a39ace9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc\" (UID: \"fa25bbe8-398d-4b88-a513-a1db1a39ace9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc"
Nov 03 17:50:45.130299 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.130229    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn"]
Nov 03 17:50:45.136609 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.136579    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn"
Nov 03 17:50:45.147495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.147463    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn"]
Nov 03 17:50:45.199915 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.199821    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa25bbe8-398d-4b88-a513-a1db1a39ace9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc\" (UID: \"fa25bbe8-398d-4b88-a513-a1db1a39ace9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc"
Nov 03 17:50:45.200091 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.200064    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nxcj\" (UniqueName: \"kubernetes.io/projected/fa25bbe8-398d-4b88-a513-a1db1a39ace9-kube-api-access-5nxcj\") pod \"payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc\" (UID: \"fa25bbe8-398d-4b88-a513-a1db1a39ace9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc"
Nov 03 17:50:45.200676 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.200638    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa25bbe8-398d-4b88-a513-a1db1a39ace9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc\" (UID: \"fa25bbe8-398d-4b88-a513-a1db1a39ace9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc"
Nov 03 17:50:45.263089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.263057    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nxcj\" (UniqueName: \"kubernetes.io/projected/fa25bbe8-398d-4b88-a513-a1db1a39ace9-kube-api-access-5nxcj\") pod \"payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc\" (UID: \"fa25bbe8-398d-4b88-a513-a1db1a39ace9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc"
Nov 03 17:50:45.301089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.301051    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nskrv\" (UniqueName: \"kubernetes.io/projected/93cfc7b5-27fd-4c80-849f-a331f4e6f7c9-kube-api-access-nskrv\") pod \"payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn\" (UID: \"93cfc7b5-27fd-4c80-849f-a331f4e6f7c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn"
Nov 03 17:50:45.301265 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.301103    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93cfc7b5-27fd-4c80-849f-a331f4e6f7c9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn\" (UID: \"93cfc7b5-27fd-4c80-849f-a331f4e6f7c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn"
Nov 03 17:50:45.402032 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.401997    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nskrv\" (UniqueName: \"kubernetes.io/projected/93cfc7b5-27fd-4c80-849f-a331f4e6f7c9-kube-api-access-nskrv\") pod \"payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn\" (UID: \"93cfc7b5-27fd-4c80-849f-a331f4e6f7c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn"
Nov 03 17:50:45.402213 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.402052    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93cfc7b5-27fd-4c80-849f-a331f4e6f7c9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn\" (UID: \"93cfc7b5-27fd-4c80-849f-a331f4e6f7c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn"
Nov 03 17:50:45.402804 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.402782    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93cfc7b5-27fd-4c80-849f-a331f4e6f7c9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn\" (UID: \"93cfc7b5-27fd-4c80-849f-a331f4e6f7c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn"
Nov 03 17:50:45.464956 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.464929    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nskrv\" (UniqueName: \"kubernetes.io/projected/93cfc7b5-27fd-4c80-849f-a331f4e6f7c9-kube-api-access-nskrv\") pod \"payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn\" (UID: \"93cfc7b5-27fd-4c80-849f-a331f4e6f7c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn"
Nov 03 17:50:45.542887 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.542850    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc"
Nov 03 17:50:45.755093 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.755006    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn"
Nov 03 17:50:45.917302 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:50:45.917272    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa25bbe8_398d_4b88_a513_a1db1a39ace9.slice/crio-aecc944de54ddd4465975f9a902006b4361b90517815d74b5688c73a14da72df WatchSource:0}: Error finding container aecc944de54ddd4465975f9a902006b4361b90517815d74b5688c73a14da72df: Status 404 returned error can't find the container with id aecc944de54ddd4465975f9a902006b4361b90517815d74b5688c73a14da72df
Nov 03 17:50:45.920846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:45.920819    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc"]
Nov 03 17:50:46.146167 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:50:46.146136    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93cfc7b5_27fd_4c80_849f_a331f4e6f7c9.slice/crio-db4474fc2c5bd840efc5275c8a57439b8f95f39140d41e482a8b96f6a7c5659c WatchSource:0}: Error finding container db4474fc2c5bd840efc5275c8a57439b8f95f39140d41e482a8b96f6a7c5659c: Status 404 returned error can't find the container with id db4474fc2c5bd840efc5275c8a57439b8f95f39140d41e482a8b96f6a7c5659c
Nov 03 17:50:46.150015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:46.149993    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn"]
Nov 03 17:50:46.513671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:46.513633    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn" event={"ID":"93cfc7b5-27fd-4c80-849f-a331f4e6f7c9","Type":"ContainerStarted","Data":"56e611a62716fad918dd966d295f7eecb490bf2d7fd3a6e717988112f9cd8aa6"}
Nov 03 17:50:46.513916 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:46.513697    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn" event={"ID":"93cfc7b5-27fd-4c80-849f-a331f4e6f7c9","Type":"ContainerStarted","Data":"db4474fc2c5bd840efc5275c8a57439b8f95f39140d41e482a8b96f6a7c5659c"}
Nov 03 17:50:46.513916 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:46.513841    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn"
Nov 03 17:50:46.515372 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:46.515337    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc" event={"ID":"fa25bbe8-398d-4b88-a513-a1db1a39ace9","Type":"ContainerStarted","Data":"115f83e568a23a1024bcae53ff2d88132cff7e1f60d8a0b1f059ac3f8217ea0b"}
Nov 03 17:50:46.515500 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:46.515385    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc" event={"ID":"fa25bbe8-398d-4b88-a513-a1db1a39ace9","Type":"ContainerStarted","Data":"aecc944de54ddd4465975f9a902006b4361b90517815d74b5688c73a14da72df"}
Nov 03 17:50:46.515500 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:46.515406    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc"
Nov 03 17:50:46.635623 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:46.635562    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn" podStartSLOduration=1.635546625 podStartE2EDuration="1.635546625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:50:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:46.633471438 +0000 UTC m=+1987.431184220" watchObservedRunningTime="2026-11-03 17:50:46.635546625 +0000 UTC m=+1987.433259422"
Nov 03 17:50:46.755823 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:46.755767    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc" podStartSLOduration=2.755751933 podStartE2EDuration="2.755751933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:46.755414516 +0000 UTC m=+1987.553127297" watchObservedRunningTime="2026-11-03 17:50:46.755751933 +0000 UTC m=+1987.553464714"
Nov 03 17:50:49.643695 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:49.643651    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:54.121656 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:54.121624    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:50:57.525631 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:57.525597    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn"
Nov 03 17:50:57.526034 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:57.525662    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc"
Nov 03 17:50:58.731394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:50:58.731317    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:02.430495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:02.430462    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:03.327316 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:03.327249    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:06.127197 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:06.127162    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:06.861250 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:51:06.861206    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:51:07.968306 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:07.968268    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:08.069832 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:08.069787    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx"]
Nov 03 17:51:08.095952 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:08.095916    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f3e4acc-wqgxx"]
Nov 03 17:51:09.336272 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:09.336237    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60403592-f827-4ff1-856a-47ccc007eb09" path="/var/lib/kubelet/pods/60403592-f827-4ff1-856a-47ccc007eb09/volumes"
Nov 03 17:51:12.540596 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:12.540556    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:14.035388 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:14.035324    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:14.277981 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:14.277942    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:17.403943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:17.403904    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:17.610748 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:17.610710    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:17.829405 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:17.829371    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:19.053061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:19.053023    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b5a3c08-mhqqf"]
Nov 03 17:51:19.104484 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:19.104445    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b5a3c08-mhqqf"]
Nov 03 17:51:19.336784 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:19.336694    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a115b104-1420-446c-aa9e-5d426677627f" path="/var/lib/kubelet/pods/a115b104-1420-446c-aa9e-5d426677627f/volumes"
Nov 03 17:51:22.059702 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:22.059651    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:24.965967 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:24.965929    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:25.303616 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:25.303526    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:26.751316 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:26.751276    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:31.059453 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:31.059416    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl"]
Nov 03 17:51:31.105025 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:31.104991    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd65e5f-5hwsl"]
Nov 03 17:51:31.359608 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:31.359522    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0b259b1-95ce-4d47-9aa3-9021be99e165" path="/var/lib/kubelet/pods/d0b259b1-95ce-4d47-9aa3-9021be99e165/volumes"
Nov 03 17:51:31.438488 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:31.429893    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:35.903006 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:35.902947    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:36.861461 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:51:36.861424    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:51:37.998016 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:37.997975    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"]
Nov 03 17:51:38.003840 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.003811    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.029771 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.029730    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-workload-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.029973 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.029789    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-credential-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.029973 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.029880    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv9x4\" (UniqueName: \"kubernetes.io/projected/4447e11f-6bed-4da8-97b6-c331cd08a4eb-kube-api-access-kv9x4\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.030122 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.030008    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istiod-ca-cert\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.030122 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.030048    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-envoy\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.030261 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.030245    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-token\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.030333 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.030282    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-data\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.030333 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.030313    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-podinfo\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.030558 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.030459    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-workload-certs\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.132078 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.132041    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-token\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.132311 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.132087    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-data\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.132311 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.132114    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-podinfo\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.132311 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.132248    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-workload-certs\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.132687 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.132323    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-workload-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.132687 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.132375    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-credential-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.132687 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.132408    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv9x4\" (UniqueName: \"kubernetes.io/projected/4447e11f-6bed-4da8-97b6-c331cd08a4eb-kube-api-access-kv9x4\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.132687 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.132555    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istiod-ca-cert\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.132687 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.132607    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-envoy\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.132958 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.132708    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-workload-certs\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.132958 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.132800    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-workload-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.133214 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.133191    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-credential-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.133396 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.133339    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-data\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.133576 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.133550    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istiod-ca-cert\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.135774 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.135752    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-podinfo\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.136870 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.136846    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-envoy\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.197692 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.197652    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv9x4\" (UniqueName: \"kubernetes.io/projected/4447e11f-6bed-4da8-97b6-c331cd08a4eb-kube-api-access-kv9x4\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.199326 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.199306    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-token\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.223001 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.222969    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"]
Nov 03 17:51:38.319245 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.319155    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:38.768683 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.768656    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"]
Nov 03 17:51:38.770127 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:51:38.770098    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4447e11f_6bed_4da8_97b6_c331cd08a4eb.slice/crio-a3fa8d5bb87bb5bba01c37bdf8d7c4eb405ead8f0cd19058920e73f74b5a7a12 WatchSource:0}: Error finding container a3fa8d5bb87bb5bba01c37bdf8d7c4eb405ead8f0cd19058920e73f74b5a7a12: Status 404 returned error can't find the container with id a3fa8d5bb87bb5bba01c37bdf8d7c4eb405ead8f0cd19058920e73f74b5a7a12
Nov 03 17:51:38.776526 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.776490    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:51:38.776613 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.776569    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:51:38.776659 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.776619    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:51:38.919471 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.919438    6059 generic.go:356] "Generic (PLEG): container finished" podID="f7c9e388-cf4f-4f23-98df-b36b39615609" containerID="a487c363356bd9016270163279b78a2674899f8b1cfb595e014595c2b2b37682" exitCode=0
Nov 03 17:51:38.919668 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.919502    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf" event={"ID":"f7c9e388-cf4f-4f23-98df-b36b39615609","Type":"ContainerDied","Data":"a487c363356bd9016270163279b78a2674899f8b1cfb595e014595c2b2b37682"}
Nov 03 17:51:38.920055 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.920037    6059 scope.go:122] "RemoveContainer" containerID="a487c363356bd9016270163279b78a2674899f8b1cfb595e014595c2b2b37682"
Nov 03 17:51:38.921836 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.921805    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:51:38.921943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.921877    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:51:38.921943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.921923    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:51:38.922755 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.922704    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx" event={"ID":"4447e11f-6bed-4da8-97b6-c331cd08a4eb","Type":"ContainerStarted","Data":"984ad4ded1213f3fd605fa12b0d103cca53531b364c829ddb3542f43b433f88f"}
Nov 03 17:51:38.922859 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:38.922791    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx" event={"ID":"4447e11f-6bed-4da8-97b6-c331cd08a4eb","Type":"ContainerStarted","Data":"a3fa8d5bb87bb5bba01c37bdf8d7c4eb405ead8f0cd19058920e73f74b5a7a12"}
Nov 03 17:51:39.175157 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:39.175100    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx" podStartSLOduration=2.17508551 podStartE2EDuration="2.17508551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:39.172887565 +0000 UTC m=+2039.970600346" watchObservedRunningTime="2026-11-03 17:51:39.17508551 +0000 UTC m=+2039.972798291"
Nov 03 17:51:39.319342 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:39.319309    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:39.320906 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:39.320879    6059 patch_prober.go:29] interesting pod/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 17:51:39.321037 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:39.320940    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx" podUID="4447e11f-6bed-4da8-97b6-c331cd08a4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 17:51:39.974385 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:39.966403    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf" event={"ID":"f7c9e388-cf4f-4f23-98df-b36b39615609","Type":"ContainerStarted","Data":"6fd130be3e200e333223d29538d46223ffea288e04a2fe3c3c852c1e31c2d1ba"}
Nov 03 17:51:40.319977 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:40.319890    6059 patch_prober.go:29] interesting pod/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 17:51:40.319977 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:40.319957    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx" podUID="4447e11f-6bed-4da8-97b6-c331cd08a4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 17:51:40.480381 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:40.475616    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:51:40.563380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:40.563133    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:40.718025 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:40.717994    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:51:40.718953 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:40.718923    6059 patch_prober.go:29] interesting pod/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 17:51:40.719066 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:40.718971    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf" podUID="f7c9e388-cf4f-4f23-98df-b36b39615609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 17:51:41.320193 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:41.320161    6059 patch_prober.go:29] interesting pod/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 17:51:41.320613 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:41.320219    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx" podUID="4447e11f-6bed-4da8-97b6-c331cd08a4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 17:51:41.718443 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:41.718407    6059 patch_prober.go:29] interesting pod/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 17:51:41.718610 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:41.718473    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf" podUID="f7c9e388-cf4f-4f23-98df-b36b39615609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 17:51:41.943379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:41.942837    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:42.084567 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:42.084459    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r"]
Nov 03 17:51:42.096855 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:42.096815    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dcf428c-j5h8r"]
Nov 03 17:51:42.319959 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:42.319921    6059 patch_prober.go:29] interesting pod/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 17:51:42.320148 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:42.319983    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx" podUID="4447e11f-6bed-4da8-97b6-c331cd08a4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 17:51:42.718917 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:42.718876    6059 patch_prober.go:29] interesting pod/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 17:51:42.719342 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:42.718934    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf" podUID="f7c9e388-cf4f-4f23-98df-b36b39615609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 17:51:43.319664 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:43.319622    6059 patch_prober.go:29] interesting pod/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 17:51:43.319846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:43.319695    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx" podUID="4447e11f-6bed-4da8-97b6-c331cd08a4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 17:51:43.367379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:43.367024    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c" path="/var/lib/kubelet/pods/6f5e4bba-7d1f-4b3c-8746-a77c794b3a8c/volumes"
Nov 03 17:51:43.718808 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:43.718766    6059 patch_prober.go:29] interesting pod/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 17:51:43.719014 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:43.718832    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf" podUID="f7c9e388-cf4f-4f23-98df-b36b39615609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 17:51:44.320020 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:44.319989    6059 patch_prober.go:29] interesting pod/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 17:51:44.320249 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:44.320229    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx" podUID="4447e11f-6bed-4da8-97b6-c331cd08a4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 17:51:44.718178 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:44.718147    6059 patch_prober.go:29] interesting pod/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 17:51:44.718348 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:44.718197    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf" podUID="f7c9e388-cf4f-4f23-98df-b36b39615609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 17:51:45.125875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:45.125647    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:45.324710 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:45.324681    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:45.325268 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:45.325251    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:45.325906 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:45.325885    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:51:45.723310 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:45.723280    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:51:45.724368 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:45.724334    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:51:46.640287 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:46.640251    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:47.570659 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:47.570591    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:48.036140 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:48.036103    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:49.670525 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:49.670281    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:52.607211 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:52.607177    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:53.157095 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:53.157056    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:54.180972 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:54.180929    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:51:58.758842 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:51:58.758791    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:03.335553 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:03.335513    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:06.861743 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:52:06.861709    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:52:07.199241 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:07.199151    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh"]
Nov 03 17:52:07.313434 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:07.313403    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q"]
Nov 03 17:52:07.331129 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:07.331067    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:07.561614 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:07.561562    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:07.770732 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:07.770699    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:08.320681 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:08.320653    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:08.582600 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:08.582481    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:12.199856 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:12.199806    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh" podUID="cc016fa5-62ce-4281-9e0b-488aab664638" containerName="payload-processing" containerID="cri-o://4a7f57275dfef1e74f8aa8c8a714ec46389970ec97e0c0d484928b2bae1aa578" gracePeriod=25
Nov 03 17:52:12.314493 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:12.314405    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q" podUID="137627b9-a30d-4ff1-a8ca-5118bed82be1" containerName="payload-pre-processing" containerID="cri-o://321c01d68d840729997e2c9c7f146a1c636c9154b218471c3bc1431306d984f3" gracePeriod=25
Nov 03 17:52:12.478935 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:12.478910    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh"
Nov 03 17:52:12.555261 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:12.555168    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc016fa5-62ce-4281-9e0b-488aab664638-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc016fa5-62ce-4281-9e0b-488aab664638-plugins-config-volume\") pod \"cc016fa5-62ce-4281-9e0b-488aab664638\" (UID: \"cc016fa5-62ce-4281-9e0b-488aab664638\") "
Nov 03 17:52:12.555495 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:12.555392    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc016fa5-62ce-4281-9e0b-488aab664638-kube-api-access-gpczl\" (UniqueName: \"kubernetes.io/projected/cc016fa5-62ce-4281-9e0b-488aab664638-kube-api-access-gpczl\") pod \"cc016fa5-62ce-4281-9e0b-488aab664638\" (UID: \"cc016fa5-62ce-4281-9e0b-488aab664638\") "
Nov 03 17:52:12.555703 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:12.555676    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc016fa5-62ce-4281-9e0b-488aab664638-plugins-config-volume" pod "cc016fa5-62ce-4281-9e0b-488aab664638" (UID: "cc016fa5-62ce-4281-9e0b-488aab664638"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:52:12.556032 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:12.556010    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc016fa5-62ce-4281-9e0b-488aab664638-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:12.557904 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:12.557873    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc016fa5-62ce-4281-9e0b-488aab664638-kube-api-access-gpczl" pod "cc016fa5-62ce-4281-9e0b-488aab664638" (UID: "cc016fa5-62ce-4281-9e0b-488aab664638"). InnerVolumeSpecName "kube-api-access-gpczl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:52:12.577270 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:12.577242    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q"
Nov 03 17:52:12.656832 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:12.656792    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137627b9-a30d-4ff1-a8ca-5118bed82be1-kube-api-access-tw8gv\" (UniqueName: \"kubernetes.io/projected/137627b9-a30d-4ff1-a8ca-5118bed82be1-kube-api-access-tw8gv\") pod \"137627b9-a30d-4ff1-a8ca-5118bed82be1\" (UID: \"137627b9-a30d-4ff1-a8ca-5118bed82be1\") "
Nov 03 17:52:12.657025 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:12.656937    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/137627b9-a30d-4ff1-a8ca-5118bed82be1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137627b9-a30d-4ff1-a8ca-5118bed82be1-plugins-config-volume\") pod \"137627b9-a30d-4ff1-a8ca-5118bed82be1\" (UID: \"137627b9-a30d-4ff1-a8ca-5118bed82be1\") "
Nov 03 17:52:12.657287 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:12.657264    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpczl\" (UniqueName: \"kubernetes.io/projected/cc016fa5-62ce-4281-9e0b-488aab664638-kube-api-access-gpczl\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:12.657436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:12.657304    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137627b9-a30d-4ff1-a8ca-5118bed82be1-plugins-config-volume" pod "137627b9-a30d-4ff1-a8ca-5118bed82be1" (UID: "137627b9-a30d-4ff1-a8ca-5118bed82be1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:52:12.658953 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:12.658930    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137627b9-a30d-4ff1-a8ca-5118bed82be1-kube-api-access-tw8gv" pod "137627b9-a30d-4ff1-a8ca-5118bed82be1" (UID: "137627b9-a30d-4ff1-a8ca-5118bed82be1"). InnerVolumeSpecName "kube-api-access-tw8gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:52:12.758456 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:12.758414    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw8gv\" (UniqueName: \"kubernetes.io/projected/137627b9-a30d-4ff1-a8ca-5118bed82be1-kube-api-access-tw8gv\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:12.758456 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:12.758457    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137627b9-a30d-4ff1-a8ca-5118bed82be1-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:13.085054 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:13.085020    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:13.225103 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:13.225072    6059 generic.go:356] "Generic (PLEG): container finished" podID="137627b9-a30d-4ff1-a8ca-5118bed82be1" containerID="321c01d68d840729997e2c9c7f146a1c636c9154b218471c3bc1431306d984f3" exitCode=0
Nov 03 17:52:13.225593 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:13.225117    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q"
Nov 03 17:52:13.225593 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:13.225149    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q" event={"ID":"137627b9-a30d-4ff1-a8ca-5118bed82be1","Type":"ContainerDied","Data":"321c01d68d840729997e2c9c7f146a1c636c9154b218471c3bc1431306d984f3"}
Nov 03 17:52:13.225593 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:13.225199    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q" event={"ID":"137627b9-a30d-4ff1-a8ca-5118bed82be1","Type":"ContainerDied","Data":"1470d3cc92479042c3157ec0054bbd0554e1b1abf9eddd9b5f6170cea6ed1a73"}
Nov 03 17:52:13.225593 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:13.225228    6059 scope.go:122] "RemoveContainer" containerID="321c01d68d840729997e2c9c7f146a1c636c9154b218471c3bc1431306d984f3"
Nov 03 17:52:13.226817 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:13.226783    6059 generic.go:356] "Generic (PLEG): container finished" podID="cc016fa5-62ce-4281-9e0b-488aab664638" containerID="4a7f57275dfef1e74f8aa8c8a714ec46389970ec97e0c0d484928b2bae1aa578" exitCode=0
Nov 03 17:52:13.226914 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:13.226812    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh" event={"ID":"cc016fa5-62ce-4281-9e0b-488aab664638","Type":"ContainerDied","Data":"4a7f57275dfef1e74f8aa8c8a714ec46389970ec97e0c0d484928b2bae1aa578"}
Nov 03 17:52:13.226914 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:13.226829    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh"
Nov 03 17:52:13.226914 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:13.226867    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh" event={"ID":"cc016fa5-62ce-4281-9e0b-488aab664638","Type":"ContainerDied","Data":"efca46464edc8fae3f41e44a06c580cb7295df03e3590f8fddb0ca131551ea8a"}
Nov 03 17:52:13.236772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:13.236748    6059 scope.go:122] "RemoveContainer" containerID="321c01d68d840729997e2c9c7f146a1c636c9154b218471c3bc1431306d984f3"
Nov 03 17:52:13.237083 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:52:13.237065    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321c01d68d840729997e2c9c7f146a1c636c9154b218471c3bc1431306d984f3\": container with ID starting with 321c01d68d840729997e2c9c7f146a1c636c9154b218471c3bc1431306d984f3 not found: ID does not exist" containerID="321c01d68d840729997e2c9c7f146a1c636c9154b218471c3bc1431306d984f3"
Nov 03 17:52:13.237137 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:13.237091    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321c01d68d840729997e2c9c7f146a1c636c9154b218471c3bc1431306d984f3"} err="failed to get container status \"321c01d68d840729997e2c9c7f146a1c636c9154b218471c3bc1431306d984f3\": rpc error: code = NotFound desc = could not find container \"321c01d68d840729997e2c9c7f146a1c636c9154b218471c3bc1431306d984f3\": container with ID starting with 321c01d68d840729997e2c9c7f146a1c636c9154b218471c3bc1431306d984f3 not found: ID does not exist"
Nov 03 17:52:13.237137 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:13.237107    6059 scope.go:122] "RemoveContainer" containerID="4a7f57275dfef1e74f8aa8c8a714ec46389970ec97e0c0d484928b2bae1aa578"
Nov 03 17:52:13.248211 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:13.248190    6059 scope.go:122] "RemoveContainer" containerID="4a7f57275dfef1e74f8aa8c8a714ec46389970ec97e0c0d484928b2bae1aa578"
Nov 03 17:52:13.248538 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:52:13.248511    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7f57275dfef1e74f8aa8c8a714ec46389970ec97e0c0d484928b2bae1aa578\": container with ID starting with 4a7f57275dfef1e74f8aa8c8a714ec46389970ec97e0c0d484928b2bae1aa578 not found: ID does not exist" containerID="4a7f57275dfef1e74f8aa8c8a714ec46389970ec97e0c0d484928b2bae1aa578"
Nov 03 17:52:13.248628 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:13.248545    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7f57275dfef1e74f8aa8c8a714ec46389970ec97e0c0d484928b2bae1aa578"} err="failed to get container status \"4a7f57275dfef1e74f8aa8c8a714ec46389970ec97e0c0d484928b2bae1aa578\": rpc error: code = NotFound desc = could not find container \"4a7f57275dfef1e74f8aa8c8a714ec46389970ec97e0c0d484928b2bae1aa578\": container with ID starting with 4a7f57275dfef1e74f8aa8c8a714ec46389970ec97e0c0d484928b2bae1aa578 not found: ID does not exist"
Nov 03 17:52:13.397824 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:13.397738    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q"]
Nov 03 17:52:13.449559 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:13.449527    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1f6be0-556d5c6bc-b4z2q"]
Nov 03 17:52:13.515678 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:13.515643    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh"]
Nov 03 17:52:13.518947 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:13.518916    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1f6be0-76696ff9b9-n7lbh"]
Nov 03 17:52:13.654968 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:13.654883    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:13.913550 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:13.913463    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:15.336883 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:15.336849    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="137627b9-a30d-4ff1-a8ca-5118bed82be1" path="/var/lib/kubelet/pods/137627b9-a30d-4ff1-a8ca-5118bed82be1/volumes"
Nov 03 17:52:15.337298 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:15.337284    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc016fa5-62ce-4281-9e0b-488aab664638" path="/var/lib/kubelet/pods/cc016fa5-62ce-4281-9e0b-488aab664638/volumes"
Nov 03 17:52:15.548081 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:15.548047    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"]
Nov 03 17:52:15.548270 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:15.548243    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68" podUID="24835092-2c35-423e-990a-1314dde40f2a" containerName="istio-proxy" containerID="cri-o://81744c8b9c10b3e9122d4856c22239fe14901e3cab5b6ec591a445520d97138e" gracePeriod=30
Nov 03 17:52:17.671336 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:17.671260    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:19.163116 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:19.163070    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c"]
Nov 03 17:52:19.274100 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:19.274064    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb"]
Nov 03 17:52:19.542534 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:19.542492    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:19.739771 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:19.739736    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:20.146439 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.146400    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:52:20.146644 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.146461    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68" podUID="24835092-2c35-423e-990a-1314dde40f2a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:52:20.424039 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.423949    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:20.749515 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.749482    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:20.862542 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.862507    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:52:20.943814 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.943781    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:20.972599 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.972503    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-istio-envoy\") pod \"24835092-2c35-423e-990a-1314dde40f2a\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") "
Nov 03 17:52:20.972599 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.972546    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24835092-2c35-423e-990a-1314dde40f2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24835092-2c35-423e-990a-1314dde40f2a-istiod-ca-cert\") pod \"24835092-2c35-423e-990a-1314dde40f2a\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") "
Nov 03 17:52:20.972599 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.972589    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-workload-socket\") pod \"24835092-2c35-423e-990a-1314dde40f2a\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") "
Nov 03 17:52:20.972891 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.972676    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-istio-data\") pod \"24835092-2c35-423e-990a-1314dde40f2a\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") "
Nov 03 17:52:20.972891 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.972747    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24835092-2c35-423e-990a-1314dde40f2a-kube-api-access-vtxx2\" (UniqueName: \"kubernetes.io/projected/24835092-2c35-423e-990a-1314dde40f2a-kube-api-access-vtxx2\") pod \"24835092-2c35-423e-990a-1314dde40f2a\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") "
Nov 03 17:52:20.972891 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.972847    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24835092-2c35-423e-990a-1314dde40f2a-istio-token\" (UniqueName: \"kubernetes.io/projected/24835092-2c35-423e-990a-1314dde40f2a-istio-token\") pod \"24835092-2c35-423e-990a-1314dde40f2a\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") "
Nov 03 17:52:20.973048 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.972922    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-workload-certs\") pod \"24835092-2c35-423e-990a-1314dde40f2a\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") "
Nov 03 17:52:20.973048 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.972941    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-workload-socket" pod "24835092-2c35-423e-990a-1314dde40f2a" (UID: "24835092-2c35-423e-990a-1314dde40f2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:52:20.973048 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.972972    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-credential-socket\") pod \"24835092-2c35-423e-990a-1314dde40f2a\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") "
Nov 03 17:52:20.973048 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.973012    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24835092-2c35-423e-990a-1314dde40f2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24835092-2c35-423e-990a-1314dde40f2a-istio-podinfo\") pod \"24835092-2c35-423e-990a-1314dde40f2a\" (UID: \"24835092-2c35-423e-990a-1314dde40f2a\") "
Nov 03 17:52:20.973048 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.973028    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24835092-2c35-423e-990a-1314dde40f2a-istiod-ca-cert" pod "24835092-2c35-423e-990a-1314dde40f2a" (UID: "24835092-2c35-423e-990a-1314dde40f2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:52:20.973048 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.973031    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-istio-data" pod "24835092-2c35-423e-990a-1314dde40f2a" (UID: "24835092-2c35-423e-990a-1314dde40f2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:52:20.973476 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.973221    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-workload-certs" pod "24835092-2c35-423e-990a-1314dde40f2a" (UID: "24835092-2c35-423e-990a-1314dde40f2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:52:20.973476 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.973420    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-credential-socket" pod "24835092-2c35-423e-990a-1314dde40f2a" (UID: "24835092-2c35-423e-990a-1314dde40f2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:52:20.973745 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.973722    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:20.973802 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.973751    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:20.973802 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.973766    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:20.973802 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.973780    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:20.973802 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.973794    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24835092-2c35-423e-990a-1314dde40f2a-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:20.974906 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.974882    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-istio-envoy" pod "24835092-2c35-423e-990a-1314dde40f2a" (UID: "24835092-2c35-423e-990a-1314dde40f2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:52:20.975121 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.975104    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24835092-2c35-423e-990a-1314dde40f2a-istio-podinfo" pod "24835092-2c35-423e-990a-1314dde40f2a" (UID: "24835092-2c35-423e-990a-1314dde40f2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:52:20.975577 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.975558    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24835092-2c35-423e-990a-1314dde40f2a-kube-api-access-vtxx2" pod "24835092-2c35-423e-990a-1314dde40f2a" (UID: "24835092-2c35-423e-990a-1314dde40f2a"). InnerVolumeSpecName "kube-api-access-vtxx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:52:20.975633 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:20.975588    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24835092-2c35-423e-990a-1314dde40f2a-istio-token" pod "24835092-2c35-423e-990a-1314dde40f2a" (UID: "24835092-2c35-423e-990a-1314dde40f2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:52:21.075377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:21.075316    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtxx2\" (UniqueName: \"kubernetes.io/projected/24835092-2c35-423e-990a-1314dde40f2a-kube-api-access-vtxx2\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:21.075377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:21.075348    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24835092-2c35-423e-990a-1314dde40f2a-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:21.075377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:21.075375    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24835092-2c35-423e-990a-1314dde40f2a-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:21.075614 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:21.075387    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24835092-2c35-423e-990a-1314dde40f2a-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:21.297754 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:21.297660    6059 generic.go:356] "Generic (PLEG): container finished" podID="24835092-2c35-423e-990a-1314dde40f2a" containerID="81744c8b9c10b3e9122d4856c22239fe14901e3cab5b6ec591a445520d97138e" exitCode=0
Nov 03 17:52:21.297754 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:21.297710    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"
Nov 03 17:52:21.297754 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:21.297719    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68" event={"ID":"24835092-2c35-423e-990a-1314dde40f2a","Type":"ContainerDied","Data":"81744c8b9c10b3e9122d4856c22239fe14901e3cab5b6ec591a445520d97138e"}
Nov 03 17:52:21.298065 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:21.297788    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68" event={"ID":"24835092-2c35-423e-990a-1314dde40f2a","Type":"ContainerDied","Data":"00f6be4bea14dba1a12c20350c32e4638ba7a7427c9612c77da9f13423597c96"}
Nov 03 17:52:21.298065 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:21.297809    6059 scope.go:122] "RemoveContainer" containerID="81744c8b9c10b3e9122d4856c22239fe14901e3cab5b6ec591a445520d97138e"
Nov 03 17:52:21.309386 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:21.309368    6059 scope.go:122] "RemoveContainer" containerID="81744c8b9c10b3e9122d4856c22239fe14901e3cab5b6ec591a445520d97138e"
Nov 03 17:52:21.309652 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:52:21.309635    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81744c8b9c10b3e9122d4856c22239fe14901e3cab5b6ec591a445520d97138e\": container with ID starting with 81744c8b9c10b3e9122d4856c22239fe14901e3cab5b6ec591a445520d97138e not found: ID does not exist" containerID="81744c8b9c10b3e9122d4856c22239fe14901e3cab5b6ec591a445520d97138e"
Nov 03 17:52:21.309717 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:21.309659    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81744c8b9c10b3e9122d4856c22239fe14901e3cab5b6ec591a445520d97138e"} err="failed to get container status \"81744c8b9c10b3e9122d4856c22239fe14901e3cab5b6ec591a445520d97138e\": rpc error: code = NotFound desc = could not find container \"81744c8b9c10b3e9122d4856c22239fe14901e3cab5b6ec591a445520d97138e\": container with ID starting with 81744c8b9c10b3e9122d4856c22239fe14901e3cab5b6ec591a445520d97138e not found: ID does not exist"
Nov 03 17:52:21.328270 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:52:21.328237    6059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24835092_2c35_423e_990a_1314dde40f2a.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 17:52:21.488042 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:21.488009    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"]
Nov 03 17:52:21.491099 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:21.491076    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-24w68"]
Nov 03 17:52:22.391233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:22.391203    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:22.817668 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:22.817630    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"]
Nov 03 17:52:22.818126 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:22.817862    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz" podUID="dfcde397-4b17-48a2-bd2b-ad43fd73be72" containerName="istio-proxy" containerID="cri-o://e36dff831e947032b996235a502c0eb80a523879c7fc534ff698b40bea77654a" gracePeriod=30
Nov 03 17:52:23.336921 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:23.336884    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24835092-2c35-423e-990a-1314dde40f2a" path="/var/lib/kubelet/pods/24835092-2c35-423e-990a-1314dde40f2a/volumes"
Nov 03 17:52:23.612939 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:23.612852    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:24.163776 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.163705    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c" podUID="c4f61445-3d05-4900-9eb7-5d8a55eadf66" containerName="payload-processing" containerID="cri-o://0dfab80b6f5cac141a6d4ecfa7aa1ff358b5de927296e3b2f80d937dfe6973a1" gracePeriod=25
Nov 03 17:52:24.275285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.275245    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb" podUID="53c5fac3-1291-4c88-b6f4-1cf0829e33f7" containerName="payload-pre-processing" containerID="cri-o://23193fe06d97a0da09bf3097dcdc3190809b5af8c59ef6671949b0206431fad1" gracePeriod=25
Nov 03 17:52:24.288079 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.288048    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:24.322875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.322845    6059 generic.go:356] "Generic (PLEG): container finished" podID="c4f61445-3d05-4900-9eb7-5d8a55eadf66" containerID="0dfab80b6f5cac141a6d4ecfa7aa1ff358b5de927296e3b2f80d937dfe6973a1" exitCode=0
Nov 03 17:52:24.323032 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.322922    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c" event={"ID":"c4f61445-3d05-4900-9eb7-5d8a55eadf66","Type":"ContainerDied","Data":"0dfab80b6f5cac141a6d4ecfa7aa1ff358b5de927296e3b2f80d937dfe6973a1"}
Nov 03 17:52:24.475049 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.475022    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c"
Nov 03 17:52:24.565632 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.565605    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb"
Nov 03 17:52:24.591651 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.591605    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"]
Nov 03 17:52:24.592343 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.592322    6059 cpu_manager.go:414] "containerMap: removing container" podUID="cc016fa5-62ce-4281-9e0b-488aab664638" containerName="payload-processing"
Nov 03 17:52:24.592435 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.592345    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc016fa5-62ce-4281-9e0b-488aab664638" containerName="payload-processing"
Nov 03 17:52:24.592435 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.592380    6059 cpu_manager.go:414] "containerMap: removing container" podUID="24835092-2c35-423e-990a-1314dde40f2a" containerName="istio-proxy"
Nov 03 17:52:24.592435 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.592386    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24835092-2c35-423e-990a-1314dde40f2a" containerName="istio-proxy"
Nov 03 17:52:24.592435 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.592393    6059 cpu_manager.go:414] "containerMap: removing container" podUID="53c5fac3-1291-4c88-b6f4-1cf0829e33f7" containerName="payload-pre-processing"
Nov 03 17:52:24.592435 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.592398    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53c5fac3-1291-4c88-b6f4-1cf0829e33f7" containerName="payload-pre-processing"
Nov 03 17:52:24.592435 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.592422    6059 cpu_manager.go:414] "containerMap: removing container" podUID="c4f61445-3d05-4900-9eb7-5d8a55eadf66" containerName="payload-processing"
Nov 03 17:52:24.592435 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.592427    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f61445-3d05-4900-9eb7-5d8a55eadf66" containerName="payload-processing"
Nov 03 17:52:24.592668 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.592445    6059 cpu_manager.go:414] "containerMap: removing container" podUID="137627b9-a30d-4ff1-a8ca-5118bed82be1" containerName="payload-pre-processing"
Nov 03 17:52:24.592668 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.592451    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137627b9-a30d-4ff1-a8ca-5118bed82be1" containerName="payload-pre-processing"
Nov 03 17:52:24.592668 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.592546    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4f61445-3d05-4900-9eb7-5d8a55eadf66" containerName="payload-processing"
Nov 03 17:52:24.592668 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.592558    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="137627b9-a30d-4ff1-a8ca-5118bed82be1" containerName="payload-pre-processing"
Nov 03 17:52:24.592668 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.592568    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc016fa5-62ce-4281-9e0b-488aab664638" containerName="payload-processing"
Nov 03 17:52:24.592668 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.592574    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="24835092-2c35-423e-990a-1314dde40f2a" containerName="istio-proxy"
Nov 03 17:52:24.592668 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.592580    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="53c5fac3-1291-4c88-b6f4-1cf0829e33f7" containerName="payload-pre-processing"
Nov 03 17:52:24.596058 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.596038    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.627971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.627936    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4f61445-3d05-4900-9eb7-5d8a55eadf66-kube-api-access-jm226\" (UniqueName: \"kubernetes.io/projected/c4f61445-3d05-4900-9eb7-5d8a55eadf66-kube-api-access-jm226\") pod \"c4f61445-3d05-4900-9eb7-5d8a55eadf66\" (UID: \"c4f61445-3d05-4900-9eb7-5d8a55eadf66\") "
Nov 03 17:52:24.628144 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.628081    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4f61445-3d05-4900-9eb7-5d8a55eadf66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f61445-3d05-4900-9eb7-5d8a55eadf66-plugins-config-volume\") pod \"c4f61445-3d05-4900-9eb7-5d8a55eadf66\" (UID: \"c4f61445-3d05-4900-9eb7-5d8a55eadf66\") "
Nov 03 17:52:24.628533 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.628499    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f61445-3d05-4900-9eb7-5d8a55eadf66-plugins-config-volume" pod "c4f61445-3d05-4900-9eb7-5d8a55eadf66" (UID: "c4f61445-3d05-4900-9eb7-5d8a55eadf66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:52:24.628850 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.628823    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f61445-3d05-4900-9eb7-5d8a55eadf66-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:24.630730 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.630697    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f61445-3d05-4900-9eb7-5d8a55eadf66-kube-api-access-jm226" pod "c4f61445-3d05-4900-9eb7-5d8a55eadf66" (UID: "c4f61445-3d05-4900-9eb7-5d8a55eadf66"). InnerVolumeSpecName "kube-api-access-jm226". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:52:24.650925 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.650887    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-fe8fde-openshift-default-dockercfg-svr97\""
Nov 03 17:52:24.671827 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.671747    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"]
Nov 03 17:52:24.719554 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.719523    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:24.729658 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.729633    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53c5fac3-1291-4c88-b6f4-1cf0829e33f7-kube-api-access-cxzw5\" (UniqueName: \"kubernetes.io/projected/53c5fac3-1291-4c88-b6f4-1cf0829e33f7-kube-api-access-cxzw5\") pod \"53c5fac3-1291-4c88-b6f4-1cf0829e33f7\" (UID: \"53c5fac3-1291-4c88-b6f4-1cf0829e33f7\") "
Nov 03 17:52:24.729869 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.729852    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53c5fac3-1291-4c88-b6f4-1cf0829e33f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c5fac3-1291-4c88-b6f4-1cf0829e33f7-plugins-config-volume\") pod \"53c5fac3-1291-4c88-b6f4-1cf0829e33f7\" (UID: \"53c5fac3-1291-4c88-b6f4-1cf0829e33f7\") "
Nov 03 17:52:24.730054 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.730033    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz5zb\" (UniqueName: \"kubernetes.io/projected/945d8e0f-a47e-49e3-ae35-d25ad12aca81-kube-api-access-vz5zb\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.730141 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.730111    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-data\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.730203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.730182    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-credential-socket\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.730261 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.730206    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-podinfo\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.730261 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.730233    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-envoy\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.730261 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.730227    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c5fac3-1291-4c88-b6f4-1cf0829e33f7-plugins-config-volume" pod "53c5fac3-1291-4c88-b6f4-1cf0829e33f7" (UID: "53c5fac3-1291-4c88-b6f4-1cf0829e33f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:52:24.730439 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.730397    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-workload-certs\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.730569 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.730547    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istiod-ca-cert\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.730665 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.730652    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-token\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.730724 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.730678    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-workload-socket\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.730856 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.730837    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c5fac3-1291-4c88-b6f4-1cf0829e33f7-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:24.730892 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.730866    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm226\" (UniqueName: \"kubernetes.io/projected/c4f61445-3d05-4900-9eb7-5d8a55eadf66-kube-api-access-jm226\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:24.731943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.731915    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c5fac3-1291-4c88-b6f4-1cf0829e33f7-kube-api-access-cxzw5" pod "53c5fac3-1291-4c88-b6f4-1cf0829e33f7" (UID: "53c5fac3-1291-4c88-b6f4-1cf0829e33f7"). InnerVolumeSpecName "kube-api-access-cxzw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:52:24.831912 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.831867    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istiod-ca-cert\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.832129 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.831974    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-token\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.832129 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.832012    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-workload-socket\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.832129 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.832070    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz5zb\" (UniqueName: \"kubernetes.io/projected/945d8e0f-a47e-49e3-ae35-d25ad12aca81-kube-api-access-vz5zb\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.832304 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.832145    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-data\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.832304 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.832190    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-credential-socket\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.832304 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.832218    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-podinfo\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.832304 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.832254    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-envoy\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.832304 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.832297    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-workload-certs\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.832621 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.832450    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxzw5\" (UniqueName: \"kubernetes.io/projected/53c5fac3-1291-4c88-b6f4-1cf0829e33f7-kube-api-access-cxzw5\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:24.832621 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.832522    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-workload-socket\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.832621 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.832554    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-credential-socket\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.832774 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.832719    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-data\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.832774 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.832747    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-workload-certs\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.833245 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.833215    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istiod-ca-cert\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.834971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.834873    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-podinfo\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.835184 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.835155    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-envoy\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.902564 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.902521    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-token\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.902739 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.902650    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz5zb\" (UniqueName: \"kubernetes.io/projected/945d8e0f-a47e-49e3-ae35-d25ad12aca81-kube-api-access-vz5zb\") pod \"e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") " pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.913918 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.913890    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:24.937779 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:24.937753    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:25.331881 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:25.331837    6059 generic.go:356] "Generic (PLEG): container finished" podID="53c5fac3-1291-4c88-b6f4-1cf0829e33f7" containerID="23193fe06d97a0da09bf3097dcdc3190809b5af8c59ef6671949b0206431fad1" exitCode=0
Nov 03 17:52:25.332305 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:25.331909    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb"
Nov 03 17:52:25.333595 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:25.333576    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c"
Nov 03 17:52:25.336321 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:25.336295    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb" event={"ID":"53c5fac3-1291-4c88-b6f4-1cf0829e33f7","Type":"ContainerDied","Data":"23193fe06d97a0da09bf3097dcdc3190809b5af8c59ef6671949b0206431fad1"}
Nov 03 17:52:25.336458 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:25.336350    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb" event={"ID":"53c5fac3-1291-4c88-b6f4-1cf0829e33f7","Type":"ContainerDied","Data":"5624957d004a3ee97ae20c5646ecea28ee264eb5c7b2263f9de02cec9ef2705e"}
Nov 03 17:52:25.336458 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:25.336390    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c" event={"ID":"c4f61445-3d05-4900-9eb7-5d8a55eadf66","Type":"ContainerDied","Data":"df59efc8598c2b6eacdcaaf4fd54cc78d8046167bad8d308783ca8e70a038a0f"}
Nov 03 17:52:25.336565 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:25.336457    6059 scope.go:122] "RemoveContainer" containerID="23193fe06d97a0da09bf3097dcdc3190809b5af8c59ef6671949b0206431fad1"
Nov 03 17:52:25.350658 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:25.350641    6059 scope.go:122] "RemoveContainer" containerID="23193fe06d97a0da09bf3097dcdc3190809b5af8c59ef6671949b0206431fad1"
Nov 03 17:52:25.350933 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:52:25.350915    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23193fe06d97a0da09bf3097dcdc3190809b5af8c59ef6671949b0206431fad1\": container with ID starting with 23193fe06d97a0da09bf3097dcdc3190809b5af8c59ef6671949b0206431fad1 not found: ID does not exist" containerID="23193fe06d97a0da09bf3097dcdc3190809b5af8c59ef6671949b0206431fad1"
Nov 03 17:52:25.350980 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:25.350941    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23193fe06d97a0da09bf3097dcdc3190809b5af8c59ef6671949b0206431fad1"} err="failed to get container status \"23193fe06d97a0da09bf3097dcdc3190809b5af8c59ef6671949b0206431fad1\": rpc error: code = NotFound desc = could not find container \"23193fe06d97a0da09bf3097dcdc3190809b5af8c59ef6671949b0206431fad1\": container with ID starting with 23193fe06d97a0da09bf3097dcdc3190809b5af8c59ef6671949b0206431fad1 not found: ID does not exist"
Nov 03 17:52:25.350980 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:25.350961    6059 scope.go:122] "RemoveContainer" containerID="0dfab80b6f5cac141a6d4ecfa7aa1ff358b5de927296e3b2f80d937dfe6973a1"
Nov 03 17:52:25.362909 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:25.362888    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"]
Nov 03 17:52:25.364972 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:52:25.364950    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945d8e0f_a47e_49e3_ae35_d25ad12aca81.slice/crio-c26cce9389a48951c2c0dfc844b9fd0970dfe24a7a4302c482f998e982d74d68 WatchSource:0}: Error finding container c26cce9389a48951c2c0dfc844b9fd0970dfe24a7a4302c482f998e982d74d68: Status 404 returned error can't find the container with id c26cce9389a48951c2c0dfc844b9fd0970dfe24a7a4302c482f998e982d74d68
Nov 03 17:52:25.367345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:25.367317    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:52:25.367480 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:25.367417    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:52:25.367480 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:25.367448    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:52:25.558430 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:25.558392    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb"]
Nov 03 17:52:25.565846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:25.565819    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-382697-7d9994c987-xs2pb"]
Nov 03 17:52:26.142719 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:26.142688    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:26.343141 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:26.343102    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp" event={"ID":"945d8e0f-a47e-49e3-ae35-d25ad12aca81","Type":"ContainerStarted","Data":"5f7c4e9223356daff37422b90e7e797621341a59336d0ee6fd287f95389542f4"}
Nov 03 17:52:26.343612 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:26.343160    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp" event={"ID":"945d8e0f-a47e-49e3-ae35-d25ad12aca81","Type":"ContainerStarted","Data":"c26cce9389a48951c2c0dfc844b9fd0970dfe24a7a4302c482f998e982d74d68"}
Nov 03 17:52:26.540260 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:26.540205    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp" podStartSLOduration=2.540191885 podStartE2EDuration="2.540191885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:26.537233927 +0000 UTC m=+2087.334946707" watchObservedRunningTime="2026-11-03 17:52:26.540191885 +0000 UTC m=+2087.337904665"
Nov 03 17:52:26.914975 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:26.914891    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:26.920157 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:26.920130    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:27.239879 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:27.239846    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:27.336139 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:27.336108    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53c5fac3-1291-4c88-b6f4-1cf0829e33f7" path="/var/lib/kubelet/pods/53c5fac3-1291-4c88-b6f4-1cf0829e33f7/volumes"
Nov 03 17:52:27.351238 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:27.351207    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:27.352324 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:27.352302    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:52:27.422144 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:27.422111    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:28.077001 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.076975    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:52:28.194759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.194666    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istiod-ca-cert\") pod \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") "
Nov 03 17:52:28.194759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.194723    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-envoy\") pod \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") "
Nov 03 17:52:28.195020 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.194771    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-token\" (UniqueName: \"kubernetes.io/projected/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-token\") pod \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") "
Nov 03 17:52:28.195020 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.194823    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-podinfo\") pod \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") "
Nov 03 17:52:28.195020 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.194897    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfcde397-4b17-48a2-bd2b-ad43fd73be72-kube-api-access-w5789\" (UniqueName: \"kubernetes.io/projected/dfcde397-4b17-48a2-bd2b-ad43fd73be72-kube-api-access-w5789\") pod \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") "
Nov 03 17:52:28.195020 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.194927    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-workload-certs\") pod \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") "
Nov 03 17:52:28.195247 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.195042    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-data\") pod \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") "
Nov 03 17:52:28.195247 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.195098    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-workload-socket\") pod \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") "
Nov 03 17:52:28.195247 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.195122    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istiod-ca-cert" pod "dfcde397-4b17-48a2-bd2b-ad43fd73be72" (UID: "dfcde397-4b17-48a2-bd2b-ad43fd73be72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:52:28.195247 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.195139    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-credential-socket\") pod \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\" (UID: \"dfcde397-4b17-48a2-bd2b-ad43fd73be72\") "
Nov 03 17:52:28.195505 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.195447    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-data" pod "dfcde397-4b17-48a2-bd2b-ad43fd73be72" (UID: "dfcde397-4b17-48a2-bd2b-ad43fd73be72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:52:28.195579 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.195531    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-workload-socket" pod "dfcde397-4b17-48a2-bd2b-ad43fd73be72" (UID: "dfcde397-4b17-48a2-bd2b-ad43fd73be72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:52:28.195652 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.195609    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-workload-certs" pod "dfcde397-4b17-48a2-bd2b-ad43fd73be72" (UID: "dfcde397-4b17-48a2-bd2b-ad43fd73be72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:52:28.195792 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.195770    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-credential-socket" pod "dfcde397-4b17-48a2-bd2b-ad43fd73be72" (UID: "dfcde397-4b17-48a2-bd2b-ad43fd73be72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:52:28.195829 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.195780    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:28.195829 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.195818    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:28.195900 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.195838    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:28.195900 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.195853    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:28.197240 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.197217    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfcde397-4b17-48a2-bd2b-ad43fd73be72-kube-api-access-w5789" pod "dfcde397-4b17-48a2-bd2b-ad43fd73be72" (UID: "dfcde397-4b17-48a2-bd2b-ad43fd73be72"). InnerVolumeSpecName "kube-api-access-w5789". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:52:28.197422 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.197240    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-token" pod "dfcde397-4b17-48a2-bd2b-ad43fd73be72" (UID: "dfcde397-4b17-48a2-bd2b-ad43fd73be72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:52:28.197537 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.197516    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-envoy" pod "dfcde397-4b17-48a2-bd2b-ad43fd73be72" (UID: "dfcde397-4b17-48a2-bd2b-ad43fd73be72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:52:28.197595 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.197530    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-podinfo" pod "dfcde397-4b17-48a2-bd2b-ad43fd73be72" (UID: "dfcde397-4b17-48a2-bd2b-ad43fd73be72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:52:28.297561 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.297520    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:28.297561 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.297551    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:28.297561 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.297562    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfcde397-4b17-48a2-bd2b-ad43fd73be72-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:28.297873 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.297592    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5789\" (UniqueName: \"kubernetes.io/projected/dfcde397-4b17-48a2-bd2b-ad43fd73be72-kube-api-access-w5789\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:28.297873 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.297603    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcde397-4b17-48a2-bd2b-ad43fd73be72-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:28.359938 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.359905    6059 generic.go:356] "Generic (PLEG): container finished" podID="dfcde397-4b17-48a2-bd2b-ad43fd73be72" containerID="e36dff831e947032b996235a502c0eb80a523879c7fc534ff698b40bea77654a" exitCode=0
Nov 03 17:52:28.360340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.359990    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"
Nov 03 17:52:28.360340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.359989    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz" event={"ID":"dfcde397-4b17-48a2-bd2b-ad43fd73be72","Type":"ContainerDied","Data":"e36dff831e947032b996235a502c0eb80a523879c7fc534ff698b40bea77654a"}
Nov 03 17:52:28.360340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.360114    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz" event={"ID":"dfcde397-4b17-48a2-bd2b-ad43fd73be72","Type":"ContainerDied","Data":"fd866ffdff88095f2556d379a19b092ac228d54a3e4b0a7d7dd28d57fc50cd4e"}
Nov 03 17:52:28.360340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.360138    6059 scope.go:122] "RemoveContainer" containerID="e36dff831e947032b996235a502c0eb80a523879c7fc534ff698b40bea77654a"
Nov 03 17:52:28.377015 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.376986    6059 scope.go:122] "RemoveContainer" containerID="cf5c553e8a38eee6abd5a79482b43d62d5af0e6cff4f2b308c39a055e249dee3"
Nov 03 17:52:28.387147 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.387127    6059 scope.go:122] "RemoveContainer" containerID="e36dff831e947032b996235a502c0eb80a523879c7fc534ff698b40bea77654a"
Nov 03 17:52:28.387510 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:52:28.387489    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36dff831e947032b996235a502c0eb80a523879c7fc534ff698b40bea77654a\": container with ID starting with e36dff831e947032b996235a502c0eb80a523879c7fc534ff698b40bea77654a not found: ID does not exist" containerID="e36dff831e947032b996235a502c0eb80a523879c7fc534ff698b40bea77654a"
Nov 03 17:52:28.387555 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.387520    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36dff831e947032b996235a502c0eb80a523879c7fc534ff698b40bea77654a"} err="failed to get container status \"e36dff831e947032b996235a502c0eb80a523879c7fc534ff698b40bea77654a\": rpc error: code = NotFound desc = could not find container \"e36dff831e947032b996235a502c0eb80a523879c7fc534ff698b40bea77654a\": container with ID starting with e36dff831e947032b996235a502c0eb80a523879c7fc534ff698b40bea77654a not found: ID does not exist"
Nov 03 17:52:28.387555 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.387538    6059 scope.go:122] "RemoveContainer" containerID="cf5c553e8a38eee6abd5a79482b43d62d5af0e6cff4f2b308c39a055e249dee3"
Nov 03 17:52:28.387803 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:52:28.387787    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf5c553e8a38eee6abd5a79482b43d62d5af0e6cff4f2b308c39a055e249dee3\": container with ID starting with cf5c553e8a38eee6abd5a79482b43d62d5af0e6cff4f2b308c39a055e249dee3 not found: ID does not exist" containerID="cf5c553e8a38eee6abd5a79482b43d62d5af0e6cff4f2b308c39a055e249dee3"
Nov 03 17:52:28.387842 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.387807    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5c553e8a38eee6abd5a79482b43d62d5af0e6cff4f2b308c39a055e249dee3"} err="failed to get container status \"cf5c553e8a38eee6abd5a79482b43d62d5af0e6cff4f2b308c39a055e249dee3\": rpc error: code = NotFound desc = could not find container \"cf5c553e8a38eee6abd5a79482b43d62d5af0e6cff4f2b308c39a055e249dee3\": container with ID starting with cf5c553e8a38eee6abd5a79482b43d62d5af0e6cff4f2b308c39a055e249dee3 not found: ID does not exist"
Nov 03 17:52:28.532865 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.532822    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"]
Nov 03 17:52:28.536421 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.536396    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-1f6be0-openshift-default-75964987ff-8j9sz"]
Nov 03 17:52:28.746833 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:28.746797    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:29.277196 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:29.277166    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:29.401406 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:29.399551    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfcde397-4b17-48a2-bd2b-ad43fd73be72" path="/var/lib/kubelet/pods/dfcde397-4b17-48a2-bd2b-ad43fd73be72/volumes"
Nov 03 17:52:30.093391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:30.093335    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:30.434518 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:30.434435    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:30.781083 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:30.781045    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"]
Nov 03 17:52:30.781323 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:30.781286    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4" podUID="e1e64fd8-6343-4408-9303-a5b712e16186" containerName="istio-proxy" containerID="cri-o://f74b261521d51e01a2eedaab6c0e2080e62ed80118dc944c8781ec46ffb79ae5" gracePeriod=30
Nov 03 17:52:30.877279 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:30.877221    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:31.100272 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:31.100181    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:31.299796 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:31.299754    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:33.703204 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:33.703167    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:33.892640 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:33.892603    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:34.117591 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.117554    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n"]
Nov 03 17:52:34.118283 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.118257    6059 cpu_manager.go:414] "containerMap: removing container" podUID="dfcde397-4b17-48a2-bd2b-ad43fd73be72" containerName="istio-proxy"
Nov 03 17:52:34.118476 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.118284    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfcde397-4b17-48a2-bd2b-ad43fd73be72" containerName="istio-proxy"
Nov 03 17:52:34.118476 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.118311    6059 cpu_manager.go:414] "containerMap: removing container" podUID="dfcde397-4b17-48a2-bd2b-ad43fd73be72" containerName="istio-proxy"
Nov 03 17:52:34.118476 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.118322    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfcde397-4b17-48a2-bd2b-ad43fd73be72" containerName="istio-proxy"
Nov 03 17:52:34.118682 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.118552    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfcde397-4b17-48a2-bd2b-ad43fd73be72" containerName="istio-proxy"
Nov 03 17:52:34.118682 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.118571    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfcde397-4b17-48a2-bd2b-ad43fd73be72" containerName="istio-proxy"
Nov 03 17:52:34.148378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.136671    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n"
Nov 03 17:52:34.193382 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.189122    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n"]
Nov 03 17:52:34.233954 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.232941    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-fe8fde\""
Nov 03 17:52:34.233954 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.233006    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-fe8fde-dockercfg-vktbn\""
Nov 03 17:52:34.253397 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.253343    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9"]
Nov 03 17:52:34.258833 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.258802    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9"
Nov 03 17:52:34.279829 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.279788    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f787e049-8746-4ab8-83f0-d80a63fd3465-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n\" (UID: \"f787e049-8746-4ab8-83f0-d80a63fd3465\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n"
Nov 03 17:52:34.280049 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.280020    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stqbc\" (UniqueName: \"kubernetes.io/projected/f787e049-8746-4ab8-83f0-d80a63fd3465-kube-api-access-stqbc\") pod \"payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n\" (UID: \"f787e049-8746-4ab8-83f0-d80a63fd3465\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n"
Nov 03 17:52:34.316260 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.316226    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9"]
Nov 03 17:52:34.381261 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.381162    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhcgx\" (UniqueName: \"kubernetes.io/projected/8e50ad7e-f804-40ed-b806-b2c904dc96cd-kube-api-access-fhcgx\") pod \"payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9\" (UID: \"8e50ad7e-f804-40ed-b806-b2c904dc96cd\") " pod="openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9"
Nov 03 17:52:34.381261 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.381214    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f787e049-8746-4ab8-83f0-d80a63fd3465-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n\" (UID: \"f787e049-8746-4ab8-83f0-d80a63fd3465\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n"
Nov 03 17:52:34.381541 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.381315    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e50ad7e-f804-40ed-b806-b2c904dc96cd-plugins-config-volume\") pod \"payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9\" (UID: \"8e50ad7e-f804-40ed-b806-b2c904dc96cd\") " pod="openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9"
Nov 03 17:52:34.381541 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.381485    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stqbc\" (UniqueName: \"kubernetes.io/projected/f787e049-8746-4ab8-83f0-d80a63fd3465-kube-api-access-stqbc\") pod \"payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n\" (UID: \"f787e049-8746-4ab8-83f0-d80a63fd3465\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n"
Nov 03 17:52:34.382004 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.381983    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f787e049-8746-4ab8-83f0-d80a63fd3465-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n\" (UID: \"f787e049-8746-4ab8-83f0-d80a63fd3465\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n"
Nov 03 17:52:34.444765 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.444735    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stqbc\" (UniqueName: \"kubernetes.io/projected/f787e049-8746-4ab8-83f0-d80a63fd3465-kube-api-access-stqbc\") pod \"payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n\" (UID: \"f787e049-8746-4ab8-83f0-d80a63fd3465\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n"
Nov 03 17:52:34.469614 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.469587    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n"
Nov 03 17:52:34.482948 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.482908    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhcgx\" (UniqueName: \"kubernetes.io/projected/8e50ad7e-f804-40ed-b806-b2c904dc96cd-kube-api-access-fhcgx\") pod \"payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9\" (UID: \"8e50ad7e-f804-40ed-b806-b2c904dc96cd\") " pod="openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9"
Nov 03 17:52:34.483019 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.482986    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e50ad7e-f804-40ed-b806-b2c904dc96cd-plugins-config-volume\") pod \"payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9\" (UID: \"8e50ad7e-f804-40ed-b806-b2c904dc96cd\") " pod="openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9"
Nov 03 17:52:34.483820 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.483794    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e50ad7e-f804-40ed-b806-b2c904dc96cd-plugins-config-volume\") pod \"payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9\" (UID: \"8e50ad7e-f804-40ed-b806-b2c904dc96cd\") " pod="openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9"
Nov 03 17:52:34.548601 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.548566    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhcgx\" (UniqueName: \"kubernetes.io/projected/8e50ad7e-f804-40ed-b806-b2c904dc96cd-kube-api-access-fhcgx\") pod \"payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9\" (UID: \"8e50ad7e-f804-40ed-b806-b2c904dc96cd\") " pod="openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9"
Nov 03 17:52:34.579214 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.579172    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9"
Nov 03 17:52:34.851936 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.851907    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n"]
Nov 03 17:52:34.853671 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:52:34.853644    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf787e049_8746_4ab8_83f0_d80a63fd3465.slice/crio-5d8b7de90d3ae7a6039bde78c591a50bd43adca55bf01abcd480df53500e6e2a WatchSource:0}: Error finding container 5d8b7de90d3ae7a6039bde78c591a50bd43adca55bf01abcd480df53500e6e2a: Status 404 returned error can't find the container with id 5d8b7de90d3ae7a6039bde78c591a50bd43adca55bf01abcd480df53500e6e2a
Nov 03 17:52:34.965703 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:34.965671    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9"]
Nov 03 17:52:34.967559 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:52:34.967530    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e50ad7e_f804_40ed_b806_b2c904dc96cd.slice/crio-09ba6a91b8489582f666a17a2c826b0ed42291eefb1291346a958c3ea619a682 WatchSource:0}: Error finding container 09ba6a91b8489582f666a17a2c826b0ed42291eefb1291346a958c3ea619a682: Status 404 returned error can't find the container with id 09ba6a91b8489582f666a17a2c826b0ed42291eefb1291346a958c3ea619a682
Nov 03 17:52:35.056302 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:35.056267    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e258a30-m7xfw"]
Nov 03 17:52:35.059390 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:35.059335    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e258a30-m7xfw"]
Nov 03 17:52:35.336482 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:35.336449    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b925950e-a95c-4b36-9103-2f83aff2f1ce" path="/var/lib/kubelet/pods/b925950e-a95c-4b36-9103-2f83aff2f1ce/volumes"
Nov 03 17:52:35.446517 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:35.446478    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n" event={"ID":"f787e049-8746-4ab8-83f0-d80a63fd3465","Type":"ContainerStarted","Data":"bbce04b1b3827634c82e520bcff59e1d1e0fdec966deb426f688c5b4aae0df23"}
Nov 03 17:52:35.446744 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:35.446549    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n" event={"ID":"f787e049-8746-4ab8-83f0-d80a63fd3465","Type":"ContainerStarted","Data":"5d8b7de90d3ae7a6039bde78c591a50bd43adca55bf01abcd480df53500e6e2a"}
Nov 03 17:52:35.446744 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:35.446582    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n"
Nov 03 17:52:35.448675 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:35.448645    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9" event={"ID":"8e50ad7e-f804-40ed-b806-b2c904dc96cd","Type":"ContainerStarted","Data":"35d2c6507f1de876f8b7fa683eb77f11cb04241b13e41995821019d482ba94cf"}
Nov 03 17:52:35.448803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:35.448704    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9"
Nov 03 17:52:35.448803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:35.448729    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9" event={"ID":"8e50ad7e-f804-40ed-b806-b2c904dc96cd","Type":"ContainerStarted","Data":"09ba6a91b8489582f666a17a2c826b0ed42291eefb1291346a958c3ea619a682"}
Nov 03 17:52:35.571524 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:35.571474    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n" podStartSLOduration=1.5714588809999999 podStartE2EDuration="1.571458881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:35.56722939 +0000 UTC m=+2096.364942171" watchObservedRunningTime="2026-11-03 17:52:35.571458881 +0000 UTC m=+2096.369171658"
Nov 03 17:52:35.692001 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:35.691863    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9" podStartSLOduration=1.691843913 podStartE2EDuration="1.691843913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:35.691368081 +0000 UTC m=+2096.489080863" watchObservedRunningTime="2026-11-03 17:52:35.691843913 +0000 UTC m=+2096.489556694"
Nov 03 17:52:36.055627 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.055597    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:52:36.204566 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.204531    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-credential-socket\") pod \"e1e64fd8-6343-4408-9303-a5b712e16186\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") "
Nov 03 17:52:36.204740 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.204651    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1e64fd8-6343-4408-9303-a5b712e16186-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1e64fd8-6343-4408-9303-a5b712e16186-istio-podinfo\") pod \"e1e64fd8-6343-4408-9303-a5b712e16186\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") "
Nov 03 17:52:36.204740 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.204685    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e64fd8-6343-4408-9303-a5b712e16186-istio-token\" (UniqueName: \"kubernetes.io/projected/e1e64fd8-6343-4408-9303-a5b712e16186-istio-token\") pod \"e1e64fd8-6343-4408-9303-a5b712e16186\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") "
Nov 03 17:52:36.204740 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.204716    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-istio-envoy\") pod \"e1e64fd8-6343-4408-9303-a5b712e16186\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") "
Nov 03 17:52:36.204916 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.204752    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-workload-certs\") pod \"e1e64fd8-6343-4408-9303-a5b712e16186\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") "
Nov 03 17:52:36.204916 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.204817    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1e64fd8-6343-4408-9303-a5b712e16186-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1e64fd8-6343-4408-9303-a5b712e16186-istiod-ca-cert\") pod \"e1e64fd8-6343-4408-9303-a5b712e16186\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") "
Nov 03 17:52:36.204916 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.204836    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-credential-socket" pod "e1e64fd8-6343-4408-9303-a5b712e16186" (UID: "e1e64fd8-6343-4408-9303-a5b712e16186"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:52:36.204916 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.204896    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-workload-socket\") pod \"e1e64fd8-6343-4408-9303-a5b712e16186\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") "
Nov 03 17:52:36.205123 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.204987    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e64fd8-6343-4408-9303-a5b712e16186-kube-api-access-rmbx6\" (UniqueName: \"kubernetes.io/projected/e1e64fd8-6343-4408-9303-a5b712e16186-kube-api-access-rmbx6\") pod \"e1e64fd8-6343-4408-9303-a5b712e16186\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") "
Nov 03 17:52:36.205123 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.205053    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-istio-data\") pod \"e1e64fd8-6343-4408-9303-a5b712e16186\" (UID: \"e1e64fd8-6343-4408-9303-a5b712e16186\") "
Nov 03 17:52:36.205238 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.205160    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-workload-certs" pod "e1e64fd8-6343-4408-9303-a5b712e16186" (UID: "e1e64fd8-6343-4408-9303-a5b712e16186"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:52:36.205509 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.205447    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-istio-data" pod "e1e64fd8-6343-4408-9303-a5b712e16186" (UID: "e1e64fd8-6343-4408-9303-a5b712e16186"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:52:36.205509 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.205477    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e64fd8-6343-4408-9303-a5b712e16186-istiod-ca-cert" pod "e1e64fd8-6343-4408-9303-a5b712e16186" (UID: "e1e64fd8-6343-4408-9303-a5b712e16186"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:52:36.205881 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.205588    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-workload-socket" pod "e1e64fd8-6343-4408-9303-a5b712e16186" (UID: "e1e64fd8-6343-4408-9303-a5b712e16186"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:52:36.205881 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.205825    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:36.205881 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.205850    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1e64fd8-6343-4408-9303-a5b712e16186-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:36.205881 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.205866    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:36.205881 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.205880    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:36.206237 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.205894    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:36.207462 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.207435    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e64fd8-6343-4408-9303-a5b712e16186-kube-api-access-rmbx6" pod "e1e64fd8-6343-4408-9303-a5b712e16186" (UID: "e1e64fd8-6343-4408-9303-a5b712e16186"). InnerVolumeSpecName "kube-api-access-rmbx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:52:36.207599 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.207578    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-istio-envoy" pod "e1e64fd8-6343-4408-9303-a5b712e16186" (UID: "e1e64fd8-6343-4408-9303-a5b712e16186"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:52:36.207695 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.207672    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e64fd8-6343-4408-9303-a5b712e16186-istio-token" pod "e1e64fd8-6343-4408-9303-a5b712e16186" (UID: "e1e64fd8-6343-4408-9303-a5b712e16186"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:52:36.208832 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.208741    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1e64fd8-6343-4408-9303-a5b712e16186-istio-podinfo" pod "e1e64fd8-6343-4408-9303-a5b712e16186" (UID: "e1e64fd8-6343-4408-9303-a5b712e16186"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:52:36.307671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.307508    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1e64fd8-6343-4408-9303-a5b712e16186-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:36.307671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.307552    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e64fd8-6343-4408-9303-a5b712e16186-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:36.307671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.307570    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1e64fd8-6343-4408-9303-a5b712e16186-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:36.307671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.307633    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmbx6\" (UniqueName: \"kubernetes.io/projected/e1e64fd8-6343-4408-9303-a5b712e16186-kube-api-access-rmbx6\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:36.464458 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.454819    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:36.474404 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.474336    6059 generic.go:356] "Generic (PLEG): container finished" podID="e1e64fd8-6343-4408-9303-a5b712e16186" containerID="f74b261521d51e01a2eedaab6c0e2080e62ed80118dc944c8781ec46ffb79ae5" exitCode=0
Nov 03 17:52:36.474404 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.474389    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4" event={"ID":"e1e64fd8-6343-4408-9303-a5b712e16186","Type":"ContainerDied","Data":"f74b261521d51e01a2eedaab6c0e2080e62ed80118dc944c8781ec46ffb79ae5"}
Nov 03 17:52:36.474656 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.474434    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"
Nov 03 17:52:36.474656 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.474486    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4" event={"ID":"e1e64fd8-6343-4408-9303-a5b712e16186","Type":"ContainerDied","Data":"096ca15e1c422cb35cfd673b6fd51664964237c57b5350bfc80f40cdb3ff5369"}
Nov 03 17:52:36.474656 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.474520    6059 scope.go:122] "RemoveContainer" containerID="f74b261521d51e01a2eedaab6c0e2080e62ed80118dc944c8781ec46ffb79ae5"
Nov 03 17:52:36.487021 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.486994    6059 scope.go:122] "RemoveContainer" containerID="f74b261521d51e01a2eedaab6c0e2080e62ed80118dc944c8781ec46ffb79ae5"
Nov 03 17:52:36.487398 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:52:36.487368    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f74b261521d51e01a2eedaab6c0e2080e62ed80118dc944c8781ec46ffb79ae5\": container with ID starting with f74b261521d51e01a2eedaab6c0e2080e62ed80118dc944c8781ec46ffb79ae5 not found: ID does not exist" containerID="f74b261521d51e01a2eedaab6c0e2080e62ed80118dc944c8781ec46ffb79ae5"
Nov 03 17:52:36.487486 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.487417    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f74b261521d51e01a2eedaab6c0e2080e62ed80118dc944c8781ec46ffb79ae5"} err="failed to get container status \"f74b261521d51e01a2eedaab6c0e2080e62ed80118dc944c8781ec46ffb79ae5\": rpc error: code = NotFound desc = could not find container \"f74b261521d51e01a2eedaab6c0e2080e62ed80118dc944c8781ec46ffb79ae5\": container with ID starting with f74b261521d51e01a2eedaab6c0e2080e62ed80118dc944c8781ec46ffb79ae5 not found: ID does not exist"
Nov 03 17:52:36.647255 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.647221    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"]
Nov 03 17:52:36.652497 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.652466    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-c6sp4"]
Nov 03 17:52:36.862963 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:52:36.862878    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:52:36.940610 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.940566    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:36.955394 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:36.955347    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn"]
Nov 03 17:52:37.060426 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:37.060396    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc"]
Nov 03 17:52:37.321674 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:37.321627    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:37.342921 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:37.340150    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1e64fd8-6343-4408-9303-a5b712e16186" path="/var/lib/kubelet/pods/e1e64fd8-6343-4408-9303-a5b712e16186/volumes"
Nov 03 17:52:37.513081 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:37.513046    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:39.692007 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:39.691976    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 17:52:39.701416 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:39.701391    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 17:52:39.708929 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:39.708905    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:52:39.717778 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:39.717760    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:52:41.224694 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:41.224651    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:41.955962 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:41.955897    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn" podUID="93cfc7b5-27fd-4c80-849f-a331f4e6f7c9" containerName="payload-processing" containerID="cri-o://56e611a62716fad918dd966d295f7eecb490bf2d7fd3a6e717988112f9cd8aa6" gracePeriod=25
Nov 03 17:52:42.061110 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.061063    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc" podUID="fa25bbe8-398d-4b88-a513-a1db1a39ace9" containerName="payload-pre-processing" containerID="cri-o://115f83e568a23a1024bcae53ff2d88132cff7e1f60d8a0b1f059ac3f8217ea0b" gracePeriod=25
Nov 03 17:52:42.126465 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.126432    6059 scope.go:122] "RemoveContainer" containerID="06067fceb6824cf75a50d6bf40747845a477a2f486dc937b9e6fd3d7e7b35737"
Nov 03 17:52:42.177610 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.177577    6059 scope.go:122] "RemoveContainer" containerID="114aee1c147b06f4d76cca25a44faaa26b21b4fdff86d958fa15a87074bd90d7"
Nov 03 17:52:42.192579 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.192555    6059 scope.go:122] "RemoveContainer" containerID="d99de479f8eee9d7daf039118acb8f8042d8595755f009e50403174e4a5014ee"
Nov 03 17:52:42.222799 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.222775    6059 scope.go:122] "RemoveContainer" containerID="907ea931adc6c01dbd70c00fd414e76460f1fca13b4e60527830c38501a4ba77"
Nov 03 17:52:42.226152 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.226128    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn"
Nov 03 17:52:42.233184 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.233136    6059 scope.go:122] "RemoveContainer" containerID="e5d375629f06f9182487179b5a75a2a2cb619447cc332e91f7e79c50e7b6d5a9"
Nov 03 17:52:42.270055 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.270028    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:42.348947 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.348919    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc"
Nov 03 17:52:42.395976 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.395945    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93cfc7b5-27fd-4c80-849f-a331f4e6f7c9-kube-api-access-nskrv\" (UniqueName: \"kubernetes.io/projected/93cfc7b5-27fd-4c80-849f-a331f4e6f7c9-kube-api-access-nskrv\") pod \"93cfc7b5-27fd-4c80-849f-a331f4e6f7c9\" (UID: \"93cfc7b5-27fd-4c80-849f-a331f4e6f7c9\") "
Nov 03 17:52:42.396184 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.396021    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93cfc7b5-27fd-4c80-849f-a331f4e6f7c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93cfc7b5-27fd-4c80-849f-a331f4e6f7c9-plugins-config-volume\") pod \"93cfc7b5-27fd-4c80-849f-a331f4e6f7c9\" (UID: \"93cfc7b5-27fd-4c80-849f-a331f4e6f7c9\") "
Nov 03 17:52:42.396460 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.396434    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93cfc7b5-27fd-4c80-849f-a331f4e6f7c9-plugins-config-volume" pod "93cfc7b5-27fd-4c80-849f-a331f4e6f7c9" (UID: "93cfc7b5-27fd-4c80-849f-a331f4e6f7c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:52:42.398041 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.398016    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93cfc7b5-27fd-4c80-849f-a331f4e6f7c9-kube-api-access-nskrv" pod "93cfc7b5-27fd-4c80-849f-a331f4e6f7c9" (UID: "93cfc7b5-27fd-4c80-849f-a331f4e6f7c9"). InnerVolumeSpecName "kube-api-access-nskrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:52:42.497267 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.497172    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa25bbe8-398d-4b88-a513-a1db1a39ace9-kube-api-access-5nxcj\" (UniqueName: \"kubernetes.io/projected/fa25bbe8-398d-4b88-a513-a1db1a39ace9-kube-api-access-5nxcj\") pod \"fa25bbe8-398d-4b88-a513-a1db1a39ace9\" (UID: \"fa25bbe8-398d-4b88-a513-a1db1a39ace9\") "
Nov 03 17:52:42.497484 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.497418    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa25bbe8-398d-4b88-a513-a1db1a39ace9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa25bbe8-398d-4b88-a513-a1db1a39ace9-plugins-config-volume\") pod \"fa25bbe8-398d-4b88-a513-a1db1a39ace9\" (UID: \"fa25bbe8-398d-4b88-a513-a1db1a39ace9\") "
Nov 03 17:52:42.497784 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.497761    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa25bbe8-398d-4b88-a513-a1db1a39ace9-plugins-config-volume" pod "fa25bbe8-398d-4b88-a513-a1db1a39ace9" (UID: "fa25bbe8-398d-4b88-a513-a1db1a39ace9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:52:42.497856 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.497807    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nskrv\" (UniqueName: \"kubernetes.io/projected/93cfc7b5-27fd-4c80-849f-a331f4e6f7c9-kube-api-access-nskrv\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:42.497856 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.497822    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93cfc7b5-27fd-4c80-849f-a331f4e6f7c9-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:42.499409 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.499385    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa25bbe8-398d-4b88-a513-a1db1a39ace9-kube-api-access-5nxcj" pod "fa25bbe8-398d-4b88-a513-a1db1a39ace9" (UID: "fa25bbe8-398d-4b88-a513-a1db1a39ace9"). InnerVolumeSpecName "kube-api-access-5nxcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:52:42.524869 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.524828    6059 generic.go:356] "Generic (PLEG): container finished" podID="93cfc7b5-27fd-4c80-849f-a331f4e6f7c9" containerID="56e611a62716fad918dd966d295f7eecb490bf2d7fd3a6e717988112f9cd8aa6" exitCode=0
Nov 03 17:52:42.524869 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.524869    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn"
Nov 03 17:52:42.525130 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.524912    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn" event={"ID":"93cfc7b5-27fd-4c80-849f-a331f4e6f7c9","Type":"ContainerDied","Data":"56e611a62716fad918dd966d295f7eecb490bf2d7fd3a6e717988112f9cd8aa6"}
Nov 03 17:52:42.525130 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.524974    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn" event={"ID":"93cfc7b5-27fd-4c80-849f-a331f4e6f7c9","Type":"ContainerDied","Data":"db4474fc2c5bd840efc5275c8a57439b8f95f39140d41e482a8b96f6a7c5659c"}
Nov 03 17:52:42.525130 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.525007    6059 scope.go:122] "RemoveContainer" containerID="56e611a62716fad918dd966d295f7eecb490bf2d7fd3a6e717988112f9cd8aa6"
Nov 03 17:52:42.526677 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.526653    6059 generic.go:356] "Generic (PLEG): container finished" podID="fa25bbe8-398d-4b88-a513-a1db1a39ace9" containerID="115f83e568a23a1024bcae53ff2d88132cff7e1f60d8a0b1f059ac3f8217ea0b" exitCode=0
Nov 03 17:52:42.526807 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.526713    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc"
Nov 03 17:52:42.526807 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.526731    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc" event={"ID":"fa25bbe8-398d-4b88-a513-a1db1a39ace9","Type":"ContainerDied","Data":"115f83e568a23a1024bcae53ff2d88132cff7e1f60d8a0b1f059ac3f8217ea0b"}
Nov 03 17:52:42.526807 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.526800    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc" event={"ID":"fa25bbe8-398d-4b88-a513-a1db1a39ace9","Type":"ContainerDied","Data":"aecc944de54ddd4465975f9a902006b4361b90517815d74b5688c73a14da72df"}
Nov 03 17:52:42.539066 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.539039    6059 scope.go:122] "RemoveContainer" containerID="56e611a62716fad918dd966d295f7eecb490bf2d7fd3a6e717988112f9cd8aa6"
Nov 03 17:52:42.539451 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:52:42.539425    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56e611a62716fad918dd966d295f7eecb490bf2d7fd3a6e717988112f9cd8aa6\": container with ID starting with 56e611a62716fad918dd966d295f7eecb490bf2d7fd3a6e717988112f9cd8aa6 not found: ID does not exist" containerID="56e611a62716fad918dd966d295f7eecb490bf2d7fd3a6e717988112f9cd8aa6"
Nov 03 17:52:42.539543 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.539465    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e611a62716fad918dd966d295f7eecb490bf2d7fd3a6e717988112f9cd8aa6"} err="failed to get container status \"56e611a62716fad918dd966d295f7eecb490bf2d7fd3a6e717988112f9cd8aa6\": rpc error: code = NotFound desc = could not find container \"56e611a62716fad918dd966d295f7eecb490bf2d7fd3a6e717988112f9cd8aa6\": container with ID starting with 56e611a62716fad918dd966d295f7eecb490bf2d7fd3a6e717988112f9cd8aa6 not found: ID does not exist"
Nov 03 17:52:42.539543 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.539491    6059 scope.go:122] "RemoveContainer" containerID="115f83e568a23a1024bcae53ff2d88132cff7e1f60d8a0b1f059ac3f8217ea0b"
Nov 03 17:52:42.552111 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.552086    6059 scope.go:122] "RemoveContainer" containerID="115f83e568a23a1024bcae53ff2d88132cff7e1f60d8a0b1f059ac3f8217ea0b"
Nov 03 17:52:42.552462 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:52:42.552438    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"115f83e568a23a1024bcae53ff2d88132cff7e1f60d8a0b1f059ac3f8217ea0b\": container with ID starting with 115f83e568a23a1024bcae53ff2d88132cff7e1f60d8a0b1f059ac3f8217ea0b not found: ID does not exist" containerID="115f83e568a23a1024bcae53ff2d88132cff7e1f60d8a0b1f059ac3f8217ea0b"
Nov 03 17:52:42.552579 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.552470    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"115f83e568a23a1024bcae53ff2d88132cff7e1f60d8a0b1f059ac3f8217ea0b"} err="failed to get container status \"115f83e568a23a1024bcae53ff2d88132cff7e1f60d8a0b1f059ac3f8217ea0b\": rpc error: code = NotFound desc = could not find container \"115f83e568a23a1024bcae53ff2d88132cff7e1f60d8a0b1f059ac3f8217ea0b\": container with ID starting with 115f83e568a23a1024bcae53ff2d88132cff7e1f60d8a0b1f059ac3f8217ea0b not found: ID does not exist"
Nov 03 17:52:42.599203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.599162    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa25bbe8-398d-4b88-a513-a1db1a39ace9-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:42.599203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.599194    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nxcj\" (UniqueName: \"kubernetes.io/projected/fa25bbe8-398d-4b88-a513-a1db1a39ace9-kube-api-access-5nxcj\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:42.697460 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.697424    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn"]
Nov 03 17:52:42.700349 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.700323    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e095fe-6db8bb44df-ndxgn"]
Nov 03 17:52:42.813091 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.813005    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc"]
Nov 03 17:52:42.816565 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:42.816531    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e095fe-7485dd669f-wr8pc"]
Nov 03 17:52:43.081481 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:43.081303    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:43.338239 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:43.338161    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93cfc7b5-27fd-4c80-849f-a331f4e6f7c9" path="/var/lib/kubelet/pods/93cfc7b5-27fd-4c80-849f-a331f4e6f7c9/volumes"
Nov 03 17:52:43.338644 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:43.338623    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa25bbe8-398d-4b88-a513-a1db1a39ace9" path="/var/lib/kubelet/pods/fa25bbe8-398d-4b88-a513-a1db1a39ace9/volumes"
Nov 03 17:52:45.725219 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:45.725171    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:46.478208 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:46.478176    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n"
Nov 03 17:52:46.478635 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:46.478611    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9"
Nov 03 17:52:50.224712 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:50.224681    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:54.729406 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:54.729327    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:55.051014 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:55.050943    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq"]
Nov 03 17:52:55.102594 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:55.102555    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1e2f82f-vjxqq"]
Nov 03 17:52:55.335582 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:55.335504    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28714918-4efe-4c05-af87-04e1200ec028" path="/var/lib/kubelet/pods/28714918-4efe-4c05-af87-04e1200ec028/volumes"
Nov 03 17:52:55.345005 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:55.344977    6059 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","podc4f61445-3d05-4900-9eb7-5d8a55eadf66"] err="unable to destroy cgroup paths for cgroup [kubepods burstable podc4f61445-3d05-4900-9eb7-5d8a55eadf66] : Timed out while waiting for systemd to remove kubepods-burstable-podc4f61445_3d05_4900_9eb7_5d8a55eadf66.slice"
Nov 03 17:52:55.345121 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:52:55.345010    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable podc4f61445-3d05-4900-9eb7-5d8a55eadf66] : unable to destroy cgroup paths for cgroup [kubepods burstable podc4f61445-3d05-4900-9eb7-5d8a55eadf66] : Timed out while waiting for systemd to remove kubepods-burstable-podc4f61445_3d05_4900_9eb7_5d8a55eadf66.slice" pod="openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c" podUID="c4f61445-3d05-4900-9eb7-5d8a55eadf66"
Nov 03 17:52:55.616962 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:55.616883    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c"
Nov 03 17:52:55.856192 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:55.856159    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c"]
Nov 03 17:52:55.859646 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:55.859621    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-382697-7957766c97-6gt6c"]
Nov 03 17:52:57.335943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:57.335910    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4f61445-3d05-4900-9eb7-5d8a55eadf66" path="/var/lib/kubelet/pods/c4f61445-3d05-4900-9eb7-5d8a55eadf66/volumes"
Nov 03 17:52:59.171180 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:59.171149    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:59.343570 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:59.343537    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:52:59.585166 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:52:59.585134    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:03.820165 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:03.820128    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:04.385014 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:04.384981    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:06.862089 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:06.862048    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:53:07.755660 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:07.755615    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:09.475706 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:09.475635    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:14.100976 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:14.100913    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:15.990962 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:15.990923    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:16.235814 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:16.235766    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:17.103235 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.103198    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f1cb49-ft257"]
Nov 03 17:53:17.105832 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.105804    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f1cb49-ft257"]
Nov 03 17:53:17.190732 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.190694    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:17.335827 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.335789    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c82ee932-a22b-4780-ab90-065599279e4a" path="/var/lib/kubelet/pods/c82ee932-a22b-4780-ab90-065599279e4a/volumes"
Nov 03 17:53:17.540306 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.540256    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:17.849864 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.849788    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-662943ba-l8qpg"]
Nov 03 17:53:17.850393 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.850377    6059 cpu_manager.go:414] "containerMap: removing container" podUID="fa25bbe8-398d-4b88-a513-a1db1a39ace9" containerName="payload-pre-processing"
Nov 03 17:53:17.850446 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.850394    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa25bbe8-398d-4b88-a513-a1db1a39ace9" containerName="payload-pre-processing"
Nov 03 17:53:17.850446 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.850414    6059 cpu_manager.go:414] "containerMap: removing container" podUID="93cfc7b5-27fd-4c80-849f-a331f4e6f7c9" containerName="payload-processing"
Nov 03 17:53:17.850446 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.850419    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93cfc7b5-27fd-4c80-849f-a331f4e6f7c9" containerName="payload-processing"
Nov 03 17:53:17.850446 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.850428    6059 cpu_manager.go:414] "containerMap: removing container" podUID="e1e64fd8-6343-4408-9303-a5b712e16186" containerName="istio-proxy"
Nov 03 17:53:17.850446 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.850432    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e64fd8-6343-4408-9303-a5b712e16186" containerName="istio-proxy"
Nov 03 17:53:17.850607 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.850527    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa25bbe8-398d-4b88-a513-a1db1a39ace9" containerName="payload-pre-processing"
Nov 03 17:53:17.850607 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.850538    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="93cfc7b5-27fd-4c80-849f-a331f4e6f7c9" containerName="payload-processing"
Nov 03 17:53:17.850607 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.850544    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e64fd8-6343-4408-9303-a5b712e16186" containerName="istio-proxy"
Nov 03 17:53:17.855272 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.855255    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662943ba-l8qpg"
Nov 03 17:53:17.914332 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.914295    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-662943ba-l8qpg"]
Nov 03 17:53:17.943562 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.943527    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4c6f904-64a1-4306-9810-47623362da96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-662943ba-l8qpg\" (UID: \"f4c6f904-64a1-4306-9810-47623362da96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662943ba-l8qpg"
Nov 03 17:53:17.943762 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.943594    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7cks\" (UniqueName: \"kubernetes.io/projected/f4c6f904-64a1-4306-9810-47623362da96-kube-api-access-t7cks\") pod \"maas-api-revoke-sub-662943ba-l8qpg\" (UID: \"f4c6f904-64a1-4306-9810-47623362da96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662943ba-l8qpg"
Nov 03 17:53:17.943842 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.943763    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4c6f904-64a1-4306-9810-47623362da96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-662943ba-l8qpg\" (UID: \"f4c6f904-64a1-4306-9810-47623362da96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662943ba-l8qpg"
Nov 03 17:53:17.961677 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.961633    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:53:17.961677 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.961632    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:53:17.961677 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.961638    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:53:17.961677 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:17.961641    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:53:18.018624 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:18.018575    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"]
Nov 03 17:53:18.019103 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:18.019056    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf" podUID="f7c9e388-cf4f-4f23-98df-b36b39615609" containerName="istio-proxy" containerID="cri-o://6fd130be3e200e333223d29538d46223ffea288e04a2fe3c3c852c1e31c2d1ba" gracePeriod=30
Nov 03 17:53:18.023544 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:18.023519    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"]
Nov 03 17:53:18.023725 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:18.023707    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx" podUID="4447e11f-6bed-4da8-97b6-c331cd08a4eb" containerName="istio-proxy" containerID="cri-o://984ad4ded1213f3fd605fa12b0d103cca53531b364c829ddb3542f43b433f88f" gracePeriod=30
Nov 03 17:53:18.045371 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:18.045325    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4c6f904-64a1-4306-9810-47623362da96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-662943ba-l8qpg\" (UID: \"f4c6f904-64a1-4306-9810-47623362da96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662943ba-l8qpg"
Nov 03 17:53:18.045541 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:18.045429    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7cks\" (UniqueName: \"kubernetes.io/projected/f4c6f904-64a1-4306-9810-47623362da96-kube-api-access-t7cks\") pod \"maas-api-revoke-sub-662943ba-l8qpg\" (UID: \"f4c6f904-64a1-4306-9810-47623362da96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662943ba-l8qpg"
Nov 03 17:53:18.045607 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:18.045562    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4c6f904-64a1-4306-9810-47623362da96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-662943ba-l8qpg\" (UID: \"f4c6f904-64a1-4306-9810-47623362da96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662943ba-l8qpg"
Nov 03 17:53:18.046059 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:18.046028    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4c6f904-64a1-4306-9810-47623362da96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-662943ba-l8qpg\" (UID: \"f4c6f904-64a1-4306-9810-47623362da96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662943ba-l8qpg"
Nov 03 17:53:18.047836 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:18.047815    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4c6f904-64a1-4306-9810-47623362da96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-662943ba-l8qpg\" (UID: \"f4c6f904-64a1-4306-9810-47623362da96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662943ba-l8qpg"
Nov 03 17:53:18.091727 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:18.091685    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:18.220524 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:18.220496    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7cks\" (UniqueName: \"kubernetes.io/projected/f4c6f904-64a1-4306-9810-47623362da96-kube-api-access-t7cks\") pod \"maas-api-revoke-sub-662943ba-l8qpg\" (UID: \"f4c6f904-64a1-4306-9810-47623362da96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662943ba-l8qpg"
Nov 03 17:53:18.469519 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:18.469487    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662943ba-l8qpg"
Nov 03 17:53:18.853487 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:18.853465    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-662943ba-l8qpg"]
Nov 03 17:53:18.856596 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:53:18.856570    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c6f904_64a1_4306_9810_47623362da96.slice/crio-3580dd357067f71505e3954f0688b925a92404307d41d01b313f770be07e2132 WatchSource:0}: Error finding container 3580dd357067f71505e3954f0688b925a92404307d41d01b313f770be07e2132: Status 404 returned error can't find the container with id 3580dd357067f71505e3954f0688b925a92404307d41d01b313f770be07e2132
Nov 03 17:53:19.022314 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:19.022238    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:19.213461 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:19.213408    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:19.693879 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:19.693845    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:19.804845 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:19.804816    6059 generic.go:356] "Generic (PLEG): container finished" podID="945d8e0f-a47e-49e3-ae35-d25ad12aca81" containerID="5f7c4e9223356daff37422b90e7e797621341a59336d0ee6fd287f95389542f4" exitCode=0
Nov 03 17:53:19.805030 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:19.804884    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp" event={"ID":"945d8e0f-a47e-49e3-ae35-d25ad12aca81","Type":"ContainerDied","Data":"5f7c4e9223356daff37422b90e7e797621341a59336d0ee6fd287f95389542f4"}
Nov 03 17:53:19.805310 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:19.805287    6059 scope.go:122] "RemoveContainer" containerID="5f7c4e9223356daff37422b90e7e797621341a59336d0ee6fd287f95389542f4"
Nov 03 17:53:19.806769 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:19.806743    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:53:19.806877 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:19.806811    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:53:19.806877 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:19.806815    6059 generic.go:356] "Generic (PLEG): container finished" podID="f4c6f904-64a1-4306-9810-47623362da96" containerID="e5e4e928d461b5416aefbb77e69c6d38bca14c28a9b14d99493ac25150ff223a" exitCode=0
Nov 03 17:53:19.806877 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:19.806848    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:53:19.806877 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:19.806859    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662943ba-l8qpg" event={"ID":"f4c6f904-64a1-4306-9810-47623362da96","Type":"ContainerDied","Data":"e5e4e928d461b5416aefbb77e69c6d38bca14c28a9b14d99493ac25150ff223a"}
Nov 03 17:53:19.807038 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:19.806885    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662943ba-l8qpg" event={"ID":"f4c6f904-64a1-4306-9810-47623362da96","Type":"ContainerStarted","Data":"3580dd357067f71505e3954f0688b925a92404307d41d01b313f770be07e2132"}
Nov 03 17:53:20.815061 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:20.815015    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp" event={"ID":"945d8e0f-a47e-49e3-ae35-d25ad12aca81","Type":"ContainerStarted","Data":"b122d7e190368d629cf5ffda951a54dfa3df66efce74488bad8be6d34e2518e4"}
Nov 03 17:53:20.914203 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:20.914181    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:53:20.914483 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:20.914412    6059 patch_prober.go:29] interesting pod/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 17:53:20.914647 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:20.914624    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp" podUID="945d8e0f-a47e-49e3-ae35-d25ad12aca81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 17:53:20.992780 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:20.992756    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662943ba-l8qpg"
Nov 03 17:53:21.114522 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:21.114441    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4c6f904-64a1-4306-9810-47623362da96-kube-api-access-t7cks\" (UniqueName: \"kubernetes.io/projected/f4c6f904-64a1-4306-9810-47623362da96-kube-api-access-t7cks\") pod \"f4c6f904-64a1-4306-9810-47623362da96\" (UID: \"f4c6f904-64a1-4306-9810-47623362da96\") "
Nov 03 17:53:21.114522 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:21.114517    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4c6f904-64a1-4306-9810-47623362da96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4c6f904-64a1-4306-9810-47623362da96-maas-api-openshift-ca\") pod \"f4c6f904-64a1-4306-9810-47623362da96\" (UID: \"f4c6f904-64a1-4306-9810-47623362da96\") "
Nov 03 17:53:21.114713 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:21.114569    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4c6f904-64a1-4306-9810-47623362da96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4c6f904-64a1-4306-9810-47623362da96-maas-api-xks-ca\") pod \"f4c6f904-64a1-4306-9810-47623362da96\" (UID: \"f4c6f904-64a1-4306-9810-47623362da96\") "
Nov 03 17:53:21.114957 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:21.114916    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c6f904-64a1-4306-9810-47623362da96-maas-api-openshift-ca" pod "f4c6f904-64a1-4306-9810-47623362da96" (UID: "f4c6f904-64a1-4306-9810-47623362da96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:53:21.115207 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:21.115190    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4c6f904-64a1-4306-9810-47623362da96-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:21.116654 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:21.116623    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4c6f904-64a1-4306-9810-47623362da96-maas-api-xks-ca" pod "f4c6f904-64a1-4306-9810-47623362da96" (UID: "f4c6f904-64a1-4306-9810-47623362da96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:53:21.116744 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:21.116689    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c6f904-64a1-4306-9810-47623362da96-kube-api-access-t7cks" pod "f4c6f904-64a1-4306-9810-47623362da96" (UID: "f4c6f904-64a1-4306-9810-47623362da96"). InnerVolumeSpecName "kube-api-access-t7cks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:53:21.216239 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:21.216205    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7cks\" (UniqueName: \"kubernetes.io/projected/f4c6f904-64a1-4306-9810-47623362da96-kube-api-access-t7cks\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:21.216239 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:21.216237    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4c6f904-64a1-4306-9810-47623362da96-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:21.565655 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:21.565600    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:21.823310 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:21.823269    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662943ba-l8qpg"
Nov 03 17:53:21.823310 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:21.823293    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662943ba-l8qpg" event={"ID":"f4c6f904-64a1-4306-9810-47623362da96","Type":"ContainerDied","Data":"3580dd357067f71505e3954f0688b925a92404307d41d01b313f770be07e2132"}
Nov 03 17:53:21.823867 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:21.823344    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3580dd357067f71505e3954f0688b925a92404307d41d01b313f770be07e2132"
Nov 03 17:53:21.914416 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:21.914380    6059 patch_prober.go:29] interesting pod/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 17:53:21.914582 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:21.914442    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp" podUID="945d8e0f-a47e-49e3-ae35-d25ad12aca81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 17:53:22.131536 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:22.131446    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"]
Nov 03 17:53:22.147391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:22.134876    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"]
Nov 03 17:53:22.293403 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:22.293350    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:22.478880 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:22.478847    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:22.915344 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:22.915251    6059 patch_prober.go:29] interesting pod/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 17:53:22.916800 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:22.916757    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp" podUID="945d8e0f-a47e-49e3-ae35-d25ad12aca81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 17:53:23.366154 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.366134    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:53:23.388734 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.388711    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:53:23.397399 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.397372    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"]
Nov 03 17:53:23.397924 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.397909    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f4c6f904-64a1-4306-9810-47623362da96" containerName="revoke-keys"
Nov 03 17:53:23.398003 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.397925    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4c6f904-64a1-4306-9810-47623362da96" containerName="revoke-keys"
Nov 03 17:53:23.398003 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.397949    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f7c9e388-cf4f-4f23-98df-b36b39615609" containerName="istio-proxy"
Nov 03 17:53:23.398003 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.397954    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7c9e388-cf4f-4f23-98df-b36b39615609" containerName="istio-proxy"
Nov 03 17:53:23.398003 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.397969    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f7c9e388-cf4f-4f23-98df-b36b39615609" containerName="istio-proxy"
Nov 03 17:53:23.398003 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.397974    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7c9e388-cf4f-4f23-98df-b36b39615609" containerName="istio-proxy"
Nov 03 17:53:23.398003 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.397983    6059 cpu_manager.go:414] "containerMap: removing container" podUID="4447e11f-6bed-4da8-97b6-c331cd08a4eb" containerName="istio-proxy"
Nov 03 17:53:23.398003 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.397987    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4447e11f-6bed-4da8-97b6-c331cd08a4eb" containerName="istio-proxy"
Nov 03 17:53:23.398391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.398061    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7c9e388-cf4f-4f23-98df-b36b39615609" containerName="istio-proxy"
Nov 03 17:53:23.398391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.398072    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4447e11f-6bed-4da8-97b6-c331cd08a4eb" containerName="istio-proxy"
Nov 03 17:53:23.398391 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.398084    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4c6f904-64a1-4306-9810-47623362da96" containerName="revoke-keys"
Nov 03 17:53:23.401524 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.401503    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"
Nov 03 17:53:23.472497 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.472422    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"]
Nov 03 17:53:23.480649 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.480627    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4447e11f-6bed-4da8-97b6-c331cd08a4eb-kube-api-access-kv9x4\" (UniqueName: \"kubernetes.io/projected/4447e11f-6bed-4da8-97b6-c331cd08a4eb-kube-api-access-kv9x4\") pod \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") "
Nov 03 17:53:23.480875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.480862    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-credential-socket\") pod \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") "
Nov 03 17:53:23.480974 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.480964    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-token\" (UniqueName: \"kubernetes.io/projected/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-token\") pod \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") "
Nov 03 17:53:23.481059 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.481050    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7c9e388-cf4f-4f23-98df-b36b39615609-kube-api-access-c72bq\" (UniqueName: \"kubernetes.io/projected/f7c9e388-cf4f-4f23-98df-b36b39615609-kube-api-access-c72bq\") pod \"f7c9e388-cf4f-4f23-98df-b36b39615609\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") "
Nov 03 17:53:23.481129 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.481121    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istiod-ca-cert\") pod \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") "
Nov 03 17:53:23.481201 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.481193    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-envoy\") pod \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") "
Nov 03 17:53:23.481322 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.481314    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-envoy\") pod \"f7c9e388-cf4f-4f23-98df-b36b39615609\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") "
Nov 03 17:53:23.481416 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.481407    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-data\") pod \"f7c9e388-cf4f-4f23-98df-b36b39615609\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") "
Nov 03 17:53:23.481515 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.481506    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-workload-socket\") pod \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") "
Nov 03 17:53:23.481614 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.481604    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7c9e388-cf4f-4f23-98df-b36b39615609-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c9e388-cf4f-4f23-98df-b36b39615609-istiod-ca-cert\") pod \"f7c9e388-cf4f-4f23-98df-b36b39615609\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") "
Nov 03 17:53:23.481681 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.481672    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-workload-certs\") pod \"f7c9e388-cf4f-4f23-98df-b36b39615609\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") "
Nov 03 17:53:23.481751 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.481737    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-data\") pod \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") "
Nov 03 17:53:23.481868 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.481850    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-workload-certs\") pod \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") "
Nov 03 17:53:23.481999 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.481984    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-token\" (UniqueName: \"kubernetes.io/projected/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-token\") pod \"f7c9e388-cf4f-4f23-98df-b36b39615609\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") "
Nov 03 17:53:23.482162 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.482145    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-workload-socket\") pod \"f7c9e388-cf4f-4f23-98df-b36b39615609\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") "
Nov 03 17:53:23.482272 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.482257    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-podinfo\") pod \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\" (UID: \"4447e11f-6bed-4da8-97b6-c331cd08a4eb\") "
Nov 03 17:53:23.482425 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.482408    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-credential-socket\") pod \"f7c9e388-cf4f-4f23-98df-b36b39615609\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") "
Nov 03 17:53:23.482666 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.482648    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-podinfo\") pod \"f7c9e388-cf4f-4f23-98df-b36b39615609\" (UID: \"f7c9e388-cf4f-4f23-98df-b36b39615609\") "
Nov 03 17:53:23.483071 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.483046    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5556f3c-1664-4f50-8f40-e4a44724c27d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8\" (UID: \"f5556f3c-1664-4f50-8f40-e4a44724c27d\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"
Nov 03 17:53:23.483379 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.483333    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5556f3c-1664-4f50-8f40-e4a44724c27d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8\" (UID: \"f5556f3c-1664-4f50-8f40-e4a44724c27d\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"
Nov 03 17:53:23.483502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.483485    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpwrx\" (UniqueName: \"kubernetes.io/projected/f5556f3c-1664-4f50-8f40-e4a44724c27d-kube-api-access-gpwrx\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8\" (UID: \"f5556f3c-1664-4f50-8f40-e4a44724c27d\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"
Nov 03 17:53:23.483564 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.483540    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5556f3c-1664-4f50-8f40-e4a44724c27d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8\" (UID: \"f5556f3c-1664-4f50-8f40-e4a44724c27d\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"
Nov 03 17:53:23.483905 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.481192    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-credential-socket" pod "4447e11f-6bed-4da8-97b6-c331cd08a4eb" (UID: "4447e11f-6bed-4da8-97b6-c331cd08a4eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:23.483998 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.483897    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-credential-socket" pod "f7c9e388-cf4f-4f23-98df-b36b39615609" (UID: "f7c9e388-cf4f-4f23-98df-b36b39615609"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:23.483998 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.482233    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7c9e388-cf4f-4f23-98df-b36b39615609-istiod-ca-cert" pod "f7c9e388-cf4f-4f23-98df-b36b39615609" (UID: "f7c9e388-cf4f-4f23-98df-b36b39615609"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:53:23.483998 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.482481    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-data" pod "f7c9e388-cf4f-4f23-98df-b36b39615609" (UID: "f7c9e388-cf4f-4f23-98df-b36b39615609"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:23.483998 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.482787    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-workload-socket" pod "4447e11f-6bed-4da8-97b6-c331cd08a4eb" (UID: "4447e11f-6bed-4da8-97b6-c331cd08a4eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:23.483998 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.482862    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istiod-ca-cert" pod "4447e11f-6bed-4da8-97b6-c331cd08a4eb" (UID: "4447e11f-6bed-4da8-97b6-c331cd08a4eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:53:23.483998 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.483134    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-data" pod "4447e11f-6bed-4da8-97b6-c331cd08a4eb" (UID: "4447e11f-6bed-4da8-97b6-c331cd08a4eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:23.483998 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.483203    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4447e11f-6bed-4da8-97b6-c331cd08a4eb-kube-api-access-kv9x4" pod "4447e11f-6bed-4da8-97b6-c331cd08a4eb" (UID: "4447e11f-6bed-4da8-97b6-c331cd08a4eb"). InnerVolumeSpecName "kube-api-access-kv9x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:53:23.483998 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.483319    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-workload-certs" pod "f7c9e388-cf4f-4f23-98df-b36b39615609" (UID: "f7c9e388-cf4f-4f23-98df-b36b39615609"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:23.483998 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.483496    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-workload-certs" pod "4447e11f-6bed-4da8-97b6-c331cd08a4eb" (UID: "4447e11f-6bed-4da8-97b6-c331cd08a4eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:23.483998 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.483715    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-workload-socket" pod "f7c9e388-cf4f-4f23-98df-b36b39615609" (UID: "f7c9e388-cf4f-4f23-98df-b36b39615609"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:23.484921 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.484902    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-envoy" pod "f7c9e388-cf4f-4f23-98df-b36b39615609" (UID: "f7c9e388-cf4f-4f23-98df-b36b39615609"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:23.486461 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.486433    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-podinfo" pod "4447e11f-6bed-4da8-97b6-c331cd08a4eb" (UID: "4447e11f-6bed-4da8-97b6-c331cd08a4eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:53:23.486667 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.486622    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-podinfo" pod "f7c9e388-cf4f-4f23-98df-b36b39615609" (UID: "f7c9e388-cf4f-4f23-98df-b36b39615609"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:53:23.486767 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.486709    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-envoy" pod "4447e11f-6bed-4da8-97b6-c331cd08a4eb" (UID: "4447e11f-6bed-4da8-97b6-c331cd08a4eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:23.486767 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.486716    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-token" pod "4447e11f-6bed-4da8-97b6-c331cd08a4eb" (UID: "4447e11f-6bed-4da8-97b6-c331cd08a4eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:53:23.486909 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.486799    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c9e388-cf4f-4f23-98df-b36b39615609-kube-api-access-c72bq" pod "f7c9e388-cf4f-4f23-98df-b36b39615609" (UID: "f7c9e388-cf4f-4f23-98df-b36b39615609"). InnerVolumeSpecName "kube-api-access-c72bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:53:23.487888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.487869    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-token" pod "f7c9e388-cf4f-4f23-98df-b36b39615609" (UID: "f7c9e388-cf4f-4f23-98df-b36b39615609"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:53:23.584845 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.584800    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5556f3c-1664-4f50-8f40-e4a44724c27d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8\" (UID: \"f5556f3c-1664-4f50-8f40-e4a44724c27d\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"
Nov 03 17:53:23.585038 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.584876    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpwrx\" (UniqueName: \"kubernetes.io/projected/f5556f3c-1664-4f50-8f40-e4a44724c27d-kube-api-access-gpwrx\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8\" (UID: \"f5556f3c-1664-4f50-8f40-e4a44724c27d\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"
Nov 03 17:53:23.585038 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.584904    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5556f3c-1664-4f50-8f40-e4a44724c27d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8\" (UID: \"f5556f3c-1664-4f50-8f40-e4a44724c27d\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"
Nov 03 17:53:23.585038 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.584976    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5556f3c-1664-4f50-8f40-e4a44724c27d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8\" (UID: \"f5556f3c-1664-4f50-8f40-e4a44724c27d\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"
Nov 03 17:53:23.585234 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.585110    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c9e388-cf4f-4f23-98df-b36b39615609-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:23.585234 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.585126    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:23.585234 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.585140    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:23.585234 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.585154    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:23.585234 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.585168    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:23.585234 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.585182    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:23.585234 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.585196    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:23.585234 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.585210    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:23.585234 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.585224    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:23.585746 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.585239    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv9x4\" (UniqueName: \"kubernetes.io/projected/4447e11f-6bed-4da8-97b6-c331cd08a4eb-kube-api-access-kv9x4\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:23.585746 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.585254    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:23.585746 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.585268    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:23.585746 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.585282    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c72bq\" (UniqueName: \"kubernetes.io/projected/f7c9e388-cf4f-4f23-98df-b36b39615609-kube-api-access-c72bq\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:23.585746 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.585296    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:23.585746 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.585309    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:23.585746 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.585323    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:23.585746 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.585336    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c9e388-cf4f-4f23-98df-b36b39615609-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:23.585746 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.585351    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4447e11f-6bed-4da8-97b6-c331cd08a4eb-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:23.586342 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.586265    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5556f3c-1664-4f50-8f40-e4a44724c27d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8\" (UID: \"f5556f3c-1664-4f50-8f40-e4a44724c27d\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"
Nov 03 17:53:23.586755 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.586734    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5556f3c-1664-4f50-8f40-e4a44724c27d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8\" (UID: \"f5556f3c-1664-4f50-8f40-e4a44724c27d\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"
Nov 03 17:53:23.589150 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.589128    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5556f3c-1664-4f50-8f40-e4a44724c27d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8\" (UID: \"f5556f3c-1664-4f50-8f40-e4a44724c27d\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"
Nov 03 17:53:23.665572 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.665535    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpwrx\" (UniqueName: \"kubernetes.io/projected/f5556f3c-1664-4f50-8f40-e4a44724c27d-kube-api-access-gpwrx\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8\" (UID: \"f5556f3c-1664-4f50-8f40-e4a44724c27d\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"
Nov 03 17:53:23.714278 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.714238    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"
Nov 03 17:53:23.757083 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.757002    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:23.873506 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.873469    6059 generic.go:356] "Generic (PLEG): container finished" podID="4447e11f-6bed-4da8-97b6-c331cd08a4eb" containerID="984ad4ded1213f3fd605fa12b0d103cca53531b364c829ddb3542f43b433f88f" exitCode=0
Nov 03 17:53:23.873692 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.873674    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx" event={"ID":"4447e11f-6bed-4da8-97b6-c331cd08a4eb","Type":"ContainerDied","Data":"984ad4ded1213f3fd605fa12b0d103cca53531b364c829ddb3542f43b433f88f"}
Nov 03 17:53:23.873766 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.873742    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx" event={"ID":"4447e11f-6bed-4da8-97b6-c331cd08a4eb","Type":"ContainerDied","Data":"a3fa8d5bb87bb5bba01c37bdf8d7c4eb405ead8f0cd19058920e73f74b5a7a12"}
Nov 03 17:53:23.873828 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.873769    6059 scope.go:122] "RemoveContainer" containerID="984ad4ded1213f3fd605fa12b0d103cca53531b364c829ddb3542f43b433f88f"
Nov 03 17:53:23.874091 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.873887    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"
Nov 03 17:53:23.880472 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.880447    6059 generic.go:356] "Generic (PLEG): container finished" podID="f7c9e388-cf4f-4f23-98df-b36b39615609" containerID="6fd130be3e200e333223d29538d46223ffea288e04a2fe3c3c852c1e31c2d1ba" exitCode=0
Nov 03 17:53:23.880584 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.880505    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf" event={"ID":"f7c9e388-cf4f-4f23-98df-b36b39615609","Type":"ContainerDied","Data":"6fd130be3e200e333223d29538d46223ffea288e04a2fe3c3c852c1e31c2d1ba"}
Nov 03 17:53:23.880584 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.880543    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf" event={"ID":"f7c9e388-cf4f-4f23-98df-b36b39615609","Type":"ContainerDied","Data":"5f73f1db7ab92417ce12092b30c4ce6e3cbe1e104b359e930fff353e361a71af"}
Nov 03 17:53:23.880707 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.880615    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"
Nov 03 17:53:23.905898 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.897751    6059 scope.go:122] "RemoveContainer" containerID="984ad4ded1213f3fd605fa12b0d103cca53531b364c829ddb3542f43b433f88f"
Nov 03 17:53:23.905898 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:23.898070    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"984ad4ded1213f3fd605fa12b0d103cca53531b364c829ddb3542f43b433f88f\": container with ID starting with 984ad4ded1213f3fd605fa12b0d103cca53531b364c829ddb3542f43b433f88f not found: ID does not exist" containerID="984ad4ded1213f3fd605fa12b0d103cca53531b364c829ddb3542f43b433f88f"
Nov 03 17:53:23.905898 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.898097    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"984ad4ded1213f3fd605fa12b0d103cca53531b364c829ddb3542f43b433f88f"} err="failed to get container status \"984ad4ded1213f3fd605fa12b0d103cca53531b364c829ddb3542f43b433f88f\": rpc error: code = NotFound desc = could not find container \"984ad4ded1213f3fd605fa12b0d103cca53531b364c829ddb3542f43b433f88f\": container with ID starting with 984ad4ded1213f3fd605fa12b0d103cca53531b364c829ddb3542f43b433f88f not found: ID does not exist"
Nov 03 17:53:23.905898 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.898116    6059 scope.go:122] "RemoveContainer" containerID="6fd130be3e200e333223d29538d46223ffea288e04a2fe3c3c852c1e31c2d1ba"
Nov 03 17:53:23.917888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.914941    6059 patch_prober.go:29] interesting pod/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 17:53:23.917888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.915005    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp" podUID="945d8e0f-a47e-49e3-ae35-d25ad12aca81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 17:53:23.917888 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.916530    6059 scope.go:122] "RemoveContainer" containerID="a487c363356bd9016270163279b78a2674899f8b1cfb595e014595c2b2b37682"
Nov 03 17:53:23.960093 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.960056    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:23.960320 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.960300    6059 scope.go:122] "RemoveContainer" containerID="6fd130be3e200e333223d29538d46223ffea288e04a2fe3c3c852c1e31c2d1ba"
Nov 03 17:53:23.960689 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:23.960664    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fd130be3e200e333223d29538d46223ffea288e04a2fe3c3c852c1e31c2d1ba\": container with ID starting with 6fd130be3e200e333223d29538d46223ffea288e04a2fe3c3c852c1e31c2d1ba not found: ID does not exist" containerID="6fd130be3e200e333223d29538d46223ffea288e04a2fe3c3c852c1e31c2d1ba"
Nov 03 17:53:23.960766 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.960707    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fd130be3e200e333223d29538d46223ffea288e04a2fe3c3c852c1e31c2d1ba"} err="failed to get container status \"6fd130be3e200e333223d29538d46223ffea288e04a2fe3c3c852c1e31c2d1ba\": rpc error: code = NotFound desc = could not find container \"6fd130be3e200e333223d29538d46223ffea288e04a2fe3c3c852c1e31c2d1ba\": container with ID starting with 6fd130be3e200e333223d29538d46223ffea288e04a2fe3c3c852c1e31c2d1ba not found: ID does not exist"
Nov 03 17:53:23.960766 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.960733    6059 scope.go:122] "RemoveContainer" containerID="a487c363356bd9016270163279b78a2674899f8b1cfb595e014595c2b2b37682"
Nov 03 17:53:23.961050 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:23.961032    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a487c363356bd9016270163279b78a2674899f8b1cfb595e014595c2b2b37682\": container with ID starting with a487c363356bd9016270163279b78a2674899f8b1cfb595e014595c2b2b37682 not found: ID does not exist" containerID="a487c363356bd9016270163279b78a2674899f8b1cfb595e014595c2b2b37682"
Nov 03 17:53:23.961100 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:23.961057    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a487c363356bd9016270163279b78a2674899f8b1cfb595e014595c2b2b37682"} err="failed to get container status \"a487c363356bd9016270163279b78a2674899f8b1cfb595e014595c2b2b37682\": rpc error: code = NotFound desc = could not find container \"a487c363356bd9016270163279b78a2674899f8b1cfb595e014595c2b2b37682\": container with ID starting with a487c363356bd9016270163279b78a2674899f8b1cfb595e014595c2b2b37682 not found: ID does not exist"
Nov 03 17:53:24.065073 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:24.064962    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"]
Nov 03 17:53:24.082911 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:24.082880    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-khxpx"]
Nov 03 17:53:24.260636 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:24.260607    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"]
Nov 03 17:53:24.261215 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:53:24.261189    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5556f3c_1664_4f50_8f40_e4a44724c27d.slice/crio-2b7be89694173e7ee1e9823c475e0df48164362a35e5f58d06660cdec6f39649 WatchSource:0}: Error finding container 2b7be89694173e7ee1e9823c475e0df48164362a35e5f58d06660cdec6f39649: Status 404 returned error can't find the container with id 2b7be89694173e7ee1e9823c475e0df48164362a35e5f58d06660cdec6f39649
Nov 03 17:53:24.264390 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:24.264345    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"]
Nov 03 17:53:24.267621 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:24.267595    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-ftlpf"]
Nov 03 17:53:24.302014 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:24.301991    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:24.888921 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:24.888886    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8" event={"ID":"f5556f3c-1664-4f50-8f40-e4a44724c27d","Type":"ContainerStarted","Data":"04661d2f6489a1ef59ad7562c3f9bc683593c88b910a90171ba32fb1883cb705"}
Nov 03 17:53:24.889070 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:24.888938    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8" event={"ID":"f5556f3c-1664-4f50-8f40-e4a44724c27d","Type":"ContainerStarted","Data":"2b7be89694173e7ee1e9823c475e0df48164362a35e5f58d06660cdec6f39649"}
Nov 03 17:53:24.889132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:24.889102    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"
Nov 03 17:53:24.915141 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:24.915114    6059 patch_prober.go:29] interesting pod/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 17:53:24.915291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:24.915160    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp" podUID="945d8e0f-a47e-49e3-ae35-d25ad12aca81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 17:53:25.021833 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:25.021785    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8" podStartSLOduration=2.021771471 podStartE2EDuration="2.021771471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:53:25.020274166 +0000 UTC m=+2145.817986946" watchObservedRunningTime="2026-11-03 17:53:25.021771471 +0000 UTC m=+2145.819484252"
Nov 03 17:53:25.342097 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:25.342062    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4447e11f-6bed-4da8-97b6-c331cd08a4eb" path="/var/lib/kubelet/pods/4447e11f-6bed-4da8-97b6-c331cd08a4eb/volumes"
Nov 03 17:53:25.343045 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:25.343021    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7c9e388-cf4f-4f23-98df-b36b39615609" path="/var/lib/kubelet/pods/f7c9e388-cf4f-4f23-98df-b36b39615609/volumes"
Nov 03 17:53:26.012157 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:26.012118    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:53:26.012502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:26.012467    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:53:26.013296 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:26.013269    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:53:26.420247 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:26.420172    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:26.606126 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:26.606095    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:27.037263 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:27.037221    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:28.477602 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:28.477564    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:30.175977 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:30.175937    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:30.367649 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:30.367617    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:30.612609 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:30.612573    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f"]
Nov 03 17:53:30.613309 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:30.613287    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7c9e388-cf4f-4f23-98df-b36b39615609" containerName="istio-proxy"
Nov 03 17:53:30.616775 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:30.616755    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f"
Nov 03 17:53:30.673619 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:30.673587    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d86c7366-53ba-4c9e-bc59-345abef181f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-759f5627-hl66f\" (UID: \"d86c7366-53ba-4c9e-bc59-345abef181f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f"
Nov 03 17:53:30.673818 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:30.673765    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d86c7366-53ba-4c9e-bc59-345abef181f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-759f5627-hl66f\" (UID: \"d86c7366-53ba-4c9e-bc59-345abef181f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f"
Nov 03 17:53:30.673902 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:30.673850    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d86nk\" (UniqueName: \"kubernetes.io/projected/d86c7366-53ba-4c9e-bc59-345abef181f6-kube-api-access-d86nk\") pod \"maas-api-revoke-sub-759f5627-hl66f\" (UID: \"d86c7366-53ba-4c9e-bc59-345abef181f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f"
Nov 03 17:53:30.723977 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:30.723940    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:53:30.723977 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:30.723964    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 17:53:30.723977 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:30.723983    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:53:30.724253 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:30.724100    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:53:30.727002 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:30.726980    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f"]
Nov 03 17:53:30.775028 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:30.774998    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d86nk\" (UniqueName: \"kubernetes.io/projected/d86c7366-53ba-4c9e-bc59-345abef181f6-kube-api-access-d86nk\") pod \"maas-api-revoke-sub-759f5627-hl66f\" (UID: \"d86c7366-53ba-4c9e-bc59-345abef181f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f"
Nov 03 17:53:30.775200 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:30.775072    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d86c7366-53ba-4c9e-bc59-345abef181f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-759f5627-hl66f\" (UID: \"d86c7366-53ba-4c9e-bc59-345abef181f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f"
Nov 03 17:53:30.775200 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:30.775179    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d86c7366-53ba-4c9e-bc59-345abef181f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-759f5627-hl66f\" (UID: \"d86c7366-53ba-4c9e-bc59-345abef181f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f"
Nov 03 17:53:30.775856 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:30.775833    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d86c7366-53ba-4c9e-bc59-345abef181f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-759f5627-hl66f\" (UID: \"d86c7366-53ba-4c9e-bc59-345abef181f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f"
Nov 03 17:53:30.777510 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:30.777491    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d86c7366-53ba-4c9e-bc59-345abef181f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-759f5627-hl66f\" (UID: \"d86c7366-53ba-4c9e-bc59-345abef181f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f"
Nov 03 17:53:30.838156 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:30.838122    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d86nk\" (UniqueName: \"kubernetes.io/projected/d86c7366-53ba-4c9e-bc59-345abef181f6-kube-api-access-d86nk\") pod \"maas-api-revoke-sub-759f5627-hl66f\" (UID: \"d86c7366-53ba-4c9e-bc59-345abef181f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f"
Nov 03 17:53:30.906724 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:30.906633    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:30.929509 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:30.929468    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f"
Nov 03 17:53:31.052146 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:31.052097    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53d5534c-knmgq"]
Nov 03 17:53:31.055256 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:31.055225    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53d5534c-knmgq"]
Nov 03 17:53:31.232324 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:31.232289    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:31.310507 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:31.310483    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f"]
Nov 03 17:53:31.312473 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:53:31.312448    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86c7366_53ba_4c9e_bc59_345abef181f6.slice/crio-6300b9f84284b2db5a2f4cca6860c6dc810b5872672888f18e301f2414f6bea2 WatchSource:0}: Error finding container 6300b9f84284b2db5a2f4cca6860c6dc810b5872672888f18e301f2414f6bea2: Status 404 returned error can't find the container with id 6300b9f84284b2db5a2f4cca6860c6dc810b5872672888f18e301f2414f6bea2
Nov 03 17:53:31.336677 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:31.336653    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a770d2-0510-48b2-83d0-244d9f3d2816" path="/var/lib/kubelet/pods/11a770d2-0510-48b2-83d0-244d9f3d2816/volumes"
Nov 03 17:53:31.403685 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:31.403646    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:31.940900 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:31.940813    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f" event={"ID":"d86c7366-53ba-4c9e-bc59-345abef181f6","Type":"ContainerStarted","Data":"77bc15dd72351409a52172bf1b73326d5d3c2fccf538d651c19181493dc475de"}
Nov 03 17:53:31.941053 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:31.940900    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f" event={"ID":"d86c7366-53ba-4c9e-bc59-345abef181f6","Type":"ContainerStarted","Data":"6300b9f84284b2db5a2f4cca6860c6dc810b5872672888f18e301f2414f6bea2"}
Nov 03 17:53:32.065882 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:32.065828    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f" podStartSLOduration=2.065812788 podStartE2EDuration="2.065812788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:53:32.062968066 +0000 UTC m=+2152.860680840" watchObservedRunningTime="2026-11-03 17:53:32.065812788 +0000 UTC m=+2152.863525623"
Nov 03 17:53:32.950920 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:32.950880    6059 generic.go:356] "Generic (PLEG): container finished" podID="d86c7366-53ba-4c9e-bc59-345abef181f6" containerID="77bc15dd72351409a52172bf1b73326d5d3c2fccf538d651c19181493dc475de" exitCode=0
Nov 03 17:53:32.951296 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:32.950966    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f" event={"ID":"d86c7366-53ba-4c9e-bc59-345abef181f6","Type":"ContainerDied","Data":"77bc15dd72351409a52172bf1b73326d5d3c2fccf538d651c19181493dc475de"}
Nov 03 17:53:33.263760 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:33.263711    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:33.764074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:33.764039    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:33.963943 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:33.963910    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:34.114010 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:34.113982    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f"
Nov 03 17:53:34.212744 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:34.212711    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d86c7366-53ba-4c9e-bc59-345abef181f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d86c7366-53ba-4c9e-bc59-345abef181f6-maas-api-xks-ca\") pod \"d86c7366-53ba-4c9e-bc59-345abef181f6\" (UID: \"d86c7366-53ba-4c9e-bc59-345abef181f6\") "
Nov 03 17:53:34.212918 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:34.212759    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d86c7366-53ba-4c9e-bc59-345abef181f6-kube-api-access-d86nk\" (UniqueName: \"kubernetes.io/projected/d86c7366-53ba-4c9e-bc59-345abef181f6-kube-api-access-d86nk\") pod \"d86c7366-53ba-4c9e-bc59-345abef181f6\" (UID: \"d86c7366-53ba-4c9e-bc59-345abef181f6\") "
Nov 03 17:53:34.212918 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:34.212878    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d86c7366-53ba-4c9e-bc59-345abef181f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d86c7366-53ba-4c9e-bc59-345abef181f6-maas-api-openshift-ca\") pod \"d86c7366-53ba-4c9e-bc59-345abef181f6\" (UID: \"d86c7366-53ba-4c9e-bc59-345abef181f6\") "
Nov 03 17:53:34.213276 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:34.213254    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86c7366-53ba-4c9e-bc59-345abef181f6-maas-api-openshift-ca" pod "d86c7366-53ba-4c9e-bc59-345abef181f6" (UID: "d86c7366-53ba-4c9e-bc59-345abef181f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:53:34.214814 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:34.214790    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d86c7366-53ba-4c9e-bc59-345abef181f6-maas-api-xks-ca" pod "d86c7366-53ba-4c9e-bc59-345abef181f6" (UID: "d86c7366-53ba-4c9e-bc59-345abef181f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:53:34.214908 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:34.214823    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86c7366-53ba-4c9e-bc59-345abef181f6-kube-api-access-d86nk" pod "d86c7366-53ba-4c9e-bc59-345abef181f6" (UID: "d86c7366-53ba-4c9e-bc59-345abef181f6"). InnerVolumeSpecName "kube-api-access-d86nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:53:34.314607 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:34.314570    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d86c7366-53ba-4c9e-bc59-345abef181f6-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:34.314607 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:34.314605    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d86nk\" (UniqueName: \"kubernetes.io/projected/d86c7366-53ba-4c9e-bc59-345abef181f6-kube-api-access-d86nk\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:34.314801 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:34.314616    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d86c7366-53ba-4c9e-bc59-345abef181f6-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:34.455503 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:34.448923    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:34.635839 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:34.635808    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:34.970417 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:34.970388    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f"
Nov 03 17:53:34.970947 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:34.970407    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f" event={"ID":"d86c7366-53ba-4c9e-bc59-345abef181f6","Type":"ContainerDied","Data":"6300b9f84284b2db5a2f4cca6860c6dc810b5872672888f18e301f2414f6bea2"}
Nov 03 17:53:34.971050 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:34.970983    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6300b9f84284b2db5a2f4cca6860c6dc810b5872672888f18e301f2414f6bea2"
Nov 03 17:53:35.682961 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:35.682913    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:35.900823 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:35.900786    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8" podUID="f5556f3c-1664-4f50-8f40-e4a44724c27d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 17:53:35.924515 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:35.924482    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:36.914900 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:36.914832    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:53:37.061934 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.061900    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb"]
Nov 03 17:53:37.116378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.116305    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-095a6192-kc4xb"]
Nov 03 17:53:37.204526 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.204434    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n" podUID="d6c1263b-e648-4f10-af7d-86397d6b207a" containerName="main" containerID="cri-o://a656aff129f93a9d7d50f083585d75cad313c85465b5153c0707c840968a35e6" gracePeriod=45
Nov 03 17:53:37.210869 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.210833    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv" podUID="7942e9d1-597f-47a6-9c70-1d174fe4c8e6" containerName="main" containerID="cri-o://2ed2e3f2640be105c953235c167382db1b0507b58d2313e7e5555999e1c2f16d" gracePeriod=45
Nov 03 17:53:37.343792 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.343122    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f639120d-e1f4-4700-a328-fec7ee568f3c" path="/var/lib/kubelet/pods/f639120d-e1f4-4700-a328-fec7ee568f3c/volumes"
Nov 03 17:53:37.389737 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.389682    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv" podUID="7942e9d1-597f-47a6-9c70-1d174fe4c8e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": dial tcp 10.133.0.80:8000: connect: connection refused"
Nov 03 17:53:37.593572 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.593533    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:53:37.615092 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.615068    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:53:37.680323 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.680293    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-home\" (UniqueName: \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-home\") pod \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") "
Nov 03 17:53:37.680521 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.680342    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-dshm\") pod \"d6c1263b-e648-4f10-af7d-86397d6b207a\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") "
Nov 03 17:53:37.680521 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.680402    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-tmp-dir\") pod \"d6c1263b-e648-4f10-af7d-86397d6b207a\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") "
Nov 03 17:53:37.680521 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.680424    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-kube-api-access-xvwkq\" (UniqueName: \"kubernetes.io/projected/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-kube-api-access-xvwkq\") pod \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") "
Nov 03 17:53:37.680521 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.680457    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-model-cache\") pod \"d6c1263b-e648-4f10-af7d-86397d6b207a\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") "
Nov 03 17:53:37.680521 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.680495    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-dshm\") pod \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") "
Nov 03 17:53:37.680819 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.680597    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-tmp-dir\") pod \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") "
Nov 03 17:53:37.680819 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.680644    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-model-cache\") pod \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") "
Nov 03 17:53:37.680819 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.680647    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-home" pod "7942e9d1-597f-47a6-9c70-1d174fe4c8e6" (UID: "7942e9d1-597f-47a6-9c70-1d174fe4c8e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:37.680819 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.680685    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-home\" (UniqueName: \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-home\") pod \"d6c1263b-e648-4f10-af7d-86397d6b207a\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") "
Nov 03 17:53:37.680819 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.680719    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-tmp-dir" pod "d6c1263b-e648-4f10-af7d-86397d6b207a" (UID: "d6c1263b-e648-4f10-af7d-86397d6b207a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:37.680819 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.680814    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-tls-certs\" (UniqueName: \"kubernetes.io/secret/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-tls-certs\") pod \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\" (UID: \"7942e9d1-597f-47a6-9c70-1d174fe4c8e6\") "
Nov 03 17:53:37.681149 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.680802    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-model-cache" pod "d6c1263b-e648-4f10-af7d-86397d6b207a" (UID: "d6c1263b-e648-4f10-af7d-86397d6b207a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:37.681149 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.680867    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-home" pod "d6c1263b-e648-4f10-af7d-86397d6b207a" (UID: "d6c1263b-e648-4f10-af7d-86397d6b207a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:37.681149 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.680912    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6c1263b-e648-4f10-af7d-86397d6b207a-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c1263b-e648-4f10-af7d-86397d6b207a-tls-certs\") pod \"d6c1263b-e648-4f10-af7d-86397d6b207a\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") "
Nov 03 17:53:37.681149 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.680939    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-tmp-dir" pod "7942e9d1-597f-47a6-9c70-1d174fe4c8e6" (UID: "7942e9d1-597f-47a6-9c70-1d174fe4c8e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:37.681149 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.680987    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c1263b-e648-4f10-af7d-86397d6b207a-kube-api-access-kmwrl\" (UniqueName: \"kubernetes.io/projected/d6c1263b-e648-4f10-af7d-86397d6b207a-kube-api-access-kmwrl\") pod \"d6c1263b-e648-4f10-af7d-86397d6b207a\" (UID: \"d6c1263b-e648-4f10-af7d-86397d6b207a\") "
Nov 03 17:53:37.681149 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.680991    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-model-cache" pod "7942e9d1-597f-47a6-9c70-1d174fe4c8e6" (UID: "7942e9d1-597f-47a6-9c70-1d174fe4c8e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:37.681711 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.681681    6059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-home\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:37.681797 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.681715    6059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-tmp-dir\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:37.681797 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.681732    6059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-model-cache\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:37.681797 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.681750    6059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-model-cache\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:37.681797 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.681765    6059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-home\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:37.683140 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.683114    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-dshm" pod "d6c1263b-e648-4f10-af7d-86397d6b207a" (UID: "d6c1263b-e648-4f10-af7d-86397d6b207a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:37.683291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.683268    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-tls-certs" pod "7942e9d1-597f-47a6-9c70-1d174fe4c8e6" (UID: "7942e9d1-597f-47a6-9c70-1d174fe4c8e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:53:37.683500 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.683483    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-dshm" pod "7942e9d1-597f-47a6-9c70-1d174fe4c8e6" (UID: "7942e9d1-597f-47a6-9c70-1d174fe4c8e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:37.683748 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.683731    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c1263b-e648-4f10-af7d-86397d6b207a-tls-certs" pod "d6c1263b-e648-4f10-af7d-86397d6b207a" (UID: "d6c1263b-e648-4f10-af7d-86397d6b207a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:53:37.684017 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.683994    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-kube-api-access-xvwkq" pod "7942e9d1-597f-47a6-9c70-1d174fe4c8e6" (UID: "7942e9d1-597f-47a6-9c70-1d174fe4c8e6"). InnerVolumeSpecName "kube-api-access-xvwkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:53:37.684078 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.684010    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c1263b-e648-4f10-af7d-86397d6b207a-kube-api-access-kmwrl" pod "d6c1263b-e648-4f10-af7d-86397d6b207a" (UID: "d6c1263b-e648-4f10-af7d-86397d6b207a"). InnerVolumeSpecName "kube-api-access-kmwrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:53:37.758616 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.758574    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:37.783114 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.783021    6059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-tmp-dir\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:37.783114 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.783062    6059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-tls-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:37.783114 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.783078    6059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c1263b-e648-4f10-af7d-86397d6b207a-tls-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:37.783114 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.783095    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmwrl\" (UniqueName: \"kubernetes.io/projected/d6c1263b-e648-4f10-af7d-86397d6b207a-kube-api-access-kmwrl\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:37.783114 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.783109    6059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c1263b-e648-4f10-af7d-86397d6b207a-dshm\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:37.783533 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.783123    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvwkq\" (UniqueName: \"kubernetes.io/projected/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-kube-api-access-xvwkq\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:37.783533 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.783136    6059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7942e9d1-597f-47a6-9c70-1d174fe4c8e6-dshm\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:37.993876 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.993835    6059 generic.go:356] "Generic (PLEG): container finished" podID="7942e9d1-597f-47a6-9c70-1d174fe4c8e6" containerID="2ed2e3f2640be105c953235c167382db1b0507b58d2313e7e5555999e1c2f16d" exitCode=0
Nov 03 17:53:37.994327 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.993899    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"
Nov 03 17:53:37.994327 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.993910    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv" event={"ID":"7942e9d1-597f-47a6-9c70-1d174fe4c8e6","Type":"ContainerDied","Data":"2ed2e3f2640be105c953235c167382db1b0507b58d2313e7e5555999e1c2f16d"}
Nov 03 17:53:37.994327 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.993961    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv" event={"ID":"7942e9d1-597f-47a6-9c70-1d174fe4c8e6","Type":"ContainerDied","Data":"d9718b309cee6579f5de0f11e025d668f9a8877027c900b76a8b7ca3b2902a52"}
Nov 03 17:53:37.994327 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.993988    6059 scope.go:122] "RemoveContainer" containerID="2ed2e3f2640be105c953235c167382db1b0507b58d2313e7e5555999e1c2f16d"
Nov 03 17:53:37.996310 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.996287    6059 generic.go:356] "Generic (PLEG): container finished" podID="d6c1263b-e648-4f10-af7d-86397d6b207a" containerID="a656aff129f93a9d7d50f083585d75cad313c85465b5153c0707c840968a35e6" exitCode=0
Nov 03 17:53:37.996484 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.996315    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n" event={"ID":"d6c1263b-e648-4f10-af7d-86397d6b207a","Type":"ContainerDied","Data":"a656aff129f93a9d7d50f083585d75cad313c85465b5153c0707c840968a35e6"}
Nov 03 17:53:37.996632 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.996499    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n" event={"ID":"d6c1263b-e648-4f10-af7d-86397d6b207a","Type":"ContainerDied","Data":"90582841d9011b5798b2d29a43f8241444f63895004383b6f3f4478bdb4c9a5b"}
Nov 03 17:53:37.996632 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:37.996414    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"
Nov 03 17:53:38.008436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:38.008419    6059 scope.go:122] "RemoveContainer" containerID="2ed2e3f2640be105c953235c167382db1b0507b58d2313e7e5555999e1c2f16d"
Nov 03 17:53:38.008734 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:38.008717    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ed2e3f2640be105c953235c167382db1b0507b58d2313e7e5555999e1c2f16d\": container with ID starting with 2ed2e3f2640be105c953235c167382db1b0507b58d2313e7e5555999e1c2f16d not found: ID does not exist" containerID="2ed2e3f2640be105c953235c167382db1b0507b58d2313e7e5555999e1c2f16d"
Nov 03 17:53:38.008819 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:38.008747    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed2e3f2640be105c953235c167382db1b0507b58d2313e7e5555999e1c2f16d"} err="failed to get container status \"2ed2e3f2640be105c953235c167382db1b0507b58d2313e7e5555999e1c2f16d\": rpc error: code = NotFound desc = could not find container \"2ed2e3f2640be105c953235c167382db1b0507b58d2313e7e5555999e1c2f16d\": container with ID starting with 2ed2e3f2640be105c953235c167382db1b0507b58d2313e7e5555999e1c2f16d not found: ID does not exist"
Nov 03 17:53:38.008819 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:38.008769    6059 scope.go:122] "RemoveContainer" containerID="a656aff129f93a9d7d50f083585d75cad313c85465b5153c0707c840968a35e6"
Nov 03 17:53:38.019417 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:38.019400    6059 scope.go:122] "RemoveContainer" containerID="a656aff129f93a9d7d50f083585d75cad313c85465b5153c0707c840968a35e6"
Nov 03 17:53:38.019732 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:38.019714    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a656aff129f93a9d7d50f083585d75cad313c85465b5153c0707c840968a35e6\": container with ID starting with a656aff129f93a9d7d50f083585d75cad313c85465b5153c0707c840968a35e6 not found: ID does not exist" containerID="a656aff129f93a9d7d50f083585d75cad313c85465b5153c0707c840968a35e6"
Nov 03 17:53:38.019807 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:38.019742    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a656aff129f93a9d7d50f083585d75cad313c85465b5153c0707c840968a35e6"} err="failed to get container status \"a656aff129f93a9d7d50f083585d75cad313c85465b5153c0707c840968a35e6\": rpc error: code = NotFound desc = could not find container \"a656aff129f93a9d7d50f083585d75cad313c85465b5153c0707c840968a35e6\": container with ID starting with a656aff129f93a9d7d50f083585d75cad313c85465b5153c0707c840968a35e6 not found: ID does not exist"
Nov 03 17:53:38.099050 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:38.098970    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"]
Nov 03 17:53:38.100341 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:38.100308    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8" podUID="f5556f3c-1664-4f50-8f40-e4a44724c27d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 17:53:38.102425 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:38.102398    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:38.174945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:38.174910    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"]
Nov 03 17:53:38.187651 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:38.187622    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/distinct-2-w3-2355f4-kserve-59958c9658-vtz5n"]
Nov 03 17:53:38.299901 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:38.299860    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"]
Nov 03 17:53:38.305635 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:38.305607    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/distinct-w3-2355f4-kserve-6c8756668f-n6zcv"]
Nov 03 17:53:38.536731 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:38.536694    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:39.339277 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:39.339235    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7942e9d1-597f-47a6-9c70-1d174fe4c8e6" path="/var/lib/kubelet/pods/7942e9d1-597f-47a6-9c70-1d174fe4c8e6/volumes"
Nov 03 17:53:39.339790 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:39.339773    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c1263b-e648-4f10-af7d-86397d6b207a" path="/var/lib/kubelet/pods/d6c1263b-e648-4f10-af7d-86397d6b207a/volumes"
Nov 03 17:53:40.740850 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:40.740819    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:42.341742 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:42.341715    6059 scope.go:122] "RemoveContainer" containerID="0633096f2f9e390380a0657ee17353f74254690fb1276a89bb048feb518a3473"
Nov 03 17:53:42.617929 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:42.617851    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:42.947829 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:42.947743    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:43.100255 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:43.100211    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8" podUID="f5556f3c-1664-4f50-8f40-e4a44724c27d" containerName="payload-processing" containerID="cri-o://04661d2f6489a1ef59ad7562c3f9bc683593c88b910a90171ba32fb1883cb705" gracePeriod=25
Nov 03 17:53:43.353062 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:43.353038    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"
Nov 03 17:53:43.448815 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:43.448781    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5556f3c-1664-4f50-8f40-e4a44724c27d-kube-api-access-gpwrx\" (UniqueName: \"kubernetes.io/projected/f5556f3c-1664-4f50-8f40-e4a44724c27d-kube-api-access-gpwrx\") pod \"f5556f3c-1664-4f50-8f40-e4a44724c27d\" (UID: \"f5556f3c-1664-4f50-8f40-e4a44724c27d\") "
Nov 03 17:53:43.448970 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:43.448854    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5556f3c-1664-4f50-8f40-e4a44724c27d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5556f3c-1664-4f50-8f40-e4a44724c27d-plugins-config-volume\") pod \"f5556f3c-1664-4f50-8f40-e4a44724c27d\" (UID: \"f5556f3c-1664-4f50-8f40-e4a44724c27d\") "
Nov 03 17:53:43.449034 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:43.449007    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5556f3c-1664-4f50-8f40-e4a44724c27d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5556f3c-1664-4f50-8f40-e4a44724c27d-routing-overlay-volume\") pod \"f5556f3c-1664-4f50-8f40-e4a44724c27d\" (UID: \"f5556f3c-1664-4f50-8f40-e4a44724c27d\") "
Nov 03 17:53:43.449082 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:43.449045    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5556f3c-1664-4f50-8f40-e4a44724c27d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5556f3c-1664-4f50-8f40-e4a44724c27d-provider-credentials\") pod \"f5556f3c-1664-4f50-8f40-e4a44724c27d\" (UID: \"f5556f3c-1664-4f50-8f40-e4a44724c27d\") "
Nov 03 17:53:43.449285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:43.449259    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5556f3c-1664-4f50-8f40-e4a44724c27d-plugins-config-volume" pod "f5556f3c-1664-4f50-8f40-e4a44724c27d" (UID: "f5556f3c-1664-4f50-8f40-e4a44724c27d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:53:43.449285 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:43.449275    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5556f3c-1664-4f50-8f40-e4a44724c27d-routing-overlay-volume" pod "f5556f3c-1664-4f50-8f40-e4a44724c27d" (UID: "f5556f3c-1664-4f50-8f40-e4a44724c27d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:53:43.449555 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:43.449535    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5556f3c-1664-4f50-8f40-e4a44724c27d-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:43.449657 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:43.449561    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5556f3c-1664-4f50-8f40-e4a44724c27d-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:43.450996 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:43.450970    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5556f3c-1664-4f50-8f40-e4a44724c27d-kube-api-access-gpwrx" pod "f5556f3c-1664-4f50-8f40-e4a44724c27d" (UID: "f5556f3c-1664-4f50-8f40-e4a44724c27d"). InnerVolumeSpecName "kube-api-access-gpwrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:53:43.451114 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:43.451080    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5556f3c-1664-4f50-8f40-e4a44724c27d-provider-credentials" pod "f5556f3c-1664-4f50-8f40-e4a44724c27d" (UID: "f5556f3c-1664-4f50-8f40-e4a44724c27d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:53:43.551150 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:43.551069    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5556f3c-1664-4f50-8f40-e4a44724c27d-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:43.551150 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:43.551102    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpwrx\" (UniqueName: \"kubernetes.io/projected/f5556f3c-1664-4f50-8f40-e4a44724c27d-kube-api-access-gpwrx\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:44.046876 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:44.046840    6059 generic.go:356] "Generic (PLEG): container finished" podID="f5556f3c-1664-4f50-8f40-e4a44724c27d" containerID="04661d2f6489a1ef59ad7562c3f9bc683593c88b910a90171ba32fb1883cb705" exitCode=0
Nov 03 17:53:44.047051 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:44.046892    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"
Nov 03 17:53:44.047051 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:44.046898    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8" event={"ID":"f5556f3c-1664-4f50-8f40-e4a44724c27d","Type":"ContainerDied","Data":"04661d2f6489a1ef59ad7562c3f9bc683593c88b910a90171ba32fb1883cb705"}
Nov 03 17:53:44.047051 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:44.046988    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8" event={"ID":"f5556f3c-1664-4f50-8f40-e4a44724c27d","Type":"ContainerDied","Data":"2b7be89694173e7ee1e9823c475e0df48164362a35e5f58d06660cdec6f39649"}
Nov 03 17:53:44.047051 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:44.047010    6059 scope.go:122] "RemoveContainer" containerID="04661d2f6489a1ef59ad7562c3f9bc683593c88b910a90171ba32fb1883cb705"
Nov 03 17:53:44.057663 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:44.057644    6059 scope.go:122] "RemoveContainer" containerID="04661d2f6489a1ef59ad7562c3f9bc683593c88b910a90171ba32fb1883cb705"
Nov 03 17:53:44.057922 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:44.057901    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04661d2f6489a1ef59ad7562c3f9bc683593c88b910a90171ba32fb1883cb705\": container with ID starting with 04661d2f6489a1ef59ad7562c3f9bc683593c88b910a90171ba32fb1883cb705 not found: ID does not exist" containerID="04661d2f6489a1ef59ad7562c3f9bc683593c88b910a90171ba32fb1883cb705"
Nov 03 17:53:44.057983 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:44.057930    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04661d2f6489a1ef59ad7562c3f9bc683593c88b910a90171ba32fb1883cb705"} err="failed to get container status \"04661d2f6489a1ef59ad7562c3f9bc683593c88b910a90171ba32fb1883cb705\": rpc error: code = NotFound desc = could not find container \"04661d2f6489a1ef59ad7562c3f9bc683593c88b910a90171ba32fb1883cb705\": container with ID starting with 04661d2f6489a1ef59ad7562c3f9bc683593c88b910a90171ba32fb1883cb705 not found: ID does not exist"
Nov 03 17:53:44.224633 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:44.224596    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"]
Nov 03 17:53:44.275898 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:44.275869    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b885d6h8"]
Nov 03 17:53:44.317407 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:44.317267    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:44.521841 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:44.521793    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:44.825761 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:44.825733    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:45.336794 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:45.336760    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5556f3c-1664-4f50-8f40-e4a44724c27d" path="/var/lib/kubelet/pods/f5556f3c-1664-4f50-8f40-e4a44724c27d/volumes"
Nov 03 17:53:45.743606 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:45.743573    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:45.929262 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:45.929227    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:46.055020 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:46.054920    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5"]
Nov 03 17:53:46.103144 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:46.103111    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7fac61c-8f5s5"]
Nov 03 17:53:47.336761 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:47.336686    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09273357-8630-4ae8-a45e-d90373bcb506" path="/var/lib/kubelet/pods/09273357-8630-4ae8-a45e-d90373bcb506/volumes"
Nov 03 17:53:47.501107 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:47.501077    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-d27edff159f1: secret "agc-cred-d27edff159f1" not found
Nov 03 17:53:47.501107 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:47.501106    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-ec4400c42669: secret "agc-cred-ec4400c42669" not found
Nov 03 17:53:47.501107 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:47.501119    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv: [secret "agc-cred-d27edff159f1" not found, secret "agc-cred-ec4400c42669" not found]
Nov 03 17:53:47.501477 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:47.501186    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-provider-credentials podName:3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1 nodeName:}" failed. No retries permitted until 2026-11-03 17:53:48.001166315 +0000 UTC m=+2168.798879073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv" (UID: "3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1") : [secret "agc-cred-d27edff159f1" not found, secret "agc-cred-ec4400c42669" not found]
Nov 03 17:53:47.738967 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:47.738903    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:48.007518 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:48.007447    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-d27edff159f1: secret "agc-cred-d27edff159f1" not found
Nov 03 17:53:48.007518 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:48.007474    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-ec4400c42669: secret "agc-cred-ec4400c42669" not found
Nov 03 17:53:48.007518 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:48.007483    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv: [secret "agc-cred-d27edff159f1" not found, secret "agc-cred-ec4400c42669" not found]
Nov 03 17:53:48.007755 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:48.007549    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-provider-credentials podName:3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1 nodeName:}" failed. No retries permitted until 2026-11-03 17:53:49.007534684 +0000 UTC m=+2169.805247444 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv" (UID: "3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1") : [secret "agc-cred-d27edff159f1" not found, secret "agc-cred-ec4400c42669" not found]
Nov 03 17:53:48.050253 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:48.050223    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8"]
Nov 03 17:53:48.101551 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:48.101519    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c800894-kzjg8"]
Nov 03 17:53:48.878277 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:48.878244    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"]
Nov 03 17:53:48.878815 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:48.878800    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f5556f3c-1664-4f50-8f40-e4a44724c27d" containerName="payload-processing"
Nov 03 17:53:48.878862 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:48.878816    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5556f3c-1664-4f50-8f40-e4a44724c27d" containerName="payload-processing"
Nov 03 17:53:48.878862 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:48.878839    6059 cpu_manager.go:414] "containerMap: removing container" podUID="d86c7366-53ba-4c9e-bc59-345abef181f6" containerName="revoke-keys"
Nov 03 17:53:48.878862 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:48.878844    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d86c7366-53ba-4c9e-bc59-345abef181f6" containerName="revoke-keys"
Nov 03 17:53:48.878962 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:48.878868    6059 cpu_manager.go:414] "containerMap: removing container" podUID="d6c1263b-e648-4f10-af7d-86397d6b207a" containerName="main"
Nov 03 17:53:48.878962 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:48.878873    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c1263b-e648-4f10-af7d-86397d6b207a" containerName="main"
Nov 03 17:53:48.878962 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:48.878885    6059 cpu_manager.go:414] "containerMap: removing container" podUID="7942e9d1-597f-47a6-9c70-1d174fe4c8e6" containerName="main"
Nov 03 17:53:48.878962 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:48.878891    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7942e9d1-597f-47a6-9c70-1d174fe4c8e6" containerName="main"
Nov 03 17:53:48.879090 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:48.878964    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c1263b-e648-4f10-af7d-86397d6b207a" containerName="main"
Nov 03 17:53:48.879090 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:48.878974    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7942e9d1-597f-47a6-9c70-1d174fe4c8e6" containerName="main"
Nov 03 17:53:48.879090 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:48.878983    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5556f3c-1664-4f50-8f40-e4a44724c27d" containerName="payload-processing"
Nov 03 17:53:48.879090 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:48.878991    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d86c7366-53ba-4c9e-bc59-345abef181f6" containerName="revoke-keys"
Nov 03 17:53:48.884020 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:48.883999    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"
Nov 03 17:53:48.938225 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:48.938206    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-8758ca05391f\""
Nov 03 17:53:48.992992 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:48.992958    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"]
Nov 03 17:53:49.020724 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:49.020695    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5a452c7-51f0-4c5a-b092-d0507af37bb7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg\" (UID: \"f5a452c7-51f0-4c5a-b092-d0507af37bb7\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"
Nov 03 17:53:49.020903 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:49.020739    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5a452c7-51f0-4c5a-b092-d0507af37bb7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg\" (UID: \"f5a452c7-51f0-4c5a-b092-d0507af37bb7\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"
Nov 03 17:53:49.020903 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:49.020840    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72mhl\" (UniqueName: \"kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-kube-api-access-72mhl\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg\" (UID: \"f5a452c7-51f0-4c5a-b092-d0507af37bb7\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"
Nov 03 17:53:49.020903 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:49.020873    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg\" (UID: \"f5a452c7-51f0-4c5a-b092-d0507af37bb7\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"
Nov 03 17:53:49.021083 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:49.021070    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-d27edff159f1: secret "agc-cred-d27edff159f1" not found
Nov 03 17:53:49.021124 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:49.021088    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-ec4400c42669: secret "agc-cred-ec4400c42669" not found
Nov 03 17:53:49.021124 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:49.021098    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv: [secret "agc-cred-d27edff159f1" not found, secret "agc-cred-ec4400c42669" not found]
Nov 03 17:53:49.021189 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:49.021142    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-provider-credentials podName:3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1 nodeName:}" failed. No retries permitted until 2026-11-03 17:53:51.021128905 +0000 UTC m=+2171.818841667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv" (UID: "3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1") : [secret "agc-cred-d27edff159f1" not found, secret "agc-cred-ec4400c42669" not found]
Nov 03 17:53:49.122002 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:49.121968    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5a452c7-51f0-4c5a-b092-d0507af37bb7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg\" (UID: \"f5a452c7-51f0-4c5a-b092-d0507af37bb7\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"
Nov 03 17:53:49.122213 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:49.122029    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5a452c7-51f0-4c5a-b092-d0507af37bb7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg\" (UID: \"f5a452c7-51f0-4c5a-b092-d0507af37bb7\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"
Nov 03 17:53:49.122213 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:49.122082    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72mhl\" (UniqueName: \"kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-kube-api-access-72mhl\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg\" (UID: \"f5a452c7-51f0-4c5a-b092-d0507af37bb7\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"
Nov 03 17:53:49.122213 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:49.122111    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg\" (UID: \"f5a452c7-51f0-4c5a-b092-d0507af37bb7\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"
Nov 03 17:53:49.122766 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:49.122737    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5a452c7-51f0-4c5a-b092-d0507af37bb7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg\" (UID: \"f5a452c7-51f0-4c5a-b092-d0507af37bb7\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"
Nov 03 17:53:49.122865 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:49.122776    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5a452c7-51f0-4c5a-b092-d0507af37bb7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg\" (UID: \"f5a452c7-51f0-4c5a-b092-d0507af37bb7\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"
Nov 03 17:53:49.124963 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:49.124940    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg\" (UID: \"f5a452c7-51f0-4c5a-b092-d0507af37bb7\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"
Nov 03 17:53:49.185817 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:49.185753    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72mhl\" (UniqueName: \"kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-kube-api-access-72mhl\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg\" (UID: \"f5a452c7-51f0-4c5a-b092-d0507af37bb7\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"
Nov 03 17:53:49.197619 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:49.197597    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"
Nov 03 17:53:49.337832 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:49.337800    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd" path="/var/lib/kubelet/pods/1594d9c1-d6b7-44b0-9ef6-d87fd83e36dd/volumes"
Nov 03 17:53:49.501459 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:49.501423    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"]
Nov 03 17:53:49.501664 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:49.501639    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp" podUID="a04c0ae8-7886-46c1-98c1-b2053479c608" containerName="istio-proxy" containerID="cri-o://9e9e14cac618766d4e04ff853ddba7be78f2cf83cee586db43a4998baac4975e" gracePeriod=30
Nov 03 17:53:49.505563 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:49.505541    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"]
Nov 03 17:53:49.505713 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:49.505695    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn" podUID="4983714f-6635-4557-9e25-80cf0c914d9d" containerName="istio-proxy" containerID="cri-o://96f71e58b485513665faa6f2e6feb84f72ca5c305dc7592ecd68f1d5b8998e43" gracePeriod=30
Nov 03 17:53:49.625584 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:49.625544    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"]
Nov 03 17:53:49.628808 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:53:49.628767    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a452c7_51f0_4c5a_b092_d0507af37bb7.slice/crio-7d554549ef510f14085f2c581895fb2edabded43ab7492cb7415b824c17c2c01 WatchSource:0}: Error finding container 7d554549ef510f14085f2c581895fb2edabded43ab7492cb7415b824c17c2c01: Status 404 returned error can't find the container with id 7d554549ef510f14085f2c581895fb2edabded43ab7492cb7415b824c17c2c01
Nov 03 17:53:50.091677 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:50.091635    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg" event={"ID":"f5a452c7-51f0-4c5a-b092-d0507af37bb7","Type":"ContainerStarted","Data":"d68f73640f2b68e1e63a336489b25df5b0b839b2a2019ed26cbb6c08d0765183"}
Nov 03 17:53:50.092096 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:50.091694    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg" event={"ID":"f5a452c7-51f0-4c5a-b092-d0507af37bb7","Type":"ContainerStarted","Data":"7d554549ef510f14085f2c581895fb2edabded43ab7492cb7415b824c17c2c01"}
Nov 03 17:53:50.092096 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:50.091839    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"
Nov 03 17:53:50.214657 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:50.214609    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg" podStartSLOduration=2.214596227 podStartE2EDuration="2.214596227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:53:50.212588768 +0000 UTC m=+2171.010301550" watchObservedRunningTime="2026-11-03 17:53:50.214596227 +0000 UTC m=+2171.012309007"
Nov 03 17:53:51.044704 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:51.044674    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-d27edff159f1: secret "agc-cred-d27edff159f1" not found
Nov 03 17:53:51.044704 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:51.044702    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-ec4400c42669: secret "agc-cred-ec4400c42669" not found
Nov 03 17:53:51.044704 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:51.044712    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv: [secret "agc-cred-d27edff159f1" not found, secret "agc-cred-ec4400c42669" not found]
Nov 03 17:53:51.044979 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:51.044772    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-provider-credentials podName:3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1 nodeName:}" failed. No retries permitted until 2026-11-03 17:53:55.044758031 +0000 UTC m=+2175.842470802 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv" (UID: "3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1") : [secret "agc-cred-d27edff159f1" not found, secret "agc-cred-ec4400c42669" not found]
Nov 03 17:53:52.488761 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:52.488714    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:54.292492 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.292455    6059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:53:54.292875 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.292514    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp" podUID="a04c0ae8-7886-46c1-98c1-b2053479c608" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:53:54.510417 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.510381    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"]
Nov 03 17:53:54.510627 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.510601    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w" podUID="029774c2-7278-4d5b-ba88-055cedf047eb" containerName="istio-proxy" containerID="cri-o://7bad9c5394a4255e0fe64928f63376cbadadc017764609e2f30f1e6a26ad4294" gracePeriod=30
Nov 03 17:53:54.767780 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.767750    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:53:54.785854 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.785828    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-data\") pod \"a04c0ae8-7886-46c1-98c1-b2053479c608\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") "
Nov 03 17:53:54.786007 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.785900    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-envoy\") pod \"a04c0ae8-7886-46c1-98c1-b2053479c608\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") "
Nov 03 17:53:54.786007 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.786000    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-token\" (UniqueName: \"kubernetes.io/projected/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-token\") pod \"a04c0ae8-7886-46c1-98c1-b2053479c608\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") "
Nov 03 17:53:54.786125 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.786058    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a04c0ae8-7886-46c1-98c1-b2053479c608-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a04c0ae8-7886-46c1-98c1-b2053479c608-istiod-ca-cert\") pod \"a04c0ae8-7886-46c1-98c1-b2053479c608\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") "
Nov 03 17:53:54.786125 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.786095    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-workload-socket\") pod \"a04c0ae8-7886-46c1-98c1-b2053479c608\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") "
Nov 03 17:53:54.786125 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.786101    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-data" pod "a04c0ae8-7886-46c1-98c1-b2053479c608" (UID: "a04c0ae8-7886-46c1-98c1-b2053479c608"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:54.786277 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.786133    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-podinfo\") pod \"a04c0ae8-7886-46c1-98c1-b2053479c608\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") "
Nov 03 17:53:54.786277 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.786202    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-workload-certs\") pod \"a04c0ae8-7886-46c1-98c1-b2053479c608\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") "
Nov 03 17:53:54.786277 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.786235    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a04c0ae8-7886-46c1-98c1-b2053479c608-kube-api-access-pjjr8\" (UniqueName: \"kubernetes.io/projected/a04c0ae8-7886-46c1-98c1-b2053479c608-kube-api-access-pjjr8\") pod \"a04c0ae8-7886-46c1-98c1-b2053479c608\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") "
Nov 03 17:53:54.786436 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.786381    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-credential-socket\") pod \"a04c0ae8-7886-46c1-98c1-b2053479c608\" (UID: \"a04c0ae8-7886-46c1-98c1-b2053479c608\") "
Nov 03 17:53:54.786494 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.786470    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-workload-socket" pod "a04c0ae8-7886-46c1-98c1-b2053479c608" (UID: "a04c0ae8-7886-46c1-98c1-b2053479c608"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:54.786554 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.786488    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a04c0ae8-7886-46c1-98c1-b2053479c608-istiod-ca-cert" pod "a04c0ae8-7886-46c1-98c1-b2053479c608" (UID: "a04c0ae8-7886-46c1-98c1-b2053479c608"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:53:54.786865 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.786837    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-workload-certs" pod "a04c0ae8-7886-46c1-98c1-b2053479c608" (UID: "a04c0ae8-7886-46c1-98c1-b2053479c608"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:54.786964 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.786899    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:54.786964 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.786920    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a04c0ae8-7886-46c1-98c1-b2053479c608-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:54.786964 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.786936    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:54.786964 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.786941    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-credential-socket" pod "a04c0ae8-7886-46c1-98c1-b2053479c608" (UID: "a04c0ae8-7886-46c1-98c1-b2053479c608"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:54.788410 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.788373    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-envoy" pod "a04c0ae8-7886-46c1-98c1-b2053479c608" (UID: "a04c0ae8-7886-46c1-98c1-b2053479c608"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:54.788763 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.788739    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-token" pod "a04c0ae8-7886-46c1-98c1-b2053479c608" (UID: "a04c0ae8-7886-46c1-98c1-b2053479c608"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:53:54.788876 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.788795    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-podinfo" pod "a04c0ae8-7886-46c1-98c1-b2053479c608" (UID: "a04c0ae8-7886-46c1-98c1-b2053479c608"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:53:54.788956 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.788936    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a04c0ae8-7886-46c1-98c1-b2053479c608-kube-api-access-pjjr8" pod "a04c0ae8-7886-46c1-98c1-b2053479c608" (UID: "a04c0ae8-7886-46c1-98c1-b2053479c608"). InnerVolumeSpecName "kube-api-access-pjjr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:53:54.810977 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.810958    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:53:54.887746 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.887630    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-workload-socket\") pod \"4983714f-6635-4557-9e25-80cf0c914d9d\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") "
Nov 03 17:53:54.887746 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.887683    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4983714f-6635-4557-9e25-80cf0c914d9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4983714f-6635-4557-9e25-80cf0c914d9d-istio-podinfo\") pod \"4983714f-6635-4557-9e25-80cf0c914d9d\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") "
Nov 03 17:53:54.887989 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.887772    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-credential-socket\") pod \"4983714f-6635-4557-9e25-80cf0c914d9d\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") "
Nov 03 17:53:54.887989 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.887819    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-workload-certs\") pod \"4983714f-6635-4557-9e25-80cf0c914d9d\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") "
Nov 03 17:53:54.887989 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.887854    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-istio-data\") pod \"4983714f-6635-4557-9e25-80cf0c914d9d\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") "
Nov 03 17:53:54.887989 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.887910    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4983714f-6635-4557-9e25-80cf0c914d9d-kube-api-access-lk2t7\" (UniqueName: \"kubernetes.io/projected/4983714f-6635-4557-9e25-80cf0c914d9d-kube-api-access-lk2t7\") pod \"4983714f-6635-4557-9e25-80cf0c914d9d\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") "
Nov 03 17:53:54.887989 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.887924    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-workload-socket" pod "4983714f-6635-4557-9e25-80cf0c914d9d" (UID: "4983714f-6635-4557-9e25-80cf0c914d9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:54.888273 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.888012    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4983714f-6635-4557-9e25-80cf0c914d9d-istio-token\" (UniqueName: \"kubernetes.io/projected/4983714f-6635-4557-9e25-80cf0c914d9d-istio-token\") pod \"4983714f-6635-4557-9e25-80cf0c914d9d\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") "
Nov 03 17:53:54.888273 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.888031    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-credential-socket" pod "4983714f-6635-4557-9e25-80cf0c914d9d" (UID: "4983714f-6635-4557-9e25-80cf0c914d9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:54.888273 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.888064    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4983714f-6635-4557-9e25-80cf0c914d9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4983714f-6635-4557-9e25-80cf0c914d9d-istiod-ca-cert\") pod \"4983714f-6635-4557-9e25-80cf0c914d9d\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") "
Nov 03 17:53:54.888273 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.888121    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-istio-envoy\") pod \"4983714f-6635-4557-9e25-80cf0c914d9d\" (UID: \"4983714f-6635-4557-9e25-80cf0c914d9d\") "
Nov 03 17:53:54.888273 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.888129    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-workload-certs" pod "4983714f-6635-4557-9e25-80cf0c914d9d" (UID: "4983714f-6635-4557-9e25-80cf0c914d9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:54.888273 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.888264    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-istio-data" pod "4983714f-6635-4557-9e25-80cf0c914d9d" (UID: "4983714f-6635-4557-9e25-80cf0c914d9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:54.888640 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.888567    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4983714f-6635-4557-9e25-80cf0c914d9d-istiod-ca-cert" pod "4983714f-6635-4557-9e25-80cf0c914d9d" (UID: "4983714f-6635-4557-9e25-80cf0c914d9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:53:54.888759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.888714    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:54.888759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.888737    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:54.888759 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.888751    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:54.888930 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.888765    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:54.888930 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.888780    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:54.888930 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.888793    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a04c0ae8-7886-46c1-98c1-b2053479c608-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:54.888930 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.888808    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4983714f-6635-4557-9e25-80cf0c914d9d-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:54.888930 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.888823    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:54.888930 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.888838    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjjr8\" (UniqueName: \"kubernetes.io/projected/a04c0ae8-7886-46c1-98c1-b2053479c608-kube-api-access-pjjr8\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:54.888930 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.888854    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:54.888930 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.888869    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04c0ae8-7886-46c1-98c1-b2053479c608-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:54.890148 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.890126    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-istio-envoy" pod "4983714f-6635-4557-9e25-80cf0c914d9d" (UID: "4983714f-6635-4557-9e25-80cf0c914d9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:54.890266 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.890251    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4983714f-6635-4557-9e25-80cf0c914d9d-kube-api-access-lk2t7" pod "4983714f-6635-4557-9e25-80cf0c914d9d" (UID: "4983714f-6635-4557-9e25-80cf0c914d9d"). InnerVolumeSpecName "kube-api-access-lk2t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:53:54.890349 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.890333    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4983714f-6635-4557-9e25-80cf0c914d9d-istio-podinfo" pod "4983714f-6635-4557-9e25-80cf0c914d9d" (UID: "4983714f-6635-4557-9e25-80cf0c914d9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:53:54.890472 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.890457    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4983714f-6635-4557-9e25-80cf0c914d9d-istio-token" pod "4983714f-6635-4557-9e25-80cf0c914d9d" (UID: "4983714f-6635-4557-9e25-80cf0c914d9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:53:54.990482 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.990449    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4983714f-6635-4557-9e25-80cf0c914d9d-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:54.990482 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.990477    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk2t7\" (UniqueName: \"kubernetes.io/projected/4983714f-6635-4557-9e25-80cf0c914d9d-kube-api-access-lk2t7\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:54.990482 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.990487    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4983714f-6635-4557-9e25-80cf0c914d9d-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:54.990719 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:54.990495    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4983714f-6635-4557-9e25-80cf0c914d9d-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:55.091494 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:55.091461    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-d27edff159f1: secret "agc-cred-d27edff159f1" not found
Nov 03 17:53:55.091494 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:55.091498    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-ec4400c42669: secret "agc-cred-ec4400c42669" not found
Nov 03 17:53:55.091707 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:55.091509    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv: [secret "agc-cred-d27edff159f1" not found, secret "agc-cred-ec4400c42669" not found]
Nov 03 17:53:55.091707 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:55.091561    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-provider-credentials podName:3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1 nodeName:}" failed. No retries permitted until 2026-11-03 17:54:03.091546979 +0000 UTC m=+2183.889259737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv" (UID: "3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1") : [secret "agc-cred-d27edff159f1" not found, secret "agc-cred-ec4400c42669" not found]
Nov 03 17:53:55.124486 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:55.124452    6059 generic.go:356] "Generic (PLEG): container finished" podID="a04c0ae8-7886-46c1-98c1-b2053479c608" containerID="9e9e14cac618766d4e04ff853ddba7be78f2cf83cee586db43a4998baac4975e" exitCode=0
Nov 03 17:53:55.124677 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:55.124515    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"
Nov 03 17:53:55.124677 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:55.124536    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp" event={"ID":"a04c0ae8-7886-46c1-98c1-b2053479c608","Type":"ContainerDied","Data":"9e9e14cac618766d4e04ff853ddba7be78f2cf83cee586db43a4998baac4975e"}
Nov 03 17:53:55.124677 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:55.124593    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp" event={"ID":"a04c0ae8-7886-46c1-98c1-b2053479c608","Type":"ContainerDied","Data":"f2d7dda6e9ed8a1e0ae7777b0e820e94be88705b012709a2c79ba664b1d1273a"}
Nov 03 17:53:55.124677 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:55.124610    6059 scope.go:122] "RemoveContainer" containerID="9e9e14cac618766d4e04ff853ddba7be78f2cf83cee586db43a4998baac4975e"
Nov 03 17:53:55.126046 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:55.126022    6059 generic.go:356] "Generic (PLEG): container finished" podID="4983714f-6635-4557-9e25-80cf0c914d9d" containerID="96f71e58b485513665faa6f2e6feb84f72ca5c305dc7592ecd68f1d5b8998e43" exitCode=0
Nov 03 17:53:55.126183 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:55.126075    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn" event={"ID":"4983714f-6635-4557-9e25-80cf0c914d9d","Type":"ContainerDied","Data":"96f71e58b485513665faa6f2e6feb84f72ca5c305dc7592ecd68f1d5b8998e43"}
Nov 03 17:53:55.126183 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:55.126132    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn" event={"ID":"4983714f-6635-4557-9e25-80cf0c914d9d","Type":"ContainerDied","Data":"f1c6b0183901434531802d1a87e6fac1ec1a98b4401bf8788c807f61fcf4db9c"}
Nov 03 17:53:55.126183 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:55.126079    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"
Nov 03 17:53:55.134738 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:55.134713    6059 scope.go:122] "RemoveContainer" containerID="9e9e14cac618766d4e04ff853ddba7be78f2cf83cee586db43a4998baac4975e"
Nov 03 17:53:55.135067 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:55.135048    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e9e14cac618766d4e04ff853ddba7be78f2cf83cee586db43a4998baac4975e\": container with ID starting with 9e9e14cac618766d4e04ff853ddba7be78f2cf83cee586db43a4998baac4975e not found: ID does not exist" containerID="9e9e14cac618766d4e04ff853ddba7be78f2cf83cee586db43a4998baac4975e"
Nov 03 17:53:55.135140 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:55.135077    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e9e14cac618766d4e04ff853ddba7be78f2cf83cee586db43a4998baac4975e"} err="failed to get container status \"9e9e14cac618766d4e04ff853ddba7be78f2cf83cee586db43a4998baac4975e\": rpc error: code = NotFound desc = could not find container \"9e9e14cac618766d4e04ff853ddba7be78f2cf83cee586db43a4998baac4975e\": container with ID starting with 9e9e14cac618766d4e04ff853ddba7be78f2cf83cee586db43a4998baac4975e not found: ID does not exist"
Nov 03 17:53:55.135140 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:55.135096    6059 scope.go:122] "RemoveContainer" containerID="96f71e58b485513665faa6f2e6feb84f72ca5c305dc7592ecd68f1d5b8998e43"
Nov 03 17:53:55.144345 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:55.144322    6059 scope.go:122] "RemoveContainer" containerID="96f71e58b485513665faa6f2e6feb84f72ca5c305dc7592ecd68f1d5b8998e43"
Nov 03 17:53:55.144619 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:53:55.144601    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f71e58b485513665faa6f2e6feb84f72ca5c305dc7592ecd68f1d5b8998e43\": container with ID starting with 96f71e58b485513665faa6f2e6feb84f72ca5c305dc7592ecd68f1d5b8998e43 not found: ID does not exist" containerID="96f71e58b485513665faa6f2e6feb84f72ca5c305dc7592ecd68f1d5b8998e43"
Nov 03 17:53:55.144664 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:55.144628    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f71e58b485513665faa6f2e6feb84f72ca5c305dc7592ecd68f1d5b8998e43"} err="failed to get container status \"96f71e58b485513665faa6f2e6feb84f72ca5c305dc7592ecd68f1d5b8998e43\": rpc error: code = NotFound desc = could not find container \"96f71e58b485513665faa6f2e6feb84f72ca5c305dc7592ecd68f1d5b8998e43\": container with ID starting with 96f71e58b485513665faa6f2e6feb84f72ca5c305dc7592ecd68f1d5b8998e43 not found: ID does not exist"
Nov 03 17:53:55.296660 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:55.296630    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"]
Nov 03 17:53:55.308911 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:55.306880    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn"]
Nov 03 17:53:55.335438 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:55.335408    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4983714f-6635-4557-9e25-80cf0c914d9d" path="/var/lib/kubelet/pods/4983714f-6635-4557-9e25-80cf0c914d9d/volumes"
Nov 03 17:53:55.420530 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:55.420444    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"]
Nov 03 17:53:55.430503 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:55.430468    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-gwjmp"]
Nov 03 17:53:55.751956 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:55.751916    6059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:53:55.752141 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:55.751998    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-s4bvn" podUID="4983714f-6635-4557-9e25-80cf0c914d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:53:56.335343 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:56.335307    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:57.336007 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:57.335977    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a04c0ae8-7886-46c1-98c1-b2053479c608" path="/var/lib/kubelet/pods/a04c0ae8-7886-46c1-98c1-b2053479c608/volumes"
Nov 03 17:53:58.056671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:58.056636    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:53:59.053832 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.053796    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9becb93d-f24kn"]
Nov 03 17:53:59.105044 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.104983    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9becb93d-f24kn"]
Nov 03 17:53:59.159458 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.159424    6059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:53:59.159617 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.159486    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w" podUID="029774c2-7278-4d5b-ba88-055cedf047eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:53:59.336043 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.335967    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a601f158-37db-48fb-936d-e3c4f9ce235f" path="/var/lib/kubelet/pods/a601f158-37db-48fb-936d-e3c4f9ce235f/volumes"
Nov 03 17:53:59.762052 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.762028    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:53:59.846288 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.846258    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-credential-socket\") pod \"029774c2-7278-4d5b-ba88-055cedf047eb\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") "
Nov 03 17:53:59.846439 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.846385    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-istio-data\") pod \"029774c2-7278-4d5b-ba88-055cedf047eb\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") "
Nov 03 17:53:59.846439 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.846429    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-istio-envoy\") pod \"029774c2-7278-4d5b-ba88-055cedf047eb\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") "
Nov 03 17:53:59.846544 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.846483    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/029774c2-7278-4d5b-ba88-055cedf047eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029774c2-7278-4d5b-ba88-055cedf047eb-istiod-ca-cert\") pod \"029774c2-7278-4d5b-ba88-055cedf047eb\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") "
Nov 03 17:53:59.846544 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.846524    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-workload-socket\") pod \"029774c2-7278-4d5b-ba88-055cedf047eb\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") "
Nov 03 17:53:59.846659 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.846553    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/029774c2-7278-4d5b-ba88-055cedf047eb-kube-api-access-8k257\" (UniqueName: \"kubernetes.io/projected/029774c2-7278-4d5b-ba88-055cedf047eb-kube-api-access-8k257\") pod \"029774c2-7278-4d5b-ba88-055cedf047eb\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") "
Nov 03 17:53:59.846721 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.846549    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-credential-socket" pod "029774c2-7278-4d5b-ba88-055cedf047eb" (UID: "029774c2-7278-4d5b-ba88-055cedf047eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:59.846721 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.846654    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-istio-data" pod "029774c2-7278-4d5b-ba88-055cedf047eb" (UID: "029774c2-7278-4d5b-ba88-055cedf047eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:59.846721 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.846682    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/029774c2-7278-4d5b-ba88-055cedf047eb-istio-token\" (UniqueName: \"kubernetes.io/projected/029774c2-7278-4d5b-ba88-055cedf047eb-istio-token\") pod \"029774c2-7278-4d5b-ba88-055cedf047eb\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") "
Nov 03 17:53:59.846895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.846758    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-workload-certs\") pod \"029774c2-7278-4d5b-ba88-055cedf047eb\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") "
Nov 03 17:53:59.846895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.846786    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-workload-socket" pod "029774c2-7278-4d5b-ba88-055cedf047eb" (UID: "029774c2-7278-4d5b-ba88-055cedf047eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:59.846895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.846793    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/029774c2-7278-4d5b-ba88-055cedf047eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029774c2-7278-4d5b-ba88-055cedf047eb-istio-podinfo\") pod \"029774c2-7278-4d5b-ba88-055cedf047eb\" (UID: \"029774c2-7278-4d5b-ba88-055cedf047eb\") "
Nov 03 17:53:59.846895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.846880    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/029774c2-7278-4d5b-ba88-055cedf047eb-istiod-ca-cert" pod "029774c2-7278-4d5b-ba88-055cedf047eb" (UID: "029774c2-7278-4d5b-ba88-055cedf047eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:53:59.847076 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.847001    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-workload-certs" pod "029774c2-7278-4d5b-ba88-055cedf047eb" (UID: "029774c2-7278-4d5b-ba88-055cedf047eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:59.847406 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.847380    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:59.847535 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.847410    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:59.847535 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.847428    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:59.847535 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.847444    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029774c2-7278-4d5b-ba88-055cedf047eb-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:59.847535 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.847459    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:59.848741 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.848714    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-istio-envoy" pod "029774c2-7278-4d5b-ba88-055cedf047eb" (UID: "029774c2-7278-4d5b-ba88-055cedf047eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:53:59.848847 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.848779    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029774c2-7278-4d5b-ba88-055cedf047eb-istio-token" pod "029774c2-7278-4d5b-ba88-055cedf047eb" (UID: "029774c2-7278-4d5b-ba88-055cedf047eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:53:59.848847 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.848803    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029774c2-7278-4d5b-ba88-055cedf047eb-kube-api-access-8k257" pod "029774c2-7278-4d5b-ba88-055cedf047eb" (UID: "029774c2-7278-4d5b-ba88-055cedf047eb"). InnerVolumeSpecName "kube-api-access-8k257". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:53:59.848932 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.848906    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/029774c2-7278-4d5b-ba88-055cedf047eb-istio-podinfo" pod "029774c2-7278-4d5b-ba88-055cedf047eb" (UID: "029774c2-7278-4d5b-ba88-055cedf047eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:53:59.949244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.949154    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029774c2-7278-4d5b-ba88-055cedf047eb-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:59.949244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.949197    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029774c2-7278-4d5b-ba88-055cedf047eb-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:59.949244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.949215    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k257\" (UniqueName: \"kubernetes.io/projected/029774c2-7278-4d5b-ba88-055cedf047eb-kube-api-access-8k257\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:53:59.949244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:53:59.949232    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029774c2-7278-4d5b-ba88-055cedf047eb-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:54:00.166302 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:00.166267    6059 generic.go:356] "Generic (PLEG): container finished" podID="029774c2-7278-4d5b-ba88-055cedf047eb" containerID="7bad9c5394a4255e0fe64928f63376cbadadc017764609e2f30f1e6a26ad4294" exitCode=0
Nov 03 17:54:00.166792 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:00.166320    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"
Nov 03 17:54:00.166792 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:00.166375    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w" event={"ID":"029774c2-7278-4d5b-ba88-055cedf047eb","Type":"ContainerDied","Data":"7bad9c5394a4255e0fe64928f63376cbadadc017764609e2f30f1e6a26ad4294"}
Nov 03 17:54:00.166792 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:00.166439    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w" event={"ID":"029774c2-7278-4d5b-ba88-055cedf047eb","Type":"ContainerDied","Data":"537d3bb93feb470c1b5ce542ddca7cd77ac1ca2bd25e1f7317fbaa2c590928b1"}
Nov 03 17:54:00.166792 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:00.166473    6059 scope.go:122] "RemoveContainer" containerID="7bad9c5394a4255e0fe64928f63376cbadadc017764609e2f30f1e6a26ad4294"
Nov 03 17:54:00.178089 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:00.178071    6059 scope.go:122] "RemoveContainer" containerID="7bad9c5394a4255e0fe64928f63376cbadadc017764609e2f30f1e6a26ad4294"
Nov 03 17:54:00.178396 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:54:00.178376    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bad9c5394a4255e0fe64928f63376cbadadc017764609e2f30f1e6a26ad4294\": container with ID starting with 7bad9c5394a4255e0fe64928f63376cbadadc017764609e2f30f1e6a26ad4294 not found: ID does not exist" containerID="7bad9c5394a4255e0fe64928f63376cbadadc017764609e2f30f1e6a26ad4294"
Nov 03 17:54:00.178494 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:00.178402    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bad9c5394a4255e0fe64928f63376cbadadc017764609e2f30f1e6a26ad4294"} err="failed to get container status \"7bad9c5394a4255e0fe64928f63376cbadadc017764609e2f30f1e6a26ad4294\": rpc error: code = NotFound desc = could not find container \"7bad9c5394a4255e0fe64928f63376cbadadc017764609e2f30f1e6a26ad4294\": container with ID starting with 7bad9c5394a4255e0fe64928f63376cbadadc017764609e2f30f1e6a26ad4294 not found: ID does not exist"
Nov 03 17:54:00.339685 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:00.339648    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"]
Nov 03 17:54:00.342533 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:00.342501    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6l89w"]
Nov 03 17:54:00.590733 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:00.590640    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:00.774338 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:00.774308    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:00.953418 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:00.953336    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:01.101192 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:01.101158    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"
Nov 03 17:54:01.275090 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:01.275056    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"]
Nov 03 17:54:01.336310 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:01.336276    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="029774c2-7278-4d5b-ba88-055cedf047eb" path="/var/lib/kubelet/pods/029774c2-7278-4d5b-ba88-055cedf047eb/volumes"
Nov 03 17:54:01.456421 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:01.456392    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:01.640584 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:01.640496    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:02.770606 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:02.770558    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:02.998635 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:02.998603    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:03.055193 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:03.055112    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10021fbc-6q4s6"]
Nov 03 17:54:03.106207 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:03.106177    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10021fbc-6q4s6"]
Nov 03 17:54:03.190282 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:54:03.190250    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-d27edff159f1: secret "agc-cred-d27edff159f1" not found
Nov 03 17:54:03.190282 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:54:03.190277    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-ec4400c42669: secret "agc-cred-ec4400c42669" not found
Nov 03 17:54:03.190282 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:54:03.190288    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv: [secret "agc-cred-d27edff159f1" not found, secret "agc-cred-ec4400c42669" not found]
Nov 03 17:54:03.190635 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:54:03.190384    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-provider-credentials podName:3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1 nodeName:}" failed. No retries permitted until 2026-11-03 17:54:19.1903363 +0000 UTC m=+2199.988049073 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv" (UID: "3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1") : [secret "agc-cred-d27edff159f1" not found, secret "agc-cred-ec4400c42669" not found]
Nov 03 17:54:03.336046 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:03.335968    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="319eee54-1366-4f28-a395-23f1c57a0b7d" path="/var/lib/kubelet/pods/319eee54-1366-4f28-a395-23f1c57a0b7d/volumes"
Nov 03 17:54:03.525011 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:03.524980    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:06.276370 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:06.276308    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv" podUID="3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1" containerName="payload-processing" containerID="cri-o://a6d0386c1f1ccc35088e8db4c04b16675f2192ef5fb4ee7f139b311f6e8e183a" gracePeriod=25
Nov 03 17:54:06.522826 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:06.522798    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"
Nov 03 17:54:06.630191 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:06.630108    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-provider-credentials\") pod \"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1\" (UID: \"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1\") "
Nov 03 17:54:06.630191 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:06.630171    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-routing-overlay-volume\") pod \"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1\" (UID: \"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1\") "
Nov 03 17:54:06.630460 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:06.630286    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-plugins-config-volume\") pod \"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1\" (UID: \"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1\") "
Nov 03 17:54:06.630460 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:06.630395    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-kube-api-access-tmx2x\" (UniqueName: \"kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-kube-api-access-tmx2x\") pod \"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1\" (UID: \"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1\") "
Nov 03 17:54:06.630610 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:06.630583    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-routing-overlay-volume" pod "3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1" (UID: "3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:54:06.630678 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:06.630653    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-plugins-config-volume" pod "3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1" (UID: "3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:54:06.630885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:06.630865    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:54:06.630928 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:06.630887    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:54:06.632112 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:06.632092    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-provider-credentials" pod "3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1" (UID: "3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:54:06.632252 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:06.632230    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-kube-api-access-tmx2x" pod "3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1" (UID: "3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1"). InnerVolumeSpecName "kube-api-access-tmx2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:54:06.731687 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:06.731655    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmx2x\" (UniqueName: \"kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-kube-api-access-tmx2x\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:54:06.731687 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:06.731683    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:54:06.861860 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:54:06.861825    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:54:06.862926 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:06.862900    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:07.220715 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:07.220676    6059 generic.go:356] "Generic (PLEG): container finished" podID="3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1" containerID="a6d0386c1f1ccc35088e8db4c04b16675f2192ef5fb4ee7f139b311f6e8e183a" exitCode=0
Nov 03 17:54:07.220908 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:07.220736    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"
Nov 03 17:54:07.220908 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:07.220785    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv" event={"ID":"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1","Type":"ContainerDied","Data":"a6d0386c1f1ccc35088e8db4c04b16675f2192ef5fb4ee7f139b311f6e8e183a"}
Nov 03 17:54:07.220908 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:07.220890    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv" event={"ID":"3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1","Type":"ContainerDied","Data":"e479930a583e136bce42b4e6922c5246c292c842013111b587e14724af7fed8e"}
Nov 03 17:54:07.221078 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:07.220926    6059 scope.go:122] "RemoveContainer" containerID="a6d0386c1f1ccc35088e8db4c04b16675f2192ef5fb4ee7f139b311f6e8e183a"
Nov 03 17:54:07.231126 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:07.231106    6059 scope.go:122] "RemoveContainer" containerID="a6d0386c1f1ccc35088e8db4c04b16675f2192ef5fb4ee7f139b311f6e8e183a"
Nov 03 17:54:07.231461 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:54:07.231444    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6d0386c1f1ccc35088e8db4c04b16675f2192ef5fb4ee7f139b311f6e8e183a\": container with ID starting with a6d0386c1f1ccc35088e8db4c04b16675f2192ef5fb4ee7f139b311f6e8e183a not found: ID does not exist" containerID="a6d0386c1f1ccc35088e8db4c04b16675f2192ef5fb4ee7f139b311f6e8e183a"
Nov 03 17:54:07.231513 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:07.231467    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d0386c1f1ccc35088e8db4c04b16675f2192ef5fb4ee7f139b311f6e8e183a"} err="failed to get container status \"a6d0386c1f1ccc35088e8db4c04b16675f2192ef5fb4ee7f139b311f6e8e183a\": rpc error: code = NotFound desc = could not find container \"a6d0386c1f1ccc35088e8db4c04b16675f2192ef5fb4ee7f139b311f6e8e183a\": container with ID starting with a6d0386c1f1ccc35088e8db4c04b16675f2192ef5fb4ee7f139b311f6e8e183a not found: ID does not exist"
Nov 03 17:54:07.393764 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:07.393734    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"]
Nov 03 17:54:07.445597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:07.445566    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6bb8mr6wv"]
Nov 03 17:54:07.715906 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:07.715880    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:09.335767 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:09.335735    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1" path="/var/lib/kubelet/pods/3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1/volumes"
Nov 03 17:54:09.546205 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:09.546170    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:10.005861 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:10.005825    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:10.096182 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:10.096150    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-270e09a7-wcwsw"]
Nov 03 17:54:10.098318 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:10.098296    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-270e09a7-wcwsw"]
Nov 03 17:54:11.335317 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:11.335281    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723" path="/var/lib/kubelet/pods/8c6cba9e-87d0-4a9b-a94e-9fdf8b4dc723/volumes"
Nov 03 17:54:12.222270 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:12.222218    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:13.022419 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:13.022376    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:13.197389 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:13.197347    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:15.036793 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:15.036761    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:16.657927 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.657896    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"]
Nov 03 17:54:16.658429 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.658415    6059 cpu_manager.go:414] "containerMap: removing container" podUID="4983714f-6635-4557-9e25-80cf0c914d9d" containerName="istio-proxy"
Nov 03 17:54:16.658429 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.658430    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4983714f-6635-4557-9e25-80cf0c914d9d" containerName="istio-proxy"
Nov 03 17:54:16.658527 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.658444    6059 cpu_manager.go:414] "containerMap: removing container" podUID="3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1" containerName="payload-processing"
Nov 03 17:54:16.658527 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.658450    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1" containerName="payload-processing"
Nov 03 17:54:16.658527 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.658477    6059 cpu_manager.go:414] "containerMap: removing container" podUID="a04c0ae8-7886-46c1-98c1-b2053479c608" containerName="istio-proxy"
Nov 03 17:54:16.658527 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.658483    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a04c0ae8-7886-46c1-98c1-b2053479c608" containerName="istio-proxy"
Nov 03 17:54:16.658527 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.658497    6059 cpu_manager.go:414] "containerMap: removing container" podUID="029774c2-7278-4d5b-ba88-055cedf047eb" containerName="istio-proxy"
Nov 03 17:54:16.658527 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.658502    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="029774c2-7278-4d5b-ba88-055cedf047eb" containerName="istio-proxy"
Nov 03 17:54:16.658720 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.658594    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4983714f-6635-4557-9e25-80cf0c914d9d" containerName="istio-proxy"
Nov 03 17:54:16.658720 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.658605    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3713f17a-8bdf-4d6b-9a20-e8d0a3c752a1" containerName="payload-processing"
Nov 03 17:54:16.658720 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.658621    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a04c0ae8-7886-46c1-98c1-b2053479c608" containerName="istio-proxy"
Nov 03 17:54:16.658720 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.658630    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="029774c2-7278-4d5b-ba88-055cedf047eb" containerName="istio-proxy"
Nov 03 17:54:16.663194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.663176    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.772071 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.772041    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"]
Nov 03 17:54:16.847509 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.847474    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-podinfo\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.847509 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.847510    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-workload-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.847753 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.847538    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-credential-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.847753 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.847599    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f44k\" (UniqueName: \"kubernetes.io/projected/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-kube-api-access-2f44k\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.847753 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.847673    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-workload-certs\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.847753 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.847716    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-envoy\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.847753 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.847743    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istiod-ca-cert\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.848022 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.847818    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-token\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.848022 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.847881    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-data\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.949234 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.949139    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-data\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.949234 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.949223    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-podinfo\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.949526 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.949249    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-workload-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.949526 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.949277    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-credential-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.949526 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.949381    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f44k\" (UniqueName: \"kubernetes.io/projected/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-kube-api-access-2f44k\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.949526 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.949471    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-workload-certs\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.949762 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.949540    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-envoy\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.949762 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.949584    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istiod-ca-cert\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.949762 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.949620    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-data\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.949762 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.949628    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-token\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.949762 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.949659    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-workload-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.950053 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.949863    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-workload-certs\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.950053 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.949989    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-credential-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.950287 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.950265    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istiod-ca-cert\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.951704 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.951683    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-envoy\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:16.951810 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:16.951752    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-podinfo\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:17.012822 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:17.012789    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-token\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:17.012985 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:17.012948    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f44k\" (UniqueName: \"kubernetes.io/projected/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-kube-api-access-2f44k\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:17.027129 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:17.027077    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:17.276420 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:17.276381    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:17.766113 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:17.766086    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"]
Nov 03 17:54:17.768788 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:54:17.768760    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4a3d51_7e8f_4bac_a514_dd4072dbbdb7.slice/crio-c890b89dab0e30bc096c9e85b1accf3d26d6b55d595a59e4f39788a9c050881e WatchSource:0}: Error finding container c890b89dab0e30bc096c9e85b1accf3d26d6b55d595a59e4f39788a9c050881e: Status 404 returned error can't find the container with id c890b89dab0e30bc096c9e85b1accf3d26d6b55d595a59e4f39788a9c050881e
Nov 03 17:54:17.771075 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:17.771045    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:54:17.771129 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:17.771108    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:54:17.771171 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:17.771138    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 17:54:18.298371 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:18.298327    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49" event={"ID":"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7","Type":"ContainerStarted","Data":"92666eedae1e55cd91bd4bb8aed9d30fdfc64803b6ab9f9449ca95574f93fb45"}
Nov 03 17:54:18.298571 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:18.298408    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49" event={"ID":"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7","Type":"ContainerStarted","Data":"c890b89dab0e30bc096c9e85b1accf3d26d6b55d595a59e4f39788a9c050881e"}
Nov 03 17:54:18.554442 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:18.532215    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49" podStartSLOduration=2.532196619 podStartE2EDuration="2.532196619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:54:18.477521472 +0000 UTC m=+2199.275234252" watchObservedRunningTime="2026-11-03 17:54:18.532196619 +0000 UTC m=+2199.329909401"
Nov 03 17:54:19.276755 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:19.276715    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:19.278638 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:19.278616    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 03 17:54:19.278713 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:19.278671    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49" podUID="4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 03 17:54:20.276836 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:20.276800    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 03 17:54:20.277252 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:20.276853    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49" podUID="4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 03 17:54:21.276890 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:21.276851    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 03 17:54:21.277340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:21.276911    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49" podUID="4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 03 17:54:21.551435 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:21.550221    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:22.277376 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:22.277319    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 03 17:54:22.277740 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:22.277389    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49" podUID="4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 03 17:54:23.277211 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:23.277175    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 03 17:54:23.277498 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:23.277231    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49" podUID="4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 03 17:54:24.286075 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:24.286040    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:24.286572 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:24.286405    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:24.287246 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:24.287230    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:54:26.016519 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:26.016487    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:30.548785 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:30.548744    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:35.012685 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:35.012653    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:36.860830 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:54:36.860796    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:54:37.051301 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:37.051272    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn"]
Nov 03 17:54:37.102683 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:37.102639    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecf52bc6-ghpzn"]
Nov 03 17:54:37.335755 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:37.335713    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="833d9419-150a-4400-a138-037f3604f62b" path="/var/lib/kubelet/pods/833d9419-150a-4400-a138-037f3604f62b/volumes"
Nov 03 17:54:39.548399 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:39.548347    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:42.395067 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:42.395029    6059 scope.go:122] "RemoveContainer" containerID="080859bb97ebb0ee35adc5754688501df431f80da038a19662acb1a03aa5bb4b"
Nov 03 17:54:42.404612 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:42.404591    6059 scope.go:122] "RemoveContainer" containerID="6724da802912d322cee7de47450a23e54116f20e4fbb36871b4e087056aeef7e"
Nov 03 17:54:42.412938 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:42.412916    6059 scope.go:122] "RemoveContainer" containerID="fb34841d783d408f2a835433c534daf991a52cb62bc07b7325064463a7a678ff"
Nov 03 17:54:42.421556 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:42.421528    6059 scope.go:122] "RemoveContainer" containerID="cdad340cdb7dad6c2d4d6814f5bac6f5d5a3175ae0a7385145cddb0018e3ac13"
Nov 03 17:54:42.946306 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:42.946266    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:44.116464 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:44.116429    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:48.622995 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:48.622956    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:51.538650 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:51.538616    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9"]
Nov 03 17:54:51.637102 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:51.637069    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n"]
Nov 03 17:54:51.902822 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:51.902746    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:52.147506 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:52.147460    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:52.383610 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:52.383568    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:53.214025 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:53.213994    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:56.538903 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:56.538859    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9" podUID="8e50ad7e-f804-40ed-b806-b2c904dc96cd" containerName="payload-processing" containerID="cri-o://35d2c6507f1de876f8b7fa683eb77f11cb04241b13e41995821019d482ba94cf" gracePeriod=25
Nov 03 17:54:56.638090 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:56.638050    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n" podUID="f787e049-8746-4ab8-83f0-d80a63fd3465" containerName="payload-pre-processing" containerID="cri-o://bbce04b1b3827634c82e520bcff59e1d1e0fdec966deb426f688c5b4aae0df23" gracePeriod=25
Nov 03 17:54:56.795802 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:56.795729    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9"
Nov 03 17:54:56.880201 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:56.880170    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e50ad7e-f804-40ed-b806-b2c904dc96cd-kube-api-access-fhcgx\" (UniqueName: \"kubernetes.io/projected/8e50ad7e-f804-40ed-b806-b2c904dc96cd-kube-api-access-fhcgx\") pod \"8e50ad7e-f804-40ed-b806-b2c904dc96cd\" (UID: \"8e50ad7e-f804-40ed-b806-b2c904dc96cd\") "
Nov 03 17:54:56.880451 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:56.880433    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e50ad7e-f804-40ed-b806-b2c904dc96cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e50ad7e-f804-40ed-b806-b2c904dc96cd-plugins-config-volume\") pod \"8e50ad7e-f804-40ed-b806-b2c904dc96cd\" (UID: \"8e50ad7e-f804-40ed-b806-b2c904dc96cd\") "
Nov 03 17:54:56.880756 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:56.880733    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e50ad7e-f804-40ed-b806-b2c904dc96cd-plugins-config-volume" pod "8e50ad7e-f804-40ed-b806-b2c904dc96cd" (UID: "8e50ad7e-f804-40ed-b806-b2c904dc96cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:54:56.880940 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:56.880920    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e50ad7e-f804-40ed-b806-b2c904dc96cd-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:54:56.882082 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:56.882063    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e50ad7e-f804-40ed-b806-b2c904dc96cd-kube-api-access-fhcgx" pod "8e50ad7e-f804-40ed-b806-b2c904dc96cd" (UID: "8e50ad7e-f804-40ed-b806-b2c904dc96cd"). InnerVolumeSpecName "kube-api-access-fhcgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:54:56.895619 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:56.895594    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n"
Nov 03 17:54:56.981289 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:56.981256    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f787e049-8746-4ab8-83f0-d80a63fd3465-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f787e049-8746-4ab8-83f0-d80a63fd3465-plugins-config-volume\") pod \"f787e049-8746-4ab8-83f0-d80a63fd3465\" (UID: \"f787e049-8746-4ab8-83f0-d80a63fd3465\") "
Nov 03 17:54:56.981508 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:56.981313    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f787e049-8746-4ab8-83f0-d80a63fd3465-kube-api-access-stqbc\" (UniqueName: \"kubernetes.io/projected/f787e049-8746-4ab8-83f0-d80a63fd3465-kube-api-access-stqbc\") pod \"f787e049-8746-4ab8-83f0-d80a63fd3465\" (UID: \"f787e049-8746-4ab8-83f0-d80a63fd3465\") "
Nov 03 17:54:56.981671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:56.981650    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f787e049-8746-4ab8-83f0-d80a63fd3465-plugins-config-volume" pod "f787e049-8746-4ab8-83f0-d80a63fd3465" (UID: "f787e049-8746-4ab8-83f0-d80a63fd3465"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:54:56.981830 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:56.981803    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f787e049-8746-4ab8-83f0-d80a63fd3465-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:54:56.981830 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:56.981823    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhcgx\" (UniqueName: \"kubernetes.io/projected/8e50ad7e-f804-40ed-b806-b2c904dc96cd-kube-api-access-fhcgx\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:54:56.983426 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:56.983406    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f787e049-8746-4ab8-83f0-d80a63fd3465-kube-api-access-stqbc" pod "f787e049-8746-4ab8-83f0-d80a63fd3465" (UID: "f787e049-8746-4ab8-83f0-d80a63fd3465"). InnerVolumeSpecName "kube-api-access-stqbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:54:57.082928 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:57.082848    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stqbc\" (UniqueName: \"kubernetes.io/projected/f787e049-8746-4ab8-83f0-d80a63fd3465-kube-api-access-stqbc\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:54:57.384180 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:57.384153    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:57.598126 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:57.598020    6059 generic.go:356] "Generic (PLEG): container finished" podID="8e50ad7e-f804-40ed-b806-b2c904dc96cd" containerID="35d2c6507f1de876f8b7fa683eb77f11cb04241b13e41995821019d482ba94cf" exitCode=0
Nov 03 17:54:57.598126 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:57.598095    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9"
Nov 03 17:54:57.598743 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:57.598201    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9" event={"ID":"8e50ad7e-f804-40ed-b806-b2c904dc96cd","Type":"ContainerDied","Data":"35d2c6507f1de876f8b7fa683eb77f11cb04241b13e41995821019d482ba94cf"}
Nov 03 17:54:57.598743 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:57.598262    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9" event={"ID":"8e50ad7e-f804-40ed-b806-b2c904dc96cd","Type":"ContainerDied","Data":"09ba6a91b8489582f666a17a2c826b0ed42291eefb1291346a958c3ea619a682"}
Nov 03 17:54:57.598743 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:57.598344    6059 scope.go:122] "RemoveContainer" containerID="35d2c6507f1de876f8b7fa683eb77f11cb04241b13e41995821019d482ba94cf"
Nov 03 17:54:57.602490 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:57.600544    6059 generic.go:356] "Generic (PLEG): container finished" podID="f787e049-8746-4ab8-83f0-d80a63fd3465" containerID="bbce04b1b3827634c82e520bcff59e1d1e0fdec966deb426f688c5b4aae0df23" exitCode=0
Nov 03 17:54:57.602490 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:57.600592    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n" event={"ID":"f787e049-8746-4ab8-83f0-d80a63fd3465","Type":"ContainerDied","Data":"bbce04b1b3827634c82e520bcff59e1d1e0fdec966deb426f688c5b4aae0df23"}
Nov 03 17:54:57.602490 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:57.600645    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n" event={"ID":"f787e049-8746-4ab8-83f0-d80a63fd3465","Type":"ContainerDied","Data":"5d8b7de90d3ae7a6039bde78c591a50bd43adca55bf01abcd480df53500e6e2a"}
Nov 03 17:54:57.602490 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:57.600746    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n"
Nov 03 17:54:57.614826 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:57.614803    6059 scope.go:122] "RemoveContainer" containerID="35d2c6507f1de876f8b7fa683eb77f11cb04241b13e41995821019d482ba94cf"
Nov 03 17:54:57.615225 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:54:57.615200    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d2c6507f1de876f8b7fa683eb77f11cb04241b13e41995821019d482ba94cf\": container with ID starting with 35d2c6507f1de876f8b7fa683eb77f11cb04241b13e41995821019d482ba94cf not found: ID does not exist" containerID="35d2c6507f1de876f8b7fa683eb77f11cb04241b13e41995821019d482ba94cf"
Nov 03 17:54:57.615455 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:57.615231    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d2c6507f1de876f8b7fa683eb77f11cb04241b13e41995821019d482ba94cf"} err="failed to get container status \"35d2c6507f1de876f8b7fa683eb77f11cb04241b13e41995821019d482ba94cf\": rpc error: code = NotFound desc = could not find container \"35d2c6507f1de876f8b7fa683eb77f11cb04241b13e41995821019d482ba94cf\": container with ID starting with 35d2c6507f1de876f8b7fa683eb77f11cb04241b13e41995821019d482ba94cf not found: ID does not exist"
Nov 03 17:54:57.615455 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:57.615262    6059 scope.go:122] "RemoveContainer" containerID="bbce04b1b3827634c82e520bcff59e1d1e0fdec966deb426f688c5b4aae0df23"
Nov 03 17:54:57.628001 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:57.627976    6059 scope.go:122] "RemoveContainer" containerID="bbce04b1b3827634c82e520bcff59e1d1e0fdec966deb426f688c5b4aae0df23"
Nov 03 17:54:57.628668 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:54:57.628644    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbce04b1b3827634c82e520bcff59e1d1e0fdec966deb426f688c5b4aae0df23\": container with ID starting with bbce04b1b3827634c82e520bcff59e1d1e0fdec966deb426f688c5b4aae0df23 not found: ID does not exist" containerID="bbce04b1b3827634c82e520bcff59e1d1e0fdec966deb426f688c5b4aae0df23"
Nov 03 17:54:57.628813 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:57.628674    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbce04b1b3827634c82e520bcff59e1d1e0fdec966deb426f688c5b4aae0df23"} err="failed to get container status \"bbce04b1b3827634c82e520bcff59e1d1e0fdec966deb426f688c5b4aae0df23\": rpc error: code = NotFound desc = could not find container \"bbce04b1b3827634c82e520bcff59e1d1e0fdec966deb426f688c5b4aae0df23\": container with ID starting with bbce04b1b3827634c82e520bcff59e1d1e0fdec966deb426f688c5b4aae0df23 not found: ID does not exist"
Nov 03 17:54:57.774424 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:57.774331    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n"]
Nov 03 17:54:57.778327 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:57.778292    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-fe8fde-8f4b7fb67-jb59n"]
Nov 03 17:54:57.891742 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:57.891638    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9"]
Nov 03 17:54:57.898502 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:57.898476    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-fe8fde-6db8976cf7-86qs9"]
Nov 03 17:54:58.532232 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:58.532195    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:58.718387 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:58.718346    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:58.900578 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:58.900496    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:59.120902 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:59.120861    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:54:59.335334 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:59.335304    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e50ad7e-f804-40ed-b806-b2c904dc96cd" path="/var/lib/kubelet/pods/8e50ad7e-f804-40ed-b806-b2c904dc96cd/volumes"
Nov 03 17:54:59.335736 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:59.335722    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f787e049-8746-4ab8-83f0-d80a63fd3465" path="/var/lib/kubelet/pods/f787e049-8746-4ab8-83f0-d80a63fd3465/volumes"
Nov 03 17:54:59.797146 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:54:59.797106    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:00.114297 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:00.114216    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:00.672296 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:00.672263    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"]
Nov 03 17:55:00.672524 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:00.672474    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp" podUID="945d8e0f-a47e-49e3-ae35-d25ad12aca81" containerName="istio-proxy" containerID="cri-o://b122d7e190368d629cf5ffda951a54dfa3df66efce74488bad8be6d34e2518e4" gracePeriod=30
Nov 03 17:55:00.698422 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:00.698378    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:01.336762 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:01.336726    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:01.982315 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:01.982282    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:02.172466 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:02.172425    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:04.021783 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:04.021742    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:04.333340 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:04.333228    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:04.586778 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:04.586690    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:04.765135 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:04.765095    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:05.933874 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.933848    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:55:05.986444 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.986349    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-credential-socket\") pod \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") "
Nov 03 17:55:05.986603 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.986497    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-workload-socket\") pod \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") "
Nov 03 17:55:05.986603 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.986526    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-workload-certs\") pod \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") "
Nov 03 17:55:05.986603 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.986556    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-token\" (UniqueName: \"kubernetes.io/projected/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-token\") pod \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") "
Nov 03 17:55:05.986603 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.986588    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istiod-ca-cert\") pod \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") "
Nov 03 17:55:05.986745 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.986603    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-credential-socket" pod "945d8e0f-a47e-49e3-ae35-d25ad12aca81" (UID: "945d8e0f-a47e-49e3-ae35-d25ad12aca81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:55:05.986745 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.986677    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945d8e0f-a47e-49e3-ae35-d25ad12aca81-kube-api-access-vz5zb\" (UniqueName: \"kubernetes.io/projected/945d8e0f-a47e-49e3-ae35-d25ad12aca81-kube-api-access-vz5zb\") pod \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") "
Nov 03 17:55:05.986745 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.986733    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-envoy\") pod \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") "
Nov 03 17:55:05.986910 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.986793    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-data\") pod \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") "
Nov 03 17:55:05.986910 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.986802    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-workload-certs" pod "945d8e0f-a47e-49e3-ae35-d25ad12aca81" (UID: "945d8e0f-a47e-49e3-ae35-d25ad12aca81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:55:05.987029 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.986831    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-workload-socket" pod "945d8e0f-a47e-49e3-ae35-d25ad12aca81" (UID: "945d8e0f-a47e-49e3-ae35-d25ad12aca81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:55:05.987029 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.986834    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-podinfo\") pod \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\" (UID: \"945d8e0f-a47e-49e3-ae35-d25ad12aca81\") "
Nov 03 17:55:05.987153 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.987118    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-data" pod "945d8e0f-a47e-49e3-ae35-d25ad12aca81" (UID: "945d8e0f-a47e-49e3-ae35-d25ad12aca81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:55:05.987153 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.987129    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istiod-ca-cert" pod "945d8e0f-a47e-49e3-ae35-d25ad12aca81" (UID: "945d8e0f-a47e-49e3-ae35-d25ad12aca81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:55:05.987581 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.987556    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:55:05.987710 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.987585    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:55:05.987710 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.987602    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:55:05.987710 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.987616    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:55:05.987710 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.987631    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:55:05.988892 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.988862    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-envoy" pod "945d8e0f-a47e-49e3-ae35-d25ad12aca81" (UID: "945d8e0f-a47e-49e3-ae35-d25ad12aca81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:55:05.989001 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.988943    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-token" pod "945d8e0f-a47e-49e3-ae35-d25ad12aca81" (UID: "945d8e0f-a47e-49e3-ae35-d25ad12aca81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:55:05.989001 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.988962    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-podinfo" pod "945d8e0f-a47e-49e3-ae35-d25ad12aca81" (UID: "945d8e0f-a47e-49e3-ae35-d25ad12aca81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:55:05.989078 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:05.989058    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945d8e0f-a47e-49e3-ae35-d25ad12aca81-kube-api-access-vz5zb" pod "945d8e0f-a47e-49e3-ae35-d25ad12aca81" (UID: "945d8e0f-a47e-49e3-ae35-d25ad12aca81"). InnerVolumeSpecName "kube-api-access-vz5zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:55:06.088974 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:06.088940    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz5zb\" (UniqueName: \"kubernetes.io/projected/945d8e0f-a47e-49e3-ae35-d25ad12aca81-kube-api-access-vz5zb\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:55:06.088974 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:06.088969    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:55:06.088974 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:06.088978    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:55:06.089228 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:06.088987    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945d8e0f-a47e-49e3-ae35-d25ad12aca81-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:55:06.666167 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:06.666134    6059 generic.go:356] "Generic (PLEG): container finished" podID="945d8e0f-a47e-49e3-ae35-d25ad12aca81" containerID="b122d7e190368d629cf5ffda951a54dfa3df66efce74488bad8be6d34e2518e4" exitCode=0
Nov 03 17:55:06.666344 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:06.666174    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp" event={"ID":"945d8e0f-a47e-49e3-ae35-d25ad12aca81","Type":"ContainerDied","Data":"b122d7e190368d629cf5ffda951a54dfa3df66efce74488bad8be6d34e2518e4"}
Nov 03 17:55:06.666344 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:06.666227    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp" event={"ID":"945d8e0f-a47e-49e3-ae35-d25ad12aca81","Type":"ContainerDied","Data":"c26cce9389a48951c2c0dfc844b9fd0970dfe24a7a4302c482f998e982d74d68"}
Nov 03 17:55:06.666344 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:06.666251    6059 scope.go:122] "RemoveContainer" containerID="b122d7e190368d629cf5ffda951a54dfa3df66efce74488bad8be6d34e2518e4"
Nov 03 17:55:06.666344 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:06.666188    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"
Nov 03 17:55:06.677176 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:06.677160    6059 scope.go:122] "RemoveContainer" containerID="5f7c4e9223356daff37422b90e7e797621341a59336d0ee6fd287f95389542f4"
Nov 03 17:55:06.685683 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:06.685664    6059 scope.go:122] "RemoveContainer" containerID="b122d7e190368d629cf5ffda951a54dfa3df66efce74488bad8be6d34e2518e4"
Nov 03 17:55:06.685930 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:55:06.685911    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b122d7e190368d629cf5ffda951a54dfa3df66efce74488bad8be6d34e2518e4\": container with ID starting with b122d7e190368d629cf5ffda951a54dfa3df66efce74488bad8be6d34e2518e4 not found: ID does not exist" containerID="b122d7e190368d629cf5ffda951a54dfa3df66efce74488bad8be6d34e2518e4"
Nov 03 17:55:06.686010 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:06.685943    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b122d7e190368d629cf5ffda951a54dfa3df66efce74488bad8be6d34e2518e4"} err="failed to get container status \"b122d7e190368d629cf5ffda951a54dfa3df66efce74488bad8be6d34e2518e4\": rpc error: code = NotFound desc = could not find container \"b122d7e190368d629cf5ffda951a54dfa3df66efce74488bad8be6d34e2518e4\": container with ID starting with b122d7e190368d629cf5ffda951a54dfa3df66efce74488bad8be6d34e2518e4 not found: ID does not exist"
Nov 03 17:55:06.686010 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:06.685968    6059 scope.go:122] "RemoveContainer" containerID="5f7c4e9223356daff37422b90e7e797621341a59336d0ee6fd287f95389542f4"
Nov 03 17:55:06.686213 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:55:06.686196    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f7c4e9223356daff37422b90e7e797621341a59336d0ee6fd287f95389542f4\": container with ID starting with 5f7c4e9223356daff37422b90e7e797621341a59336d0ee6fd287f95389542f4 not found: ID does not exist" containerID="5f7c4e9223356daff37422b90e7e797621341a59336d0ee6fd287f95389542f4"
Nov 03 17:55:06.686255 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:06.686221    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f7c4e9223356daff37422b90e7e797621341a59336d0ee6fd287f95389542f4"} err="failed to get container status \"5f7c4e9223356daff37422b90e7e797621341a59336d0ee6fd287f95389542f4\": rpc error: code = NotFound desc = could not find container \"5f7c4e9223356daff37422b90e7e797621341a59336d0ee6fd287f95389542f4\": container with ID starting with 5f7c4e9223356daff37422b90e7e797621341a59336d0ee6fd287f95389542f4 not found: ID does not exist"
Nov 03 17:55:06.837810 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:06.837782    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"]
Nov 03 17:55:06.860982 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:55:06.860953    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:55:06.889469 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:06.889444    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-fe8fde-openshift-default-65db8b7569-kv6lp"]
Nov 03 17:55:07.336542 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:07.336500    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="945d8e0f-a47e-49e3-ae35-d25ad12aca81" path="/var/lib/kubelet/pods/945d8e0f-a47e-49e3-ae35-d25ad12aca81/volumes"
Nov 03 17:55:08.534976 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:08.534610    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:08.758273 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:08.758236    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:13.030753 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:13.030719    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:17.668877 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:17.668785    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:22.216462 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:22.216424    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:26.003031 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:26.002995    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:26.208764 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:26.208734    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:26.530682 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:26.530648    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:26.740501 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:26.740465    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:26.931525 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:26.931441    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:31.420330 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:31.420297    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:31.728333 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:31.728301    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:35.928601 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:35.928560    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:36.860571 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:55:36.860536    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:55:40.420322 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:40.420273    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:42.481460 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:42.481420    6059 scope.go:122] "RemoveContainer" containerID="2523de2fd34c4ef3e674a9fd2c9f4a40e62c5d5abb0d9e85af8c4e184030aafb"
Nov 03 17:55:42.490853 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:42.490832    6059 scope.go:122] "RemoveContainer" containerID="650d4f1508ddcfd3cfc641eaa19e754642fcc526f30944cf12c25939f5badf1e"
Nov 03 17:55:42.499645 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:42.499627    6059 scope.go:122] "RemoveContainer" containerID="1aabe24a5c39a41ed6ce88c92445548236df1d06ac1e7dc173f48ec49ae18f68"
Nov 03 17:55:42.507649 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:42.507633    6059 scope.go:122] "RemoveContainer" containerID="b105c1f4217cc9ab61f03c538d1194b73c74047f6639ac55e43baffe363121f6"
Nov 03 17:55:42.515818 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:42.515798    6059 scope.go:122] "RemoveContainer" containerID="8bd87385e6dca90f145dd684e5c11489d790a25997ab7cfb2338d9f7dc723c04"
Nov 03 17:55:42.523606 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:42.523587    6059 scope.go:122] "RemoveContainer" containerID="c8d24fd9bee0d9ea5ae37594f37e65672741329e05040df561b2bdff04b3f9bc"
Nov 03 17:55:43.323404 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:43.323306    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:44.920447 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:44.920409    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:49.420383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:49.420273    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:54.021166 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:54.021129    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:57.228316 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:57.228282    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:55:58.512351 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:55:58.512303    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:03.013585 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:03.013550    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:06.860672 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:56:06.860629    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:56:07.528378 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:07.528236    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:09.041046 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:09.041013    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:09.549273 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:09.549239    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:12.034104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:12.033956    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:15.980868 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:15.980829    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:16.031288 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:16.031252    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg"]
Nov 03 17:56:16.417651 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:16.417541    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:16.608209 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:16.608157    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:16.785178 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:16.785123    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:21.032224 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:21.032187    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg" podUID="6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e" containerName="payload-processing" containerID="cri-o://ffe0b5f125351f2af54e830a750aa4486b02f13049b132dfbb9db513b0eeb8b2" gracePeriod=25
Nov 03 17:56:21.422946 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:21.422863    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:21.780109 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:21.780075    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:23.064257 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:23.064214    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg" podUID="6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 17:56:25.926519 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:25.926478    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:29.844126 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:29.844092    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:31.528329 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:31.528286    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:33.064336 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:33.064300    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg" podUID="6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 17:56:36.125138 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:36.125092    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:36.659387 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:36.659337    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:36.848572 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:36.848539    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:36.861386 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:56:36.861333    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:56:38.149724 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:38.149685    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:38.769415 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:38.769383    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"]
Nov 03 17:56:38.769623 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:38.769600    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s" podUID="fb73f8e5-cce4-4bca-9e84-717537d8a3e0" containerName="istio-proxy" containerID="cri-o://ec1ea4077664f643b0d540270b38a15ca2ceebed8a0ee6a8f2a509db8f1eccbc" gracePeriod=30
Nov 03 17:56:38.803977 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:38.803947    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:39.669479 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:39.669444    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:40.936526 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:40.936485    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:41.234149 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:41.234114    6059 generic.go:356] "Generic (PLEG): container finished" podID="6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e" containerID="ffe0b5f125351f2af54e830a750aa4486b02f13049b132dfbb9db513b0eeb8b2" exitCode=0
Nov 03 17:56:41.234336 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:41.234279    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg" event={"ID":"6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e","Type":"ContainerDied","Data":"ffe0b5f125351f2af54e830a750aa4486b02f13049b132dfbb9db513b0eeb8b2"}
Nov 03 17:56:41.289654 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:41.289626    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg"
Nov 03 17:56:41.443734 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:41.443702    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e-kube-api-access-26fpv\" (UniqueName: \"kubernetes.io/projected/6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e-kube-api-access-26fpv\") pod \"6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e\" (UID: \"6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e\") "
Nov 03 17:56:41.443917 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:41.443867    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e-plugins-config-volume\") pod \"6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e\" (UID: \"6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e\") "
Nov 03 17:56:41.444246 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:41.444218    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e-plugins-config-volume" pod "6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e" (UID: "6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:56:41.446068 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:41.446043    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e-kube-api-access-26fpv" pod "6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e" (UID: "6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e"). InnerVolumeSpecName "kube-api-access-26fpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:56:41.545819 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:41.545779    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26fpv\" (UniqueName: \"kubernetes.io/projected/6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e-kube-api-access-26fpv\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:56:41.545819 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:41.545818    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:56:42.243650 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:42.243601    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg" event={"ID":"6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e","Type":"ContainerDied","Data":"c6e7de8b8000e53d632f9319e297256cdcd48b9b4e039e68f72a4e338ae9bced"}
Nov 03 17:56:42.244122 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:42.243659    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg"
Nov 03 17:56:42.244122 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:42.243701    6059 scope.go:122] "RemoveContainer" containerID="ffe0b5f125351f2af54e830a750aa4486b02f13049b132dfbb9db513b0eeb8b2"
Nov 03 17:56:42.414493 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:42.414453    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg"]
Nov 03 17:56:42.417930 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:42.417904    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-af9e3c-589d8fb5d7-q4mxg"]
Nov 03 17:56:43.335520 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:43.335490    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e" path="/var/lib/kubelet/pods/6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e/volumes"
Nov 03 17:56:43.644402 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:43.644299    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:44.030289 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.030263    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:56:44.183566 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.183530    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istiod-ca-cert\") pod \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") "
Nov 03 17:56:44.183746 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.183575    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-kube-api-access-2zkjw\" (UniqueName: \"kubernetes.io/projected/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-kube-api-access-2zkjw\") pod \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") "
Nov 03 17:56:44.183746 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.183604    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-workload-certs\") pod \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") "
Nov 03 17:56:44.183838 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.183782    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-credential-socket\") pod \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") "
Nov 03 17:56:44.183908 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.183870    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-token\" (UniqueName: \"kubernetes.io/projected/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-token\") pod \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") "
Nov 03 17:56:44.183908 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.183903    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-workload-socket\") pod \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") "
Nov 03 17:56:44.184017 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.183930    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-data\") pod \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") "
Nov 03 17:56:44.184017 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.183949    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-workload-certs" pod "fb73f8e5-cce4-4bca-9e84-717537d8a3e0" (UID: "fb73f8e5-cce4-4bca-9e84-717537d8a3e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:56:44.184132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.184018    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-podinfo\") pod \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") "
Nov 03 17:56:44.184132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.184028    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-credential-socket" pod "fb73f8e5-cce4-4bca-9e84-717537d8a3e0" (UID: "fb73f8e5-cce4-4bca-9e84-717537d8a3e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:56:44.184132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.184062    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istiod-ca-cert" pod "fb73f8e5-cce4-4bca-9e84-717537d8a3e0" (UID: "fb73f8e5-cce4-4bca-9e84-717537d8a3e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:56:44.184132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.184072    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-envoy\") pod \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\" (UID: \"fb73f8e5-cce4-4bca-9e84-717537d8a3e0\") "
Nov 03 17:56:44.184383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.184143    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-workload-socket" pod "fb73f8e5-cce4-4bca-9e84-717537d8a3e0" (UID: "fb73f8e5-cce4-4bca-9e84-717537d8a3e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:56:44.184383 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.184271    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-data" pod "fb73f8e5-cce4-4bca-9e84-717537d8a3e0" (UID: "fb73f8e5-cce4-4bca-9e84-717537d8a3e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:56:44.184669 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.184607    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:56:44.184669 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.184636    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:56:44.184669 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.184654    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:56:44.184904 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.184671    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:56:44.184904 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.184690    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:56:44.186107 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.186086    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-token" pod "fb73f8e5-cce4-4bca-9e84-717537d8a3e0" (UID: "fb73f8e5-cce4-4bca-9e84-717537d8a3e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:56:44.186207 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.186129    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-podinfo" pod "fb73f8e5-cce4-4bca-9e84-717537d8a3e0" (UID: "fb73f8e5-cce4-4bca-9e84-717537d8a3e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:56:44.186400 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.186379    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-envoy" pod "fb73f8e5-cce4-4bca-9e84-717537d8a3e0" (UID: "fb73f8e5-cce4-4bca-9e84-717537d8a3e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:56:44.186400 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.186388    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-kube-api-access-2zkjw" pod "fb73f8e5-cce4-4bca-9e84-717537d8a3e0" (UID: "fb73f8e5-cce4-4bca-9e84-717537d8a3e0"). InnerVolumeSpecName "kube-api-access-2zkjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:56:44.257090 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.257054    6059 generic.go:356] "Generic (PLEG): container finished" podID="fb73f8e5-cce4-4bca-9e84-717537d8a3e0" containerID="ec1ea4077664f643b0d540270b38a15ca2ceebed8a0ee6a8f2a509db8f1eccbc" exitCode=0
Nov 03 17:56:44.257279 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.257116    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s" event={"ID":"fb73f8e5-cce4-4bca-9e84-717537d8a3e0","Type":"ContainerDied","Data":"ec1ea4077664f643b0d540270b38a15ca2ceebed8a0ee6a8f2a509db8f1eccbc"}
Nov 03 17:56:44.257279 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.257129    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"
Nov 03 17:56:44.257279 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.257158    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s" event={"ID":"fb73f8e5-cce4-4bca-9e84-717537d8a3e0","Type":"ContainerDied","Data":"4c46635e69e05839a6fa235ebc15b32689701f32062fbf53fea86276bafdacf7"}
Nov 03 17:56:44.257279 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.257178    6059 scope.go:122] "RemoveContainer" containerID="ec1ea4077664f643b0d540270b38a15ca2ceebed8a0ee6a8f2a509db8f1eccbc"
Nov 03 17:56:44.267682 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.267656    6059 scope.go:122] "RemoveContainer" containerID="ec1ea4077664f643b0d540270b38a15ca2ceebed8a0ee6a8f2a509db8f1eccbc"
Nov 03 17:56:44.268121 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:56:44.268094    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec1ea4077664f643b0d540270b38a15ca2ceebed8a0ee6a8f2a509db8f1eccbc\": container with ID starting with ec1ea4077664f643b0d540270b38a15ca2ceebed8a0ee6a8f2a509db8f1eccbc not found: ID does not exist" containerID="ec1ea4077664f643b0d540270b38a15ca2ceebed8a0ee6a8f2a509db8f1eccbc"
Nov 03 17:56:44.268230 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.268131    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1ea4077664f643b0d540270b38a15ca2ceebed8a0ee6a8f2a509db8f1eccbc"} err="failed to get container status \"ec1ea4077664f643b0d540270b38a15ca2ceebed8a0ee6a8f2a509db8f1eccbc\": rpc error: code = NotFound desc = could not find container \"ec1ea4077664f643b0d540270b38a15ca2ceebed8a0ee6a8f2a509db8f1eccbc\": container with ID starting with ec1ea4077664f643b0d540270b38a15ca2ceebed8a0ee6a8f2a509db8f1eccbc not found: ID does not exist"
Nov 03 17:56:44.285488 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.285461    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:56:44.285488 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.285489    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:56:44.285674 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.285500    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:56:44.285674 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.285509    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zkjw\" (UniqueName: \"kubernetes.io/projected/fb73f8e5-cce4-4bca-9e84-717537d8a3e0-kube-api-access-2zkjw\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:56:44.428692 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.428656    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"]
Nov 03 17:56:44.433155 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:44.433123    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-njl8s"]
Nov 03 17:56:45.335666 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:45.335629    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb73f8e5-cce4-4bca-9e84-717537d8a3e0" path="/var/lib/kubelet/pods/fb73f8e5-cce4-4bca-9e84-717537d8a3e0/volumes"
Nov 03 17:56:45.586824 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:45.586734    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:46.039769 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:46.039735    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq"]
Nov 03 17:56:46.120699 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:46.120658    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:46.317610 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:46.317502    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:46.490173 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:46.490132    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:50.221730 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:50.221699    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:51.040866 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:51.040825    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq" podUID="d2e5b0a4-8ae5-41d3-97b4-a13804480f77" containerName="payload-pre-processing" containerID="cri-o://f64ff090587f2e41e5e8b6bd96f46e91dd123487e982b28d20caba6de29ec0d3" gracePeriod=25
Nov 03 17:56:51.907225 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:51.907187    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:53.567322 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:53.567281    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq" podUID="d2e5b0a4-8ae5-41d3-97b4-a13804480f77" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 17:56:54.164807 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:54.164760    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:55.825760 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:55.825724    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:57.991400 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:57.991365    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:58.178311 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:58.178273    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:56:59.568250 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:56:59.568203    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:00.065836 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:00.065790    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"]
Nov 03 17:57:00.066107 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:00.066060    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx" podUID="012064a7-fd12-465b-bdc8-ef6d4ca289b8" containerName="istio-proxy" containerID="cri-o://ba3eb4fb0314502f6b0917100436f52d8e9d7065eec92be5a27ec5006eb62106" gracePeriod=30
Nov 03 17:57:00.092065 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:00.092013    6059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:57:00.092264 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:00.092090    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx" podUID="012064a7-fd12-465b-bdc8-ef6d4ca289b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:57:00.092489 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:00.092470    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:00.462899 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:00.462753    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:03.566840 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:03.566802    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq" podUID="d2e5b0a4-8ae5-41d3-97b4-a13804480f77" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 17:57:05.129144 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.129113    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:05.319770 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.319748    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:57:05.393346 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.393261    6059 generic.go:356] "Generic (PLEG): container finished" podID="012064a7-fd12-465b-bdc8-ef6d4ca289b8" containerID="ba3eb4fb0314502f6b0917100436f52d8e9d7065eec92be5a27ec5006eb62106" exitCode=0
Nov 03 17:57:05.393346 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.393313    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"
Nov 03 17:57:05.393572 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.393308    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx" event={"ID":"012064a7-fd12-465b-bdc8-ef6d4ca289b8","Type":"ContainerDied","Data":"ba3eb4fb0314502f6b0917100436f52d8e9d7065eec92be5a27ec5006eb62106"}
Nov 03 17:57:05.393572 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.393444    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx" event={"ID":"012064a7-fd12-465b-bdc8-ef6d4ca289b8","Type":"ContainerDied","Data":"d42499864b4c7fab994e37be32f8fc61f3e83ffe6fe6c7930c29943f273c3fe7"}
Nov 03 17:57:05.393572 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.393478    6059 scope.go:122] "RemoveContainer" containerID="ba3eb4fb0314502f6b0917100436f52d8e9d7065eec92be5a27ec5006eb62106"
Nov 03 17:57:05.402725 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.402708    6059 scope.go:122] "RemoveContainer" containerID="4273dbde4f713e9e9355e86370d12cd49133e36861a93b6300401162988e55ce"
Nov 03 17:57:05.410809 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.410793    6059 scope.go:122] "RemoveContainer" containerID="ba3eb4fb0314502f6b0917100436f52d8e9d7065eec92be5a27ec5006eb62106"
Nov 03 17:57:05.411067 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:05.411047    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba3eb4fb0314502f6b0917100436f52d8e9d7065eec92be5a27ec5006eb62106\": container with ID starting with ba3eb4fb0314502f6b0917100436f52d8e9d7065eec92be5a27ec5006eb62106 not found: ID does not exist" containerID="ba3eb4fb0314502f6b0917100436f52d8e9d7065eec92be5a27ec5006eb62106"
Nov 03 17:57:05.411145 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.411080    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba3eb4fb0314502f6b0917100436f52d8e9d7065eec92be5a27ec5006eb62106"} err="failed to get container status \"ba3eb4fb0314502f6b0917100436f52d8e9d7065eec92be5a27ec5006eb62106\": rpc error: code = NotFound desc = could not find container \"ba3eb4fb0314502f6b0917100436f52d8e9d7065eec92be5a27ec5006eb62106\": container with ID starting with ba3eb4fb0314502f6b0917100436f52d8e9d7065eec92be5a27ec5006eb62106 not found: ID does not exist"
Nov 03 17:57:05.411145 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.411107    6059 scope.go:122] "RemoveContainer" containerID="4273dbde4f713e9e9355e86370d12cd49133e36861a93b6300401162988e55ce"
Nov 03 17:57:05.411379 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:05.411339    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4273dbde4f713e9e9355e86370d12cd49133e36861a93b6300401162988e55ce\": container with ID starting with 4273dbde4f713e9e9355e86370d12cd49133e36861a93b6300401162988e55ce not found: ID does not exist" containerID="4273dbde4f713e9e9355e86370d12cd49133e36861a93b6300401162988e55ce"
Nov 03 17:57:05.411428 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.411388    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4273dbde4f713e9e9355e86370d12cd49133e36861a93b6300401162988e55ce"} err="failed to get container status \"4273dbde4f713e9e9355e86370d12cd49133e36861a93b6300401162988e55ce\": rpc error: code = NotFound desc = could not find container \"4273dbde4f713e9e9355e86370d12cd49133e36861a93b6300401162988e55ce\": container with ID starting with 4273dbde4f713e9e9355e86370d12cd49133e36861a93b6300401162988e55ce not found: ID does not exist"
Nov 03 17:57:05.430625 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.430598    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-token\" (UniqueName: \"kubernetes.io/projected/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-token\") pod \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") "
Nov 03 17:57:05.430753 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.430737    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-podinfo\") pod \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") "
Nov 03 17:57:05.430823 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.430793    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-credential-socket\") pod \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") "
Nov 03 17:57:05.430823 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.430819    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-data\") pod \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") "
Nov 03 17:57:05.430945 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.430906    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istiod-ca-cert\") pod \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") "
Nov 03 17:57:05.431144 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.431117    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-data" pod "012064a7-fd12-465b-bdc8-ef6d4ca289b8" (UID: "012064a7-fd12-465b-bdc8-ef6d4ca289b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:57:05.431233 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.431129    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-credential-socket" pod "012064a7-fd12-465b-bdc8-ef6d4ca289b8" (UID: "012064a7-fd12-465b-bdc8-ef6d4ca289b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:57:05.431333 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.431309    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istiod-ca-cert" pod "012064a7-fd12-465b-bdc8-ef6d4ca289b8" (UID: "012064a7-fd12-465b-bdc8-ef6d4ca289b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:57:05.431446 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.431427    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-workload-socket\") pod \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") "
Nov 03 17:57:05.431515 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.431473    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-workload-certs\") pod \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") "
Nov 03 17:57:05.431582 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.431512    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-envoy\") pod \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") "
Nov 03 17:57:05.431582 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.431574    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012064a7-fd12-465b-bdc8-ef6d4ca289b8-kube-api-access-l87ll\" (UniqueName: \"kubernetes.io/projected/012064a7-fd12-465b-bdc8-ef6d4ca289b8-kube-api-access-l87ll\") pod \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\" (UID: \"012064a7-fd12-465b-bdc8-ef6d4ca289b8\") "
Nov 03 17:57:05.431705 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.431664    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-workload-socket" pod "012064a7-fd12-465b-bdc8-ef6d4ca289b8" (UID: "012064a7-fd12-465b-bdc8-ef6d4ca289b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:57:05.431769 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.431718    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-workload-certs" pod "012064a7-fd12-465b-bdc8-ef6d4ca289b8" (UID: "012064a7-fd12-465b-bdc8-ef6d4ca289b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:57:05.432157 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.432138    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:57:05.432256 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.432163    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:57:05.432256 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.432179    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:57:05.432256 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.432197    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:57:05.432256 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.432212    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:57:05.433039 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.433018    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-podinfo" pod "012064a7-fd12-465b-bdc8-ef6d4ca289b8" (UID: "012064a7-fd12-465b-bdc8-ef6d4ca289b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:57:05.433172 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.433144    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-token" pod "012064a7-fd12-465b-bdc8-ef6d4ca289b8" (UID: "012064a7-fd12-465b-bdc8-ef6d4ca289b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:57:05.433584 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.433555    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-envoy" pod "012064a7-fd12-465b-bdc8-ef6d4ca289b8" (UID: "012064a7-fd12-465b-bdc8-ef6d4ca289b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:57:05.434060 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.434040    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012064a7-fd12-465b-bdc8-ef6d4ca289b8-kube-api-access-l87ll" pod "012064a7-fd12-465b-bdc8-ef6d4ca289b8" (UID: "012064a7-fd12-465b-bdc8-ef6d4ca289b8"). InnerVolumeSpecName "kube-api-access-l87ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:57:05.534043 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.533999    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l87ll\" (UniqueName: \"kubernetes.io/projected/012064a7-fd12-465b-bdc8-ef6d4ca289b8-kube-api-access-l87ll\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:57:05.534043 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.534033    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:57:05.534043 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.534047    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:57:05.534331 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.534061    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012064a7-fd12-465b-bdc8-ef6d4ca289b8-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:57:05.820042 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.820008    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"]
Nov 03 17:57:05.822181 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:05.822159    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-kg7qx"]
Nov 03 17:57:06.026419 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:06.026384    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:06.210128 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:06.210074    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:06.861106 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:06.861068    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:57:07.338377 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:07.337805    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="012064a7-fd12-465b-bdc8-ef6d4ca289b8" path="/var/lib/kubelet/pods/012064a7-fd12-465b-bdc8-ef6d4ca289b8/volumes"
Nov 03 17:57:09.499333 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:09.499301    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:09.675705 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:09.675657    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:11.300322 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:11.300259    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq"
Nov 03 17:57:11.393981 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:11.393943    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2e5b0a4-8ae5-41d3-97b4-a13804480f77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e5b0a4-8ae5-41d3-97b4-a13804480f77-plugins-config-volume\") pod \"d2e5b0a4-8ae5-41d3-97b4-a13804480f77\" (UID: \"d2e5b0a4-8ae5-41d3-97b4-a13804480f77\") "
Nov 03 17:57:11.394164 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:11.394074    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2e5b0a4-8ae5-41d3-97b4-a13804480f77-kube-api-access-s7268\" (UniqueName: \"kubernetes.io/projected/d2e5b0a4-8ae5-41d3-97b4-a13804480f77-kube-api-access-s7268\") pod \"d2e5b0a4-8ae5-41d3-97b4-a13804480f77\" (UID: \"d2e5b0a4-8ae5-41d3-97b4-a13804480f77\") "
Nov 03 17:57:11.394465 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:11.394436    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2e5b0a4-8ae5-41d3-97b4-a13804480f77-plugins-config-volume" pod "d2e5b0a4-8ae5-41d3-97b4-a13804480f77" (UID: "d2e5b0a4-8ae5-41d3-97b4-a13804480f77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:57:11.394716 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:11.394695    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e5b0a4-8ae5-41d3-97b4-a13804480f77-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:57:11.396242 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:11.396214    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e5b0a4-8ae5-41d3-97b4-a13804480f77-kube-api-access-s7268" pod "d2e5b0a4-8ae5-41d3-97b4-a13804480f77" (UID: "d2e5b0a4-8ae5-41d3-97b4-a13804480f77"). InnerVolumeSpecName "kube-api-access-s7268". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:57:11.430019 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:11.429985    6059 generic.go:356] "Generic (PLEG): container finished" podID="d2e5b0a4-8ae5-41d3-97b4-a13804480f77" containerID="f64ff090587f2e41e5e8b6bd96f46e91dd123487e982b28d20caba6de29ec0d3" exitCode=0
Nov 03 17:57:11.430171 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:11.430071    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq"
Nov 03 17:57:11.430171 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:11.430093    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq" event={"ID":"d2e5b0a4-8ae5-41d3-97b4-a13804480f77","Type":"ContainerDied","Data":"f64ff090587f2e41e5e8b6bd96f46e91dd123487e982b28d20caba6de29ec0d3"}
Nov 03 17:57:11.430171 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:11.430155    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq" event={"ID":"d2e5b0a4-8ae5-41d3-97b4-a13804480f77","Type":"ContainerDied","Data":"0b08d12e0bf042e14a9d61e93c45f82027d55965c9fb7669d885b2285d95b1c8"}
Nov 03 17:57:11.430332 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:11.430188    6059 scope.go:122] "RemoveContainer" containerID="f64ff090587f2e41e5e8b6bd96f46e91dd123487e982b28d20caba6de29ec0d3"
Nov 03 17:57:11.442597 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:11.442506    6059 scope.go:122] "RemoveContainer" containerID="f64ff090587f2e41e5e8b6bd96f46e91dd123487e982b28d20caba6de29ec0d3"
Nov 03 17:57:11.445114 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:11.445036    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f64ff090587f2e41e5e8b6bd96f46e91dd123487e982b28d20caba6de29ec0d3\": container with ID starting with f64ff090587f2e41e5e8b6bd96f46e91dd123487e982b28d20caba6de29ec0d3 not found: ID does not exist" containerID="f64ff090587f2e41e5e8b6bd96f46e91dd123487e982b28d20caba6de29ec0d3"
Nov 03 17:57:11.445114 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:11.445075    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f64ff090587f2e41e5e8b6bd96f46e91dd123487e982b28d20caba6de29ec0d3"} err="failed to get container status \"f64ff090587f2e41e5e8b6bd96f46e91dd123487e982b28d20caba6de29ec0d3\": rpc error: code = NotFound desc = could not find container \"f64ff090587f2e41e5e8b6bd96f46e91dd123487e982b28d20caba6de29ec0d3\": container with ID starting with f64ff090587f2e41e5e8b6bd96f46e91dd123487e982b28d20caba6de29ec0d3 not found: ID does not exist"
Nov 03 17:57:11.495318 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:11.495284    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7268\" (UniqueName: \"kubernetes.io/projected/d2e5b0a4-8ae5-41d3-97b4-a13804480f77-kube-api-access-s7268\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:57:11.601157 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:11.601076    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq"]
Nov 03 17:57:11.605194 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:11.605167    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-147834-6b7658bb6c-86rqq"]
Nov 03 17:57:13.335680 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:13.335647    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2e5b0a4-8ae5-41d3-97b4-a13804480f77" path="/var/lib/kubelet/pods/d2e5b0a4-8ae5-41d3-97b4-a13804480f77/volumes"
Nov 03 17:57:14.131554 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:14.131452    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:15.188866 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:15.188825    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:15.522352 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:15.522274    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:16.157100 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:16.157061    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:18.643094 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:18.643060    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:22.006291 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:22.006235    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:22.187889 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:22.187826    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:23.431151 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:23.431113    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:27.919940 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:27.919904    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:29.390026 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:29.389995    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-8758ca05391f: secret "agc-cred-8758ca05391f" not found
Nov 03 17:57:29.390026 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:29.390024    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg: secret "agc-cred-8758ca05391f" not found
Nov 03 17:57:29.390719 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:29.390080    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials podName:f5a452c7-51f0-4c5a-b092-d0507af37bb7 nodeName:}" failed. No retries permitted until 2026-11-03 17:57:29.890061902 +0000 UTC m=+2390.687774661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg" (UID: "f5a452c7-51f0-4c5a-b092-d0507af37bb7") : secret "agc-cred-8758ca05391f" not found
Nov 03 17:57:29.897436 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:29.897399    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-8758ca05391f: secret "agc-cred-8758ca05391f" not found
Nov 03 17:57:29.897436 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:29.897437    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg: secret "agc-cred-8758ca05391f" not found
Nov 03 17:57:29.897702 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:29.897519    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials podName:f5a452c7-51f0-4c5a-b092-d0507af37bb7 nodeName:}" failed. No retries permitted until 2026-11-03 17:57:30.897498034 +0000 UTC m=+2391.695210797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg" (UID: "f5a452c7-51f0-4c5a-b092-d0507af37bb7") : secret "agc-cred-8758ca05391f" not found
Nov 03 17:57:30.907930 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:30.907899    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-8758ca05391f: secret "agc-cred-8758ca05391f" not found
Nov 03 17:57:30.907930 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:30.907928    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg: secret "agc-cred-8758ca05391f" not found
Nov 03 17:57:30.908383 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:30.907980    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials podName:f5a452c7-51f0-4c5a-b092-d0507af37bb7 nodeName:}" failed. No retries permitted until 2026-11-03 17:57:32.907965207 +0000 UTC m=+2393.705677990 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg" (UID: "f5a452c7-51f0-4c5a-b092-d0507af37bb7") : secret "agc-cred-8758ca05391f" not found
Nov 03 17:57:32.382258 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:32.382217    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:32.935513 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:32.935473    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-8758ca05391f: secret "agc-cred-8758ca05391f" not found
Nov 03 17:57:32.935513 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:32.935509    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg: secret "agc-cred-8758ca05391f" not found
Nov 03 17:57:32.935812 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:32.935584    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials podName:f5a452c7-51f0-4c5a-b092-d0507af37bb7 nodeName:}" failed. No retries permitted until 2026-11-03 17:57:36.935566369 +0000 UTC m=+2397.733279142 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg" (UID: "f5a452c7-51f0-4c5a-b092-d0507af37bb7") : secret "agc-cred-8758ca05391f" not found
Nov 03 17:57:36.861016 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:36.860976    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:57:36.978697 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:36.978663    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-8758ca05391f: secret "agc-cred-8758ca05391f" not found
Nov 03 17:57:36.978697 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:36.978700    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg: secret "agc-cred-8758ca05391f" not found
Nov 03 17:57:36.978919 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:36.978765    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials podName:f5a452c7-51f0-4c5a-b092-d0507af37bb7 nodeName:}" failed. No retries permitted until 2026-11-03 17:57:44.978750752 +0000 UTC m=+2405.776463511 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg" (UID: "f5a452c7-51f0-4c5a-b092-d0507af37bb7") : secret "agc-cred-8758ca05391f" not found
Nov 03 17:57:37.018195 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:37.018126    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:39.732076 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:39.732040    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 17:57:39.743110 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:39.743089    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 17:57:39.750342 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:39.750322    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:57:39.760533 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:39.760515    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 17:57:41.526980 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:41.526936    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:44.031200 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:44.031157    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:45.073473 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:45.073439    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-8758ca05391f: secret "agc-cred-8758ca05391f" not found
Nov 03 17:57:45.073473 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:45.073468    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg: secret "agc-cred-8758ca05391f" not found
Nov 03 17:57:45.073894 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:57:45.073519    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials podName:f5a452c7-51f0-4c5a-b092-d0507af37bb7 nodeName:}" failed. No retries permitted until 2026-11-03 17:58:01.073503191 +0000 UTC m=+2421.871215949 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg" (UID: "f5a452c7-51f0-4c5a-b092-d0507af37bb7") : secret "agc-cred-8758ca05391f" not found
Nov 03 17:57:46.022350 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:46.022310    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:50.541132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:50.541057    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:55.011422 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:55.011378    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:57:59.514557 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:57:59.514515    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:00.278752 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:00.278717    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:00.457952 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:00.457918    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:01.150674 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:58:01.150641    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-8758ca05391f: secret "agc-cred-8758ca05391f" not found
Nov 03 17:58:01.150674 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:58:01.150671    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg: secret "agc-cred-8758ca05391f" not found
Nov 03 17:58:01.151108 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:58:01.150728    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials podName:f5a452c7-51f0-4c5a-b092-d0507af37bb7 nodeName:}" failed. No retries permitted until 2026-11-03 17:58:33.150707422 +0000 UTC m=+2453.948420181 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg" (UID: "f5a452c7-51f0-4c5a-b092-d0507af37bb7") : secret "agc-cred-8758ca05391f" not found
Nov 03 17:58:04.360077 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:04.360046    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:06.860182 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:58:06.860146    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:58:09.277214 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:09.277168    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:14.227837 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:14.227800    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:14.418380 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:14.414830    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:14.738249 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:14.738215    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:14.932104 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:14.932071    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:19.275032 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:19.274998    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:21.061889 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:21.061854    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-662943ba-l8qpg"]
Nov 03 17:58:21.113490 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:21.113460    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-662943ba-l8qpg"]
Nov 03 17:58:21.335579 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:21.335504    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4c6f904-64a1-4306-9810-47623362da96" path="/var/lib/kubelet/pods/f4c6f904-64a1-4306-9810-47623362da96/volumes"
Nov 03 17:58:24.192828 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:24.192785    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:27.310738 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:27.310708    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:27.486732 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:27.486699    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:29.070102 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:29.070071    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:33.235954 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:58:33.235916    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-198bcf954428: secret "agc-cred-198bcf954428" not found
Nov 03 17:58:33.235954 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:58:33.235946    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-8758ca05391f: secret "agc-cred-8758ca05391f" not found
Nov 03 17:58:33.235954 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:58:33.235963    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-dbae42e6384f: secret "agc-cred-dbae42e6384f" not found
Nov 03 17:58:33.236583 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:58:33.235976    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg: [secret "agc-cred-198bcf954428" not found, secret "agc-cred-8758ca05391f" not found, secret "agc-cred-dbae42e6384f" not found]
Nov 03 17:58:33.236583 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:58:33.236048    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials podName:f5a452c7-51f0-4c5a-b092-d0507af37bb7 nodeName:}" failed. No retries permitted until 2026-11-03 17:59:37.236025802 +0000 UTC m=+2518.033738561 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg" (UID: "f5a452c7-51f0-4c5a-b092-d0507af37bb7") : [secret "agc-cred-198bcf954428" not found, secret "agc-cred-8758ca05391f" not found, secret "agc-cred-dbae42e6384f" not found]
Nov 03 17:58:33.963011 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:33.962969    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:35.054052 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:35.054017    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f"]
Nov 03 17:58:35.057942 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:35.057916    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-759f5627-hl66f"]
Nov 03 17:58:35.335176 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:35.335082    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d86c7366-53ba-4c9e-bc59-345abef181f6" path="/var/lib/kubelet/pods/d86c7366-53ba-4c9e-bc59-345abef181f6/volumes"
Nov 03 17:58:36.860478 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:58:36.860444    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:58:38.863101 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:38.863065    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:39.592726 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:39.592692    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:40.441928 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:40.441897    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:40.447155 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:40.447128    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"]
Nov 03 17:58:40.625656 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:40.625623    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:41.202241 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:41.202207    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:43.894895 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:43.894840    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:44.346227 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:44.346191    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:45.448120 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:45.448079    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg" podUID="f5a452c7-51f0-4c5a-b092-d0507af37bb7" containerName="payload-processing" containerID="cri-o://d68f73640f2b68e1e63a336489b25df5b0b839b2a2019ed26cbb6c08d0765183" gracePeriod=25
Nov 03 17:58:45.699667 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:45.699601    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"
Nov 03 17:58:45.794978 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:45.794937    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials\") pod \"f5a452c7-51f0-4c5a-b092-d0507af37bb7\" (UID: \"f5a452c7-51f0-4c5a-b092-d0507af37bb7\") "
Nov 03 17:58:45.795201 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:45.795009    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-kube-api-access-72mhl\" (UniqueName: \"kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-kube-api-access-72mhl\") pod \"f5a452c7-51f0-4c5a-b092-d0507af37bb7\" (UID: \"f5a452c7-51f0-4c5a-b092-d0507af37bb7\") "
Nov 03 17:58:45.795201 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:45.795099    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5a452c7-51f0-4c5a-b092-d0507af37bb7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5a452c7-51f0-4c5a-b092-d0507af37bb7-routing-overlay-volume\") pod \"f5a452c7-51f0-4c5a-b092-d0507af37bb7\" (UID: \"f5a452c7-51f0-4c5a-b092-d0507af37bb7\") "
Nov 03 17:58:45.795201 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:45.795170    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5a452c7-51f0-4c5a-b092-d0507af37bb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5a452c7-51f0-4c5a-b092-d0507af37bb7-plugins-config-volume\") pod \"f5a452c7-51f0-4c5a-b092-d0507af37bb7\" (UID: \"f5a452c7-51f0-4c5a-b092-d0507af37bb7\") "
Nov 03 17:58:45.795567 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:45.795537    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5a452c7-51f0-4c5a-b092-d0507af37bb7-routing-overlay-volume" pod "f5a452c7-51f0-4c5a-b092-d0507af37bb7" (UID: "f5a452c7-51f0-4c5a-b092-d0507af37bb7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:58:45.795671 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:45.795650    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5a452c7-51f0-4c5a-b092-d0507af37bb7-plugins-config-volume" pod "f5a452c7-51f0-4c5a-b092-d0507af37bb7" (UID: "f5a452c7-51f0-4c5a-b092-d0507af37bb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:58:45.797292 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:45.797267    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-kube-api-access-72mhl" pod "f5a452c7-51f0-4c5a-b092-d0507af37bb7" (UID: "f5a452c7-51f0-4c5a-b092-d0507af37bb7"). InnerVolumeSpecName "kube-api-access-72mhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:58:45.797292 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:45.797281    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials" pod "f5a452c7-51f0-4c5a-b092-d0507af37bb7" (UID: "f5a452c7-51f0-4c5a-b092-d0507af37bb7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:58:45.897120 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:45.897079    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5a452c7-51f0-4c5a-b092-d0507af37bb7-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:58:45.897120 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:45.897111    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5a452c7-51f0-4c5a-b092-d0507af37bb7-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:58:45.897120 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:45.897124    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:58:45.897408 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:45.897136    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72mhl\" (UniqueName: \"kubernetes.io/projected/f5a452c7-51f0-4c5a-b092-d0507af37bb7-kube-api-access-72mhl\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:58:45.965247 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:45.965204    6059 generic.go:356] "Generic (PLEG): container finished" podID="f5a452c7-51f0-4c5a-b092-d0507af37bb7" containerID="d68f73640f2b68e1e63a336489b25df5b0b839b2a2019ed26cbb6c08d0765183" exitCode=0
Nov 03 17:58:45.965480 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:45.965255    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"
Nov 03 17:58:45.965480 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:45.965278    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg" event={"ID":"f5a452c7-51f0-4c5a-b092-d0507af37bb7","Type":"ContainerDied","Data":"d68f73640f2b68e1e63a336489b25df5b0b839b2a2019ed26cbb6c08d0765183"}
Nov 03 17:58:45.965480 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:45.965327    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg" event={"ID":"f5a452c7-51f0-4c5a-b092-d0507af37bb7","Type":"ContainerDied","Data":"7d554549ef510f14085f2c581895fb2edabded43ab7492cb7415b824c17c2c01"}
Nov 03 17:58:45.965480 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:45.965352    6059 scope.go:122] "RemoveContainer" containerID="d68f73640f2b68e1e63a336489b25df5b0b839b2a2019ed26cbb6c08d0765183"
Nov 03 17:58:45.975487 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:45.975465    6059 scope.go:122] "RemoveContainer" containerID="d68f73640f2b68e1e63a336489b25df5b0b839b2a2019ed26cbb6c08d0765183"
Nov 03 17:58:45.975816 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:58:45.975795    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d68f73640f2b68e1e63a336489b25df5b0b839b2a2019ed26cbb6c08d0765183\": container with ID starting with d68f73640f2b68e1e63a336489b25df5b0b839b2a2019ed26cbb6c08d0765183 not found: ID does not exist" containerID="d68f73640f2b68e1e63a336489b25df5b0b839b2a2019ed26cbb6c08d0765183"
Nov 03 17:58:45.975887 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:45.975826    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d68f73640f2b68e1e63a336489b25df5b0b839b2a2019ed26cbb6c08d0765183"} err="failed to get container status \"d68f73640f2b68e1e63a336489b25df5b0b839b2a2019ed26cbb6c08d0765183\": rpc error: code = NotFound desc = could not find container \"d68f73640f2b68e1e63a336489b25df5b0b839b2a2019ed26cbb6c08d0765183\": container with ID starting with d68f73640f2b68e1e63a336489b25df5b0b839b2a2019ed26cbb6c08d0765183 not found: ID does not exist"
Nov 03 17:58:46.134847 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:46.134807    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"]
Nov 03 17:58:46.139151 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:46.139114    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-5949x6xzg"]
Nov 03 17:58:46.976071 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:46.976035    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:47.337264 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:47.337185    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5a452c7-51f0-4c5a-b092-d0507af37bb7" path="/var/lib/kubelet/pods/f5a452c7-51f0-4c5a-b092-d0507af37bb7/volumes"
Nov 03 17:58:47.596949 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:47.596853    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:47.776808 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:47.776774    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:48.980574 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:48.980535    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:53.616108 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:53.616073    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:53.804169 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:53.804129    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:53.992979 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:53.992944    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:54.350682 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:54.350587    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:54.852809 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:54.852777    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:56.897496 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:56.897454    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"]
Nov 03 17:58:56.897921 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:56.897737    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c" podUID="4f0c42da-8020-48d1-9e61-7dfab0d4a65b" containerName="istio-proxy" containerID="cri-o://e8f193522683ef5e8fe2dca93fac72da4f903f4eac63caf3cdd51b4e3c62698f" gracePeriod=30
Nov 03 17:58:57.288617 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:57.288578    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:58.806797 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:58.806757    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:58:59.296074 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:59.296033    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-tdr2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:58:59.296281 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:58:59.296117    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c" podUID="4f0c42da-8020-48d1-9e61-7dfab0d4a65b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:59:02.073170 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.073136    6059 generic.go:356] "Generic (PLEG): container finished" podID="4f0c42da-8020-48d1-9e61-7dfab0d4a65b" containerID="e8f193522683ef5e8fe2dca93fac72da4f903f4eac63caf3cdd51b4e3c62698f" exitCode=0
Nov 03 17:59:02.073635 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.073196    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c" event={"ID":"4f0c42da-8020-48d1-9e61-7dfab0d4a65b","Type":"ContainerDied","Data":"e8f193522683ef5e8fe2dca93fac72da4f903f4eac63caf3cdd51b4e3c62698f"}
Nov 03 17:59:02.164486 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.164457    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:59:02.284027 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.283991    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-credential-socket\") pod \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") "
Nov 03 17:59:02.284222 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.284051    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-workload-socket\") pod \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") "
Nov 03 17:59:02.284222 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.284129    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-kube-api-access-g2tgf\" (UniqueName: \"kubernetes.io/projected/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-kube-api-access-g2tgf\") pod \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") "
Nov 03 17:59:02.284222 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.284158    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istiod-ca-cert\") pod \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") "
Nov 03 17:59:02.284222 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.284203    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-podinfo\") pod \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") "
Nov 03 17:59:02.284537 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.284237    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-data\") pod \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") "
Nov 03 17:59:02.284537 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.284266    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-workload-certs\") pod \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") "
Nov 03 17:59:02.284537 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.284296    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-token\" (UniqueName: \"kubernetes.io/projected/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-token\") pod \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") "
Nov 03 17:59:02.284537 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.284291    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-credential-socket" pod "4f0c42da-8020-48d1-9e61-7dfab0d4a65b" (UID: "4f0c42da-8020-48d1-9e61-7dfab0d4a65b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:59:02.284537 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.284347    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-envoy\") pod \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\" (UID: \"4f0c42da-8020-48d1-9e61-7dfab0d4a65b\") "
Nov 03 17:59:02.284537 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.284404    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-workload-socket" pod "4f0c42da-8020-48d1-9e61-7dfab0d4a65b" (UID: "4f0c42da-8020-48d1-9e61-7dfab0d4a65b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:59:02.284818 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.284625    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-workload-certs" pod "4f0c42da-8020-48d1-9e61-7dfab0d4a65b" (UID: "4f0c42da-8020-48d1-9e61-7dfab0d4a65b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:59:02.284818 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.284634    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istiod-ca-cert" pod "4f0c42da-8020-48d1-9e61-7dfab0d4a65b" (UID: "4f0c42da-8020-48d1-9e61-7dfab0d4a65b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:59:02.284818 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.284639    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-data" pod "4f0c42da-8020-48d1-9e61-7dfab0d4a65b" (UID: "4f0c42da-8020-48d1-9e61-7dfab0d4a65b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:59:02.284951 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.284862    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:02.284951 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.284886    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:02.284951 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.284901    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:02.284951 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.284917    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:02.284951 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.284932    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:02.286561 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.286519    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-podinfo" pod "4f0c42da-8020-48d1-9e61-7dfab0d4a65b" (UID: "4f0c42da-8020-48d1-9e61-7dfab0d4a65b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:59:02.286691 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.286588    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-token" pod "4f0c42da-8020-48d1-9e61-7dfab0d4a65b" (UID: "4f0c42da-8020-48d1-9e61-7dfab0d4a65b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:59:02.286691 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.286585    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-kube-api-access-g2tgf" pod "4f0c42da-8020-48d1-9e61-7dfab0d4a65b" (UID: "4f0c42da-8020-48d1-9e61-7dfab0d4a65b"). InnerVolumeSpecName "kube-api-access-g2tgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:59:02.286691 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.286682    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-envoy" pod "4f0c42da-8020-48d1-9e61-7dfab0d4a65b" (UID: "4f0c42da-8020-48d1-9e61-7dfab0d4a65b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:59:02.386319 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.386263    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2tgf\" (UniqueName: \"kubernetes.io/projected/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-kube-api-access-g2tgf\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:02.386319 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.386314    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:02.386319 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.386329    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:02.386614 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:02.386342    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f0c42da-8020-48d1-9e61-7dfab0d4a65b-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:03.082270 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:03.082233    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"
Nov 03 17:59:03.082270 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:03.082250    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c" event={"ID":"4f0c42da-8020-48d1-9e61-7dfab0d4a65b","Type":"ContainerDied","Data":"e42a294c379042500651251ce4d59a24fdad6e7b0cefc387e4c725437f59e906"}
Nov 03 17:59:03.082796 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:03.082312    6059 scope.go:122] "RemoveContainer" containerID="e8f193522683ef5e8fe2dca93fac72da4f903f4eac63caf3cdd51b4e3c62698f"
Nov 03 17:59:03.306451 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:03.306415    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"]
Nov 03 17:59:03.358393 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:03.358298    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-tdr2c"]
Nov 03 17:59:03.686806 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:03.686732    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:03.868856 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:03.868822    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:05.335556 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:05.335521    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f0c42da-8020-48d1-9e61-7dfab0d4a65b" path="/var/lib/kubelet/pods/4f0c42da-8020-48d1-9e61-7dfab0d4a65b/volumes"
Nov 03 17:59:06.862496 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:59:06.862457    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:59:08.669309 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:08.669224    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:12.023011 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:12.022977    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"]
Nov 03 17:59:12.023454 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:12.023193    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq" podUID="1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" containerName="istio-proxy" containerID="cri-o://79d854461eb1c0342cd39749425b53ac99d6c825a0e245f4d35eabfefb4ce24c" gracePeriod=30
Nov 03 17:59:13.555127 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:13.555089    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:14.287703 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:14.287665    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-h98kq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:59:14.287879 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:14.287739    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq" podUID="1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:59:17.166803 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.166773    6059 generic.go:356] "Generic (PLEG): container finished" podID="1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" containerID="79d854461eb1c0342cd39749425b53ac99d6c825a0e245f4d35eabfefb4ce24c" exitCode=0
Nov 03 17:59:17.167191 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.166840    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq" event={"ID":"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc","Type":"ContainerDied","Data":"79d854461eb1c0342cd39749425b53ac99d6c825a0e245f4d35eabfefb4ce24c"}
Nov 03 17:59:17.288798 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.288767    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:59:17.459971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.459934    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-credential-socket\") pod \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") "
Nov 03 17:59:17.459971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.459981    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-data\") pod \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") "
Nov 03 17:59:17.460244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.460048    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istiod-ca-cert\") pod \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") "
Nov 03 17:59:17.460244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.460073    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-token\" (UniqueName: \"kubernetes.io/projected/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-token\") pod \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") "
Nov 03 17:59:17.460244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.460111    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-podinfo\") pod \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") "
Nov 03 17:59:17.460244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.460136    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-envoy\") pod \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") "
Nov 03 17:59:17.460244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.460201    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-workload-socket\") pod \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") "
Nov 03 17:59:17.460244 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.460240    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-workload-certs\") pod \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") "
Nov 03 17:59:17.460625 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.460279    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-kube-api-access-6t6nt\" (UniqueName: \"kubernetes.io/projected/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-kube-api-access-6t6nt\") pod \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\" (UID: \"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc\") "
Nov 03 17:59:17.460625 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.460312    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-credential-socket" pod "1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" (UID: "1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:59:17.460625 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.460429    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-data" pod "1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" (UID: "1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:59:17.460625 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.460542    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-workload-socket" pod "1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" (UID: "1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:59:17.460625 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.460567    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istiod-ca-cert" pod "1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" (UID: "1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:59:17.460625 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.460592    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-workload-certs" pod "1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" (UID: "1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:59:17.460885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.460706    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:17.460885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.460728    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:17.460885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.460742    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:17.460885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.460756    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:17.460885 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.460771    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:17.462777 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.462747    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-envoy" pod "1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" (UID: "1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:59:17.462884 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.462747    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-podinfo" pod "1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" (UID: "1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:59:17.463084 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.463062    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-kube-api-access-6t6nt" pod "1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" (UID: "1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc"). InnerVolumeSpecName "kube-api-access-6t6nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:59:17.463186 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.463093    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-token" pod "1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" (UID: "1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:59:17.561986 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.561939    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:17.561986 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.561977    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:17.561986 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.561991    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:17.562246 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:17.562004    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t6nt\" (UniqueName: \"kubernetes.io/projected/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc-kube-api-access-6t6nt\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:18.173942 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:18.173909    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq" event={"ID":"1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc","Type":"ContainerDied","Data":"0826ad0c2e5c4df44973bc4d890c830db0282836a50af15c6c3119e3f181efa5"}
Nov 03 17:59:18.174386 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:18.173970    6059 scope.go:122] "RemoveContainer" containerID="79d854461eb1c0342cd39749425b53ac99d6c825a0e245f4d35eabfefb4ce24c"
Nov 03 17:59:18.174386 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:18.173920    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"
Nov 03 17:59:18.346036 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:18.346002    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"]
Nov 03 17:59:18.352250 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:18.352222    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-h98kq"]
Nov 03 17:59:18.480256 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:18.480227    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:19.336628 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:19.336580    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" path="/var/lib/kubelet/pods/1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc/volumes"
Nov 03 17:59:23.369673 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:23.369632    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:28.266909 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:28.266877    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:33.176764 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:33.176727    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:34.187934 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:34.187897    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"]
Nov 03 17:59:34.188412 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:34.188130    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49" podUID="4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7" containerName="istio-proxy" containerID="cri-o://92666eedae1e55cd91bd4bb8aed9d30fdfc64803b6ab9f9449ca95574f93fb45" gracePeriod=30
Nov 03 17:59:36.862107 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:59:36.862070    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 17:59:38.185092 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:38.185051    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:39.288770 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.288730    6059 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 03 17:59:39.289160 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.288786    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49" podUID="4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 03 17:59:39.448844 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.448809    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:59:39.595443 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.595404    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-workload-socket\") pod \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") "
Nov 03 17:59:39.595443 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.595441    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-envoy\") pod \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") "
Nov 03 17:59:39.595709 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.595463    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-podinfo\") pod \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") "
Nov 03 17:59:39.595709 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.595541    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-credential-socket\") pod \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") "
Nov 03 17:59:39.595709 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.595577    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-data\") pod \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") "
Nov 03 17:59:39.595709 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.595600    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-workload-certs\") pod \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") "
Nov 03 17:59:39.595931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.595713    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istiod-ca-cert\") pod \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") "
Nov 03 17:59:39.595931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.595750    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-token\" (UniqueName: \"kubernetes.io/projected/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-token\") pod \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") "
Nov 03 17:59:39.595931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.595752    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-workload-socket" pod "4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7" (UID: "4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:59:39.595931 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.595790    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-kube-api-access-2f44k\" (UniqueName: \"kubernetes.io/projected/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-kube-api-access-2f44k\") pod \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\" (UID: \"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7\") "
Nov 03 17:59:39.596132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.595984    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-workload-certs" pod "4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7" (UID: "4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:59:39.596132 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.596006    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-credential-socket" pod "4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7" (UID: "4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:59:39.596238 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.596203    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istiod-ca-cert" pod "4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7" (UID: "4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:59:39.596303 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.596287    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:39.596374 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.596310    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:39.596441 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.596326    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:39.596511 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.596488    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-data" pod "4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7" (UID: "4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:59:39.597826 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.597794    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-podinfo" pod "4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7" (UID: "4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:59:39.597826 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.597812    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-envoy" pod "4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7" (UID: "4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:59:39.598009 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.597877    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-token" pod "4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7" (UID: "4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:59:39.598009 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.597973    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-kube-api-access-2f44k" pod "4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7" (UID: "4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7"). InnerVolumeSpecName "kube-api-access-2f44k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:59:39.697366 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.697330    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:39.697554 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.697391    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:39.697554 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.697404    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f44k\" (UniqueName: \"kubernetes.io/projected/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-kube-api-access-2f44k\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:39.697554 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.697414    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:39.697554 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.697423    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:39.697554 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:39.697431    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:59:40.301210 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:40.301174    6059 generic.go:356] "Generic (PLEG): container finished" podID="4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7" containerID="92666eedae1e55cd91bd4bb8aed9d30fdfc64803b6ab9f9449ca95574f93fb45" exitCode=0
Nov 03 17:59:40.301715 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:40.301218    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"
Nov 03 17:59:40.301715 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:40.301223    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49" event={"ID":"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7","Type":"ContainerDied","Data":"92666eedae1e55cd91bd4bb8aed9d30fdfc64803b6ab9f9449ca95574f93fb45"}
Nov 03 17:59:40.301715 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:40.301396    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49" event={"ID":"4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7","Type":"ContainerDied","Data":"c890b89dab0e30bc096c9e85b1accf3d26d6b55d595a59e4f39788a9c050881e"}
Nov 03 17:59:40.301715 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:40.301440    6059 scope.go:122] "RemoveContainer" containerID="92666eedae1e55cd91bd4bb8aed9d30fdfc64803b6ab9f9449ca95574f93fb45"
Nov 03 17:59:40.311616 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:40.311491    6059 scope.go:122] "RemoveContainer" containerID="92666eedae1e55cd91bd4bb8aed9d30fdfc64803b6ab9f9449ca95574f93fb45"
Nov 03 17:59:40.311791 ip-10-0-143-117 kubenswrapper[6059]: E1103 17:59:40.311773    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92666eedae1e55cd91bd4bb8aed9d30fdfc64803b6ab9f9449ca95574f93fb45\": container with ID starting with 92666eedae1e55cd91bd4bb8aed9d30fdfc64803b6ab9f9449ca95574f93fb45 not found: ID does not exist" containerID="92666eedae1e55cd91bd4bb8aed9d30fdfc64803b6ab9f9449ca95574f93fb45"
Nov 03 17:59:40.311841 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:40.311800    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92666eedae1e55cd91bd4bb8aed9d30fdfc64803b6ab9f9449ca95574f93fb45"} err="failed to get container status \"92666eedae1e55cd91bd4bb8aed9d30fdfc64803b6ab9f9449ca95574f93fb45\": rpc error: code = NotFound desc = could not find container \"92666eedae1e55cd91bd4bb8aed9d30fdfc64803b6ab9f9449ca95574f93fb45\": container with ID starting with 92666eedae1e55cd91bd4bb8aed9d30fdfc64803b6ab9f9449ca95574f93fb45 not found: ID does not exist"
Nov 03 17:59:40.500595 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:40.500558    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"]
Nov 03 17:59:40.552604 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:40.552516    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-p8h49"]
Nov 03 17:59:41.334821 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:41.334777    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7" path="/var/lib/kubelet/pods/4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7/volumes"
Nov 03 17:59:42.632517 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:42.632485    6059 scope.go:122] "RemoveContainer" containerID="e5e4e928d461b5416aefbb77e69c6d38bca14c28a9b14d99493ac25150ff223a"
Nov 03 17:59:42.644320 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:42.643577    6059 scope.go:122] "RemoveContainer" containerID="77bc15dd72351409a52172bf1b73326d5d3c2fccf538d651c19181493dc475de"
Nov 03 17:59:43.064699 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:43.064664    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:44.702049 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:44.702010    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:47.857846 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:47.857762    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:48.044127 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:48.044096    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:48.274302 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:48.274265    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:48.585712 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:48.585627    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:48.889052 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:48.888969    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:49.071904 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:49.071869    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:49.273778 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:49.273723    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:49.478797 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:49.478756    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:50.328898 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:50.328865    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:50.522150 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:50.522120    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:52.967067 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:52.967034    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:53.141873 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:53.141836    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:53.492315 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:53.492270    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:54.088882 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:54.088836    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:54.269119 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:54.269086    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:54.951971 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:54.951933    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 17:59:55.430024 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.429978    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"]
Nov 03 17:59:55.430523 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430479    6059 cpu_manager.go:414] "containerMap: removing container" podUID="945d8e0f-a47e-49e3-ae35-d25ad12aca81" containerName="istio-proxy"
Nov 03 17:59:55.430523 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430492    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945d8e0f-a47e-49e3-ae35-d25ad12aca81" containerName="istio-proxy"
Nov 03 17:59:55.430523 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430502    6059 cpu_manager.go:414] "containerMap: removing container" podUID="4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7" containerName="istio-proxy"
Nov 03 17:59:55.430523 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430508    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7" containerName="istio-proxy"
Nov 03 17:59:55.430523 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430519    6059 cpu_manager.go:414] "containerMap: removing container" podUID="1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" containerName="istio-proxy"
Nov 03 17:59:55.430772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430528    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" containerName="istio-proxy"
Nov 03 17:59:55.430772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430538    6059 cpu_manager.go:414] "containerMap: removing container" podUID="d2e5b0a4-8ae5-41d3-97b4-a13804480f77" containerName="payload-pre-processing"
Nov 03 17:59:55.430772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430544    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2e5b0a4-8ae5-41d3-97b4-a13804480f77" containerName="payload-pre-processing"
Nov 03 17:59:55.430772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430551    6059 cpu_manager.go:414] "containerMap: removing container" podUID="6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e" containerName="payload-processing"
Nov 03 17:59:55.430772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430555    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e" containerName="payload-processing"
Nov 03 17:59:55.430772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430562    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f5a452c7-51f0-4c5a-b092-d0507af37bb7" containerName="payload-processing"
Nov 03 17:59:55.430772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430566    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5a452c7-51f0-4c5a-b092-d0507af37bb7" containerName="payload-processing"
Nov 03 17:59:55.430772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430582    6059 cpu_manager.go:414] "containerMap: removing container" podUID="8e50ad7e-f804-40ed-b806-b2c904dc96cd" containerName="payload-processing"
Nov 03 17:59:55.430772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430587    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e50ad7e-f804-40ed-b806-b2c904dc96cd" containerName="payload-processing"
Nov 03 17:59:55.430772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430594    6059 cpu_manager.go:414] "containerMap: removing container" podUID="4f0c42da-8020-48d1-9e61-7dfab0d4a65b" containerName="istio-proxy"
Nov 03 17:59:55.430772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430603    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f0c42da-8020-48d1-9e61-7dfab0d4a65b" containerName="istio-proxy"
Nov 03 17:59:55.430772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430618    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f787e049-8746-4ab8-83f0-d80a63fd3465" containerName="payload-pre-processing"
Nov 03 17:59:55.430772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430623    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f787e049-8746-4ab8-83f0-d80a63fd3465" containerName="payload-pre-processing"
Nov 03 17:59:55.430772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430633    6059 cpu_manager.go:414] "containerMap: removing container" podUID="012064a7-fd12-465b-bdc8-ef6d4ca289b8" containerName="istio-proxy"
Nov 03 17:59:55.430772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430640    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012064a7-fd12-465b-bdc8-ef6d4ca289b8" containerName="istio-proxy"
Nov 03 17:59:55.430772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430670    6059 cpu_manager.go:414] "containerMap: removing container" podUID="fb73f8e5-cce4-4bca-9e84-717537d8a3e0" containerName="istio-proxy"
Nov 03 17:59:55.430772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430676    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb73f8e5-cce4-4bca-9e84-717537d8a3e0" containerName="istio-proxy"
Nov 03 17:59:55.430772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430755    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e50ad7e-f804-40ed-b806-b2c904dc96cd" containerName="payload-processing"
Nov 03 17:59:55.430772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430769    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f0c42da-8020-48d1-9e61-7dfab0d4a65b" containerName="istio-proxy"
Nov 03 17:59:55.430772 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430779    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f787e049-8746-4ab8-83f0-d80a63fd3465" containerName="payload-pre-processing"
Nov 03 17:59:55.431397 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430785    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="012064a7-fd12-465b-bdc8-ef6d4ca289b8" containerName="istio-proxy"
Nov 03 17:59:55.431397 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430792    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="945d8e0f-a47e-49e3-ae35-d25ad12aca81" containerName="istio-proxy"
Nov 03 17:59:55.431397 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430801    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb73f8e5-cce4-4bca-9e84-717537d8a3e0" containerName="istio-proxy"
Nov 03 17:59:55.431397 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430807    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="945d8e0f-a47e-49e3-ae35-d25ad12aca81" containerName="istio-proxy"
Nov 03 17:59:55.431397 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430813    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4a3d51-7e8f-4bac-a514-dd4072dbbdb7" containerName="istio-proxy"
Nov 03 17:59:55.431397 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430817    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7a4f1d-3c17-41f8-b8a2-b97d039f2fbc" containerName="istio-proxy"
Nov 03 17:59:55.431397 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430821    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2e5b0a4-8ae5-41d3-97b4-a13804480f77" containerName="payload-pre-processing"
Nov 03 17:59:55.431397 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430825    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e20d0fa-0c1d-44d5-bdbc-fec8c0abcc3e" containerName="payload-processing"
Nov 03 17:59:55.431397 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.430831    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5a452c7-51f0-4c5a-b092-d0507af37bb7" containerName="payload-processing"
Nov 03 17:59:55.436138 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.436114    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"
Nov 03 17:59:55.473528 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.473491    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec9e0516-c76e-4d72-9ff0-1e416429e93f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd\" (UID: \"ec9e0516-c76e-4d72-9ff0-1e416429e93f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"
Nov 03 17:59:55.473725 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.473546    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9e0516-c76e-4d72-9ff0-1e416429e93f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd\" (UID: \"ec9e0516-c76e-4d72-9ff0-1e416429e93f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"
Nov 03 17:59:55.473725 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.473642    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzzcb\" (UniqueName: \"kubernetes.io/projected/ec9e0516-c76e-4d72-9ff0-1e416429e93f-kube-api-access-nzzcb\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd\" (UID: \"ec9e0516-c76e-4d72-9ff0-1e416429e93f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"
Nov 03 17:59:55.473725 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.473708    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec9e0516-c76e-4d72-9ff0-1e416429e93f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd\" (UID: \"ec9e0516-c76e-4d72-9ff0-1e416429e93f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"
Nov 03 17:59:55.543639 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.543606    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"openshift-service-ca.crt\""
Nov 03 17:59:55.543639 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.543636    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"routing-overlay\""
Nov 03 17:59:55.543901 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.543669    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"e2e-cfl-d35f0c-provider-cred\""
Nov 03 17:59:55.543901 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.543673    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-029ca0\""
Nov 03 17:59:55.543901 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.543733    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"payload-processing-external-model-e2e--58d254fa-dockercfg-zmxcc\""
Nov 03 17:59:55.543901 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.543778    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"kube-root-ca.crt\""
Nov 03 17:59:55.545561 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.545540    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"]
Nov 03 17:59:55.574503 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.574463    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9e0516-c76e-4d72-9ff0-1e416429e93f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd\" (UID: \"ec9e0516-c76e-4d72-9ff0-1e416429e93f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"
Nov 03 17:59:55.574690 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.574520    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzzcb\" (UniqueName: \"kubernetes.io/projected/ec9e0516-c76e-4d72-9ff0-1e416429e93f-kube-api-access-nzzcb\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd\" (UID: \"ec9e0516-c76e-4d72-9ff0-1e416429e93f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"
Nov 03 17:59:55.574690 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.574585    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec9e0516-c76e-4d72-9ff0-1e416429e93f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd\" (UID: \"ec9e0516-c76e-4d72-9ff0-1e416429e93f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"
Nov 03 17:59:55.574690 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.574635    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec9e0516-c76e-4d72-9ff0-1e416429e93f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd\" (UID: \"ec9e0516-c76e-4d72-9ff0-1e416429e93f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"
Nov 03 17:59:55.575271 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.575242    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9e0516-c76e-4d72-9ff0-1e416429e93f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd\" (UID: \"ec9e0516-c76e-4d72-9ff0-1e416429e93f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"
Nov 03 17:59:55.575376 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.575248    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec9e0516-c76e-4d72-9ff0-1e416429e93f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd\" (UID: \"ec9e0516-c76e-4d72-9ff0-1e416429e93f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"
Nov 03 17:59:55.577041 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.577020    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec9e0516-c76e-4d72-9ff0-1e416429e93f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd\" (UID: \"ec9e0516-c76e-4d72-9ff0-1e416429e93f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"
Nov 03 17:59:55.638212 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.638176    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzzcb\" (UniqueName: \"kubernetes.io/projected/ec9e0516-c76e-4d72-9ff0-1e416429e93f-kube-api-access-nzzcb\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd\" (UID: \"ec9e0516-c76e-4d72-9ff0-1e416429e93f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"
Nov 03 17:59:55.749290 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:55.749249    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"
Nov 03 17:59:56.144581 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:56.144433    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"]
Nov 03 17:59:56.147445 ip-10-0-143-117 kubenswrapper[6059]: W1103 17:59:56.147406    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9e0516_c76e_4d72_9ff0_1e416429e93f.slice/crio-97ca6302e01f185166caeef933ff560aacda70eded05125305d7dff5991c9d97 WatchSource:0}: Error finding container 97ca6302e01f185166caeef933ff560aacda70eded05125305d7dff5991c9d97: Status 404 returned error can't find the container with id 97ca6302e01f185166caeef933ff560aacda70eded05125305d7dff5991c9d97
Nov 03 17:59:56.393577 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:56.393538    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd" event={"ID":"ec9e0516-c76e-4d72-9ff0-1e416429e93f","Type":"ContainerStarted","Data":"4074faa78bea593304c8379cb7ce0629a63a846836a7235552c5d58c874400e2"}
Nov 03 17:59:56.393729 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:56.393601    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"
Nov 03 17:59:56.393729 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:56.393619    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd" event={"ID":"ec9e0516-c76e-4d72-9ff0-1e416429e93f","Type":"ContainerStarted","Data":"97ca6302e01f185166caeef933ff560aacda70eded05125305d7dff5991c9d97"}
Nov 03 17:59:56.514399 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:56.514322    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd" podStartSLOduration=1.514306671 podStartE2EDuration="1.514306671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:59:56.513593537 +0000 UTC m=+2537.311306320" watchObservedRunningTime="2026-11-03 17:59:56.514306671 +0000 UTC m=+2537.312019452"
Nov 03 17:59:57.951824 ip-10-0-143-117 kubenswrapper[6059]: I1103 17:59:57.951784    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:00.166280 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:00.166234    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt"]
Nov 03 18:00:00.166933 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:00.166909    6059 cpu_manager.go:414] "containerMap: removing container" podUID="012064a7-fd12-465b-bdc8-ef6d4ca289b8" containerName="istio-proxy"
Nov 03 18:00:00.167021 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:00.166936    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012064a7-fd12-465b-bdc8-ef6d4ca289b8" containerName="istio-proxy"
Nov 03 18:00:00.167021 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:00.166982    6059 cpu_manager.go:414] "containerMap: removing container" podUID="945d8e0f-a47e-49e3-ae35-d25ad12aca81" containerName="istio-proxy"
Nov 03 18:00:00.167021 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:00.166990    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945d8e0f-a47e-49e3-ae35-d25ad12aca81" containerName="istio-proxy"
Nov 03 18:00:00.167182 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:00.167091    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="012064a7-fd12-465b-bdc8-ef6d4ca289b8" containerName="istio-proxy"
Nov 03 18:00:00.170615 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:00.170594    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt"
Nov 03 18:00:00.219938 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:00.219899    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0cb266-2071-4abc-9c89-fa5ff6601f6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895480-t78nt\" (UID: \"ca0cb266-2071-4abc-9c89-fa5ff6601f6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt"
Nov 03 18:00:00.219938 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:00.219942    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndklp\" (UniqueName: \"kubernetes.io/projected/ca0cb266-2071-4abc-9c89-fa5ff6601f6b-kube-api-access-ndklp\") pod \"maas-api-key-cleanup-29895480-t78nt\" (UID: \"ca0cb266-2071-4abc-9c89-fa5ff6601f6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt"
Nov 03 18:00:00.224628 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:00.224595    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:00:00.224817 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:00.224802    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:00:00.276807 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:00.276777    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 18:00:00.278656 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:00.278630    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt"]
Nov 03 18:00:00.321265 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:00.321217    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0cb266-2071-4abc-9c89-fa5ff6601f6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895480-t78nt\" (UID: \"ca0cb266-2071-4abc-9c89-fa5ff6601f6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt"
Nov 03 18:00:00.321265 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:00.321268    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndklp\" (UniqueName: \"kubernetes.io/projected/ca0cb266-2071-4abc-9c89-fa5ff6601f6b-kube-api-access-ndklp\") pod \"maas-api-key-cleanup-29895480-t78nt\" (UID: \"ca0cb266-2071-4abc-9c89-fa5ff6601f6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt"
Nov 03 18:00:00.322049 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:00.322028    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0cb266-2071-4abc-9c89-fa5ff6601f6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895480-t78nt\" (UID: \"ca0cb266-2071-4abc-9c89-fa5ff6601f6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt"
Nov 03 18:00:00.386143 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:00.386106    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndklp\" (UniqueName: \"kubernetes.io/projected/ca0cb266-2071-4abc-9c89-fa5ff6601f6b-kube-api-access-ndklp\") pod \"maas-api-key-cleanup-29895480-t78nt\" (UID: \"ca0cb266-2071-4abc-9c89-fa5ff6601f6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt"
Nov 03 18:00:00.484065 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:00.484027    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt"
Nov 03 18:00:00.515977 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:00.515938    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:00.691094 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:00.691058    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:00.868335 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:00.868253    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt"]
Nov 03 18:00:00.871187 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:00:00.871149    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0cb266_2071_4abc_9c89_fa5ff6601f6b.slice/crio-a657eb4b2d76c455d8fe8ebd55a93ea3ae76e4242d78b727373c52cd83b7be04 WatchSource:0}: Error finding container a657eb4b2d76c455d8fe8ebd55a93ea3ae76e4242d78b727373c52cd83b7be04: Status 404 returned error can't find the container with id a657eb4b2d76c455d8fe8ebd55a93ea3ae76e4242d78b727373c52cd83b7be04
Nov 03 18:00:01.429089 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:01.429056    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt" event={"ID":"ca0cb266-2071-4abc-9c89-fa5ff6601f6b","Type":"ContainerStarted","Data":"08e123759f20e028e193cc17b42921b7c9a927897c529fff98a3a9c9c1975d50"}
Nov 03 18:00:01.429502 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:01.429110    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt" event={"ID":"ca0cb266-2071-4abc-9c89-fa5ff6601f6b","Type":"ContainerStarted","Data":"a657eb4b2d76c455d8fe8ebd55a93ea3ae76e4242d78b727373c52cd83b7be04"}
Nov 03 18:00:01.548812 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:01.548705    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt" podStartSLOduration=1.5486864900000001 podStartE2EDuration="1.54868649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:01.547759044 +0000 UTC m=+2542.345471827" watchObservedRunningTime="2026-11-03 18:00:01.54868649 +0000 UTC m=+2542.346399279"
Nov 03 18:00:02.437302 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:02.437268    6059 generic.go:356] "Generic (PLEG): container finished" podID="ca0cb266-2071-4abc-9c89-fa5ff6601f6b" containerID="08e123759f20e028e193cc17b42921b7c9a927897c529fff98a3a9c9c1975d50" exitCode=60
Nov 03 18:00:02.437802 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:02.437323    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt" event={"ID":"ca0cb266-2071-4abc-9c89-fa5ff6601f6b","Type":"ContainerDied","Data":"08e123759f20e028e193cc17b42921b7c9a927897c529fff98a3a9c9c1975d50"}
Nov 03 18:00:02.437802 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:02.437633    6059 scope.go:122] "RemoveContainer" containerID="08e123759f20e028e193cc17b42921b7c9a927897c529fff98a3a9c9c1975d50"
Nov 03 18:00:02.940481 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:02.940450    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:03.445976 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:03.445888    6059 generic.go:356] "Generic (PLEG): container finished" podID="ca0cb266-2071-4abc-9c89-fa5ff6601f6b" containerID="325e0f1e34551a6ed6ba86ba3815473209b8c8fcedfa01380732b74b3e8ac894" exitCode=60
Nov 03 18:00:03.446427 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:03.445968    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt" event={"ID":"ca0cb266-2071-4abc-9c89-fa5ff6601f6b","Type":"ContainerDied","Data":"325e0f1e34551a6ed6ba86ba3815473209b8c8fcedfa01380732b74b3e8ac894"}
Nov 03 18:00:03.446427 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:03.446026    6059 scope.go:122] "RemoveContainer" containerID="08e123759f20e028e193cc17b42921b7c9a927897c529fff98a3a9c9c1975d50"
Nov 03 18:00:03.446427 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:03.446339    6059 scope.go:122] "RemoveContainer" containerID="325e0f1e34551a6ed6ba86ba3815473209b8c8fcedfa01380732b74b3e8ac894"
Nov 03 18:00:03.446685 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:00:03.446666    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895480-t78nt_odh-ai-gateway-infra(ca0cb266-2071-4abc-9c89-fa5ff6601f6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt" podUID="ca0cb266-2071-4abc-9c89-fa5ff6601f6b"
Nov 03 18:00:04.453810 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:04.453776    6059 scope.go:122] "RemoveContainer" containerID="325e0f1e34551a6ed6ba86ba3815473209b8c8fcedfa01380732b74b3e8ac894"
Nov 03 18:00:04.454310 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:00:04.454052    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895480-t78nt_odh-ai-gateway-infra(ca0cb266-2071-4abc-9c89-fa5ff6601f6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt" podUID="ca0cb266-2071-4abc-9c89-fa5ff6601f6b"
Nov 03 18:00:06.861322 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:00:06.861283    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:00:07.409845 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:07.409817    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"
Nov 03 18:00:07.772470 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:07.772409    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:12.670182 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:12.670146    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:13.328736 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:13.328700    6059 scope.go:122] "RemoveContainer" containerID="325e0f1e34551a6ed6ba86ba3815473209b8c8fcedfa01380732b74b3e8ac894"
Nov 03 18:00:14.512012 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:14.511970    6059 generic.go:356] "Generic (PLEG): container finished" podID="ca0cb266-2071-4abc-9c89-fa5ff6601f6b" containerID="dd3a7e545840128f2f198136faf78e22eb77bc0a96aba23534c5a9cc8f611ddc" exitCode=60
Nov 03 18:00:14.512012 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:14.512016    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt" event={"ID":"ca0cb266-2071-4abc-9c89-fa5ff6601f6b","Type":"ContainerDied","Data":"dd3a7e545840128f2f198136faf78e22eb77bc0a96aba23534c5a9cc8f611ddc"}
Nov 03 18:00:14.512606 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:14.512091    6059 scope.go:122] "RemoveContainer" containerID="325e0f1e34551a6ed6ba86ba3815473209b8c8fcedfa01380732b74b3e8ac894"
Nov 03 18:00:14.512606 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:14.512524    6059 scope.go:122] "RemoveContainer" containerID="dd3a7e545840128f2f198136faf78e22eb77bc0a96aba23534c5a9cc8f611ddc"
Nov 03 18:00:14.512863 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:00:14.512838    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895480-t78nt_odh-ai-gateway-infra(ca0cb266-2071-4abc-9c89-fa5ff6601f6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt" podUID="ca0cb266-2071-4abc-9c89-fa5ff6601f6b"
Nov 03 18:00:15.643485 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:15.643438    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt"]
Nov 03 18:00:15.785917 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:15.785885    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt"
Nov 03 18:00:15.880514 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:15.880468    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca0cb266-2071-4abc-9c89-fa5ff6601f6b-kube-api-access-ndklp\" (UniqueName: \"kubernetes.io/projected/ca0cb266-2071-4abc-9c89-fa5ff6601f6b-kube-api-access-ndklp\") pod \"ca0cb266-2071-4abc-9c89-fa5ff6601f6b\" (UID: \"ca0cb266-2071-4abc-9c89-fa5ff6601f6b\") "
Nov 03 18:00:15.880728 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:15.880585    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca0cb266-2071-4abc-9c89-fa5ff6601f6b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0cb266-2071-4abc-9c89-fa5ff6601f6b-maas-api-service-ca\") pod \"ca0cb266-2071-4abc-9c89-fa5ff6601f6b\" (UID: \"ca0cb266-2071-4abc-9c89-fa5ff6601f6b\") "
Nov 03 18:00:15.880973 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:15.880942    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca0cb266-2071-4abc-9c89-fa5ff6601f6b-maas-api-service-ca" pod "ca0cb266-2071-4abc-9c89-fa5ff6601f6b" (UID: "ca0cb266-2071-4abc-9c89-fa5ff6601f6b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:15.882791 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:15.882766    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0cb266-2071-4abc-9c89-fa5ff6601f6b-kube-api-access-ndklp" pod "ca0cb266-2071-4abc-9c89-fa5ff6601f6b" (UID: "ca0cb266-2071-4abc-9c89-fa5ff6601f6b"). InnerVolumeSpecName "kube-api-access-ndklp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:15.982515 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:15.982477    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0cb266-2071-4abc-9c89-fa5ff6601f6b-maas-api-service-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:15.982515 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:15.982511    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndklp\" (UniqueName: \"kubernetes.io/projected/ca0cb266-2071-4abc-9c89-fa5ff6601f6b-kube-api-access-ndklp\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:16.526709 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:16.526679    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt"
Nov 03 18:00:16.526709 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:16.526693    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt" event={"ID":"ca0cb266-2071-4abc-9c89-fa5ff6601f6b","Type":"ContainerDied","Data":"a657eb4b2d76c455d8fe8ebd55a93ea3ae76e4242d78b727373c52cd83b7be04"}
Nov 03 18:00:16.526990 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:16.526750    6059 scope.go:122] "RemoveContainer" containerID="dd3a7e545840128f2f198136faf78e22eb77bc0a96aba23534c5a9cc8f611ddc"
Nov 03 18:00:16.696013 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:16.695968    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt"]
Nov 03 18:00:16.747816 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:16.747776    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-t78nt"]
Nov 03 18:00:17.335594 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:17.335560    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca0cb266-2071-4abc-9c89-fa5ff6601f6b" path="/var/lib/kubelet/pods/ca0cb266-2071-4abc-9c89-fa5ff6601f6b/volumes"
Nov 03 18:00:17.685079 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:17.684988    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:22.586393 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:22.586338    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:22.803884 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:22.803842    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:22.989628 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:22.989593    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:27.469528 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:27.469494    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:32.364160 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:32.364127    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:36.861513 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:00:36.861467    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:00:37.272210 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:37.272175    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:39.320696 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:39.320648    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"]
Nov 03 18:00:42.148637 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:42.148601    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:44.322155 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.322082    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd" podUID="ec9e0516-c76e-4d72-9ff0-1e416429e93f" containerName="payload-processing" containerID="cri-o://4074faa78bea593304c8379cb7ce0629a63a846836a7235552c5d58c874400e2" gracePeriod=25
Nov 03 18:00:44.575645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.575575    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"
Nov 03 18:00:44.678211 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.678162    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec9e0516-c76e-4d72-9ff0-1e416429e93f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec9e0516-c76e-4d72-9ff0-1e416429e93f-routing-overlay-volume\") pod \"ec9e0516-c76e-4d72-9ff0-1e416429e93f\" (UID: \"ec9e0516-c76e-4d72-9ff0-1e416429e93f\") "
Nov 03 18:00:44.678431 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.678228    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec9e0516-c76e-4d72-9ff0-1e416429e93f-kube-api-access-nzzcb\" (UniqueName: \"kubernetes.io/projected/ec9e0516-c76e-4d72-9ff0-1e416429e93f-kube-api-access-nzzcb\") pod \"ec9e0516-c76e-4d72-9ff0-1e416429e93f\" (UID: \"ec9e0516-c76e-4d72-9ff0-1e416429e93f\") "
Nov 03 18:00:44.678431 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.678306    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec9e0516-c76e-4d72-9ff0-1e416429e93f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9e0516-c76e-4d72-9ff0-1e416429e93f-plugins-config-volume\") pod \"ec9e0516-c76e-4d72-9ff0-1e416429e93f\" (UID: \"ec9e0516-c76e-4d72-9ff0-1e416429e93f\") "
Nov 03 18:00:44.678520 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.678440    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec9e0516-c76e-4d72-9ff0-1e416429e93f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec9e0516-c76e-4d72-9ff0-1e416429e93f-provider-credentials\") pod \"ec9e0516-c76e-4d72-9ff0-1e416429e93f\" (UID: \"ec9e0516-c76e-4d72-9ff0-1e416429e93f\") "
Nov 03 18:00:44.678622 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.678594    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec9e0516-c76e-4d72-9ff0-1e416429e93f-routing-overlay-volume" pod "ec9e0516-c76e-4d72-9ff0-1e416429e93f" (UID: "ec9e0516-c76e-4d72-9ff0-1e416429e93f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:44.678751 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.678725    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec9e0516-c76e-4d72-9ff0-1e416429e93f-plugins-config-volume" pod "ec9e0516-c76e-4d72-9ff0-1e416429e93f" (UID: "ec9e0516-c76e-4d72-9ff0-1e416429e93f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:44.678848 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.678832    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9e0516-c76e-4d72-9ff0-1e416429e93f-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:44.678886 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.678851    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec9e0516-c76e-4d72-9ff0-1e416429e93f-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:44.680435 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.680407    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9e0516-c76e-4d72-9ff0-1e416429e93f-kube-api-access-nzzcb" pod "ec9e0516-c76e-4d72-9ff0-1e416429e93f" (UID: "ec9e0516-c76e-4d72-9ff0-1e416429e93f"). InnerVolumeSpecName "kube-api-access-nzzcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:44.680435 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.680423    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9e0516-c76e-4d72-9ff0-1e416429e93f-provider-credentials" pod "ec9e0516-c76e-4d72-9ff0-1e416429e93f" (UID: "ec9e0516-c76e-4d72-9ff0-1e416429e93f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:44.690346 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.690312    6059 generic.go:356] "Generic (PLEG): container finished" podID="ec9e0516-c76e-4d72-9ff0-1e416429e93f" containerID="4074faa78bea593304c8379cb7ce0629a63a846836a7235552c5d58c874400e2" exitCode=0
Nov 03 18:00:44.690503 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.690388    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"
Nov 03 18:00:44.690503 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.690398    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd" event={"ID":"ec9e0516-c76e-4d72-9ff0-1e416429e93f","Type":"ContainerDied","Data":"4074faa78bea593304c8379cb7ce0629a63a846836a7235552c5d58c874400e2"}
Nov 03 18:00:44.690503 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.690488    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd" event={"ID":"ec9e0516-c76e-4d72-9ff0-1e416429e93f","Type":"ContainerDied","Data":"97ca6302e01f185166caeef933ff560aacda70eded05125305d7dff5991c9d97"}
Nov 03 18:00:44.690612 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.690512    6059 scope.go:122] "RemoveContainer" containerID="4074faa78bea593304c8379cb7ce0629a63a846836a7235552c5d58c874400e2"
Nov 03 18:00:44.706739 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.706709    6059 scope.go:122] "RemoveContainer" containerID="4074faa78bea593304c8379cb7ce0629a63a846836a7235552c5d58c874400e2"
Nov 03 18:00:44.707143 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:00:44.707115    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4074faa78bea593304c8379cb7ce0629a63a846836a7235552c5d58c874400e2\": container with ID starting with 4074faa78bea593304c8379cb7ce0629a63a846836a7235552c5d58c874400e2 not found: ID does not exist" containerID="4074faa78bea593304c8379cb7ce0629a63a846836a7235552c5d58c874400e2"
Nov 03 18:00:44.707254 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.707150    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4074faa78bea593304c8379cb7ce0629a63a846836a7235552c5d58c874400e2"} err="failed to get container status \"4074faa78bea593304c8379cb7ce0629a63a846836a7235552c5d58c874400e2\": rpc error: code = NotFound desc = could not find container \"4074faa78bea593304c8379cb7ce0629a63a846836a7235552c5d58c874400e2\": container with ID starting with 4074faa78bea593304c8379cb7ce0629a63a846836a7235552c5d58c874400e2 not found: ID does not exist"
Nov 03 18:00:44.780077 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.780034    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec9e0516-c76e-4d72-9ff0-1e416429e93f-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:44.780077 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.780071    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzzcb\" (UniqueName: \"kubernetes.io/projected/ec9e0516-c76e-4d72-9ff0-1e416429e93f-kube-api-access-nzzcb\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:44.867448 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.867345    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"]
Nov 03 18:00:44.874396 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:44.874346    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fdctd"]
Nov 03 18:00:45.045001 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:45.044973    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:45.335370 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:45.335320    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec9e0516-c76e-4d72-9ff0-1e416429e93f" path="/var/lib/kubelet/pods/ec9e0516-c76e-4d72-9ff0-1e416429e93f/volumes"
Nov 03 18:00:47.065247 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:47.065192    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:51.965179 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:51.965013    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:00:56.870136 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:00:56.870104    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:01.786641 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:01:01.786606    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:06.669453 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:01:06.669417    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:06.861318 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:01:06.861278    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:01:11.575250 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:01:11.575214    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:16.565744 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:01:16.565699    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:21.466948 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:01:21.466851    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:26.367314 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:01:26.366886    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:31.274734 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:01:31.274681    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:36.156697 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:01:36.156660    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:36.861846 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:01:36.861806    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:01:40.826489 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:01:40.826450    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:41.061310 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:01:41.061271    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:45.395126 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:01:45.395089    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:46.059297 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:01:46.059260    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:50.980888 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:01:50.980833    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:01:55.867430 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:01:55.867343    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:00.794092 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:00.794051    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:04.085751 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:04.085713    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:04.272167 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:04.272134    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:05.678945 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:05.678861    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:06.862523 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:02:06.862487    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:02:10.574267 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:10.574232    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:15.480085 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:15.480052    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:20.368973 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:20.368940    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:25.271047 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:25.271009    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:28.985732 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:28.985698    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:30.269737 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:30.269661    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:35.123788 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:35.123740    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:36.861503 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:02:36.861467    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:02:39.772499 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:39.772470    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:02:39.783618 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:39.783582    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:02:39.790234 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:39.790207    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:02:39.801056 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:39.801026    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:02:40.061877 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:40.061742    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:44.161662 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:44.161611    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:44.355301 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:44.355264    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:45.081044 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:45.081008    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:45.766761 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:45.766728    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:47.347347 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:47.347264    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:47.534552 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:47.534505    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:48.092214 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.092176    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"]
Nov 03 18:02:48.092671 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.092656    6059 cpu_manager.go:414] "containerMap: removing container" podUID="ec9e0516-c76e-4d72-9ff0-1e416429e93f" containerName="payload-processing"
Nov 03 18:02:48.092726 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.092672    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec9e0516-c76e-4d72-9ff0-1e416429e93f" containerName="payload-processing"
Nov 03 18:02:48.092726 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.092684    6059 cpu_manager.go:414] "containerMap: removing container" podUID="ca0cb266-2071-4abc-9c89-fa5ff6601f6b" containerName="cleanup"
Nov 03 18:02:48.092726 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.092690    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca0cb266-2071-4abc-9c89-fa5ff6601f6b" containerName="cleanup"
Nov 03 18:02:48.092726 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.092710    6059 cpu_manager.go:414] "containerMap: removing container" podUID="ca0cb266-2071-4abc-9c89-fa5ff6601f6b" containerName="cleanup"
Nov 03 18:02:48.092726 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.092715    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca0cb266-2071-4abc-9c89-fa5ff6601f6b" containerName="cleanup"
Nov 03 18:02:48.092726 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.092724    6059 cpu_manager.go:414] "containerMap: removing container" podUID="ca0cb266-2071-4abc-9c89-fa5ff6601f6b" containerName="cleanup"
Nov 03 18:02:48.092928 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.092729    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca0cb266-2071-4abc-9c89-fa5ff6601f6b" containerName="cleanup"
Nov 03 18:02:48.092928 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.092799    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca0cb266-2071-4abc-9c89-fa5ff6601f6b" containerName="cleanup"
Nov 03 18:02:48.092928 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.092807    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca0cb266-2071-4abc-9c89-fa5ff6601f6b" containerName="cleanup"
Nov 03 18:02:48.092928 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.092820    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec9e0516-c76e-4d72-9ff0-1e416429e93f" containerName="payload-processing"
Nov 03 18:02:48.096024 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.096002    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"
Nov 03 18:02:48.151553 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.151514    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"agc-cred-b23a6815d37d\""
Nov 03 18:02:48.151828 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.151804    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"e2e-cfl-d35f0c-provider-cred\""
Nov 03 18:02:48.151907 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.151804    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"agc-cred-efacdc6a26da\""
Nov 03 18:02:48.151907 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.151805    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"payload-processing-external-model-e2e--58d254fa-dockercfg-zmxcc\""
Nov 03 18:02:48.203473 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.203436    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-029ca0\""
Nov 03 18:02:48.203671 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.203492    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"kube-root-ca.crt\""
Nov 03 18:02:48.203671 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.203538    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"openshift-service-ca.crt\""
Nov 03 18:02:48.203671 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.203580    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"routing-overlay\""
Nov 03 18:02:48.205816 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.205792    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"]
Nov 03 18:02:48.256376 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.256326    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43fc8368-78b5-4856-a664-dcb061be31a5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7\" (UID: \"43fc8368-78b5-4856-a664-dcb061be31a5\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"
Nov 03 18:02:48.256590 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.256441    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43fc8368-78b5-4856-a664-dcb061be31a5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7\" (UID: \"43fc8368-78b5-4856-a664-dcb061be31a5\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"
Nov 03 18:02:48.256590 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.256497    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43fc8368-78b5-4856-a664-dcb061be31a5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7\" (UID: \"43fc8368-78b5-4856-a664-dcb061be31a5\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"
Nov 03 18:02:48.256590 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.256537    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stp5m\" (UniqueName: \"kubernetes.io/projected/43fc8368-78b5-4856-a664-dcb061be31a5-kube-api-access-stp5m\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7\" (UID: \"43fc8368-78b5-4856-a664-dcb061be31a5\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"
Nov 03 18:02:48.358177 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.358079    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43fc8368-78b5-4856-a664-dcb061be31a5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7\" (UID: \"43fc8368-78b5-4856-a664-dcb061be31a5\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"
Nov 03 18:02:48.358177 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.358145    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43fc8368-78b5-4856-a664-dcb061be31a5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7\" (UID: \"43fc8368-78b5-4856-a664-dcb061be31a5\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"
Nov 03 18:02:48.358787 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.358197    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43fc8368-78b5-4856-a664-dcb061be31a5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7\" (UID: \"43fc8368-78b5-4856-a664-dcb061be31a5\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"
Nov 03 18:02:48.358787 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.358247    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stp5m\" (UniqueName: \"kubernetes.io/projected/43fc8368-78b5-4856-a664-dcb061be31a5-kube-api-access-stp5m\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7\" (UID: \"43fc8368-78b5-4856-a664-dcb061be31a5\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"
Nov 03 18:02:48.358981 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.358956    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43fc8368-78b5-4856-a664-dcb061be31a5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7\" (UID: \"43fc8368-78b5-4856-a664-dcb061be31a5\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"
Nov 03 18:02:48.359047 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.358987    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43fc8368-78b5-4856-a664-dcb061be31a5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7\" (UID: \"43fc8368-78b5-4856-a664-dcb061be31a5\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"
Nov 03 18:02:48.361028 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.361001    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43fc8368-78b5-4856-a664-dcb061be31a5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7\" (UID: \"43fc8368-78b5-4856-a664-dcb061be31a5\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"
Nov 03 18:02:48.421787 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.421743    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stp5m\" (UniqueName: \"kubernetes.io/projected/43fc8368-78b5-4856-a664-dcb061be31a5-kube-api-access-stp5m\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7\" (UID: \"43fc8368-78b5-4856-a664-dcb061be31a5\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"
Nov 03 18:02:48.718843 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:48.718807    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"
Nov 03 18:02:49.101329 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:49.101246    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"]
Nov 03 18:02:49.104743 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:02:49.104717    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43fc8368_78b5_4856_a664_dcb061be31a5.slice/crio-d8bd4260bd12a89ade63869a75f5309cdeae9d52ec3fa7133ef56a699e1d6be1 WatchSource:0}: Error finding container d8bd4260bd12a89ade63869a75f5309cdeae9d52ec3fa7133ef56a699e1d6be1: Status 404 returned error can't find the container with id d8bd4260bd12a89ade63869a75f5309cdeae9d52ec3fa7133ef56a699e1d6be1
Nov 03 18:02:49.371826 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:49.371746    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7" event={"ID":"43fc8368-78b5-4856-a664-dcb061be31a5","Type":"ContainerStarted","Data":"331b334128e98e33d9304c3b0b4b8d1863e853ea03ff079f936a4234144f1202"}
Nov 03 18:02:49.371826 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:49.371796    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7" event={"ID":"43fc8368-78b5-4856-a664-dcb061be31a5","Type":"ContainerStarted","Data":"d8bd4260bd12a89ade63869a75f5309cdeae9d52ec3fa7133ef56a699e1d6be1"}
Nov 03 18:02:49.372212 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:49.371854    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"
Nov 03 18:02:49.495637 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:49.495563    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7" podStartSLOduration=1.49554574 podStartE2EDuration="1.49554574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:49.494732861 +0000 UTC m=+2710.292445646" watchObservedRunningTime="2026-11-03 18:02:49.49554574 +0000 UTC m=+2710.293258523"
Nov 03 18:02:50.083839 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:50.083807    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:52.918111 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:52.918073    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"]
Nov 03 18:02:52.970227 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:52.970185    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"]
Nov 03 18:02:52.970830 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:52.970813    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca0cb266-2071-4abc-9c89-fa5ff6601f6b" containerName="cleanup"
Nov 03 18:02:52.974225 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:52.974205    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"
Nov 03 18:02:53.028647 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:53.028611    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"agc-cred-734909566ac5\""
Nov 03 18:02:53.036736 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:53.036701    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"]
Nov 03 18:02:53.103571 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:53.103527    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq\" (UID: \"896c29df-4e9f-4b56-afc2-c17fd1d15e2c\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"
Nov 03 18:02:53.103571 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:53.103578    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmt5d\" (UniqueName: \"kubernetes.io/projected/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-kube-api-access-lmt5d\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq\" (UID: \"896c29df-4e9f-4b56-afc2-c17fd1d15e2c\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"
Nov 03 18:02:53.103854 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:53.103677    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq\" (UID: \"896c29df-4e9f-4b56-afc2-c17fd1d15e2c\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"
Nov 03 18:02:53.103854 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:53.103813    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq\" (UID: \"896c29df-4e9f-4b56-afc2-c17fd1d15e2c\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"
Nov 03 18:02:53.204919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:53.204811    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq\" (UID: \"896c29df-4e9f-4b56-afc2-c17fd1d15e2c\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"
Nov 03 18:02:53.204919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:53.204863    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmt5d\" (UniqueName: \"kubernetes.io/projected/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-kube-api-access-lmt5d\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq\" (UID: \"896c29df-4e9f-4b56-afc2-c17fd1d15e2c\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"
Nov 03 18:02:53.204919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:53.204901    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq\" (UID: \"896c29df-4e9f-4b56-afc2-c17fd1d15e2c\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"
Nov 03 18:02:53.205306 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:53.204961    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq\" (UID: \"896c29df-4e9f-4b56-afc2-c17fd1d15e2c\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"
Nov 03 18:02:53.205726 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:53.205700    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq\" (UID: \"896c29df-4e9f-4b56-afc2-c17fd1d15e2c\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"
Nov 03 18:02:53.205815 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:53.205764    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq\" (UID: \"896c29df-4e9f-4b56-afc2-c17fd1d15e2c\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"
Nov 03 18:02:53.207699 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:53.207676    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq\" (UID: \"896c29df-4e9f-4b56-afc2-c17fd1d15e2c\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"
Nov 03 18:02:53.268768 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:53.268728    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmt5d\" (UniqueName: \"kubernetes.io/projected/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-kube-api-access-lmt5d\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq\" (UID: \"896c29df-4e9f-4b56-afc2-c17fd1d15e2c\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"
Nov 03 18:02:53.287743 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:53.287702    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"
Nov 03 18:02:53.671218 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:53.671182    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"]
Nov 03 18:02:53.674606 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:02:53.674575    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896c29df_4e9f_4b56_afc2_c17fd1d15e2c.slice/crio-9b2bf587dcf729b41a33a36a1a06042a70ecf1745bd8765e7916ce37f5287bf1 WatchSource:0}: Error finding container 9b2bf587dcf729b41a33a36a1a06042a70ecf1745bd8765e7916ce37f5287bf1: Status 404 returned error can't find the container with id 9b2bf587dcf729b41a33a36a1a06042a70ecf1745bd8765e7916ce37f5287bf1
Nov 03 18:02:54.400581 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:54.400538    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq" event={"ID":"896c29df-4e9f-4b56-afc2-c17fd1d15e2c","Type":"ContainerStarted","Data":"b774c5a0e4f0a8dee5fd428bc004941f965dd163343c04b9de056c63b9b1b966"}
Nov 03 18:02:54.400983 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:54.400606    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"
Nov 03 18:02:54.400983 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:54.400625    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq" event={"ID":"896c29df-4e9f-4b56-afc2-c17fd1d15e2c","Type":"ContainerStarted","Data":"9b2bf587dcf729b41a33a36a1a06042a70ecf1745bd8765e7916ce37f5287bf1"}
Nov 03 18:02:54.523854 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:54.523792    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq" podStartSLOduration=2.523772686 podStartE2EDuration="2.523772686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:54.522887122 +0000 UTC m=+2715.320599904" watchObservedRunningTime="2026-11-03 18:02:54.523772686 +0000 UTC m=+2715.321485467"
Nov 03 18:02:55.070323 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:55.070286    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:57.310350 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:57.310299    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:57.498220 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:57.498187    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:02:57.918446 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:57.918400    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7" podUID="43fc8368-78b5-4856-a664-dcb061be31a5" containerName="payload-processing" containerID="cri-o://331b334128e98e33d9304c3b0b4b8d1863e853ea03ff079f936a4234144f1202" gracePeriod=25
Nov 03 18:02:58.179043 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:58.178979    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"
Nov 03 18:02:58.257120 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:58.257084    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43fc8368-78b5-4856-a664-dcb061be31a5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43fc8368-78b5-4856-a664-dcb061be31a5-routing-overlay-volume\") pod \"43fc8368-78b5-4856-a664-dcb061be31a5\" (UID: \"43fc8368-78b5-4856-a664-dcb061be31a5\") "
Nov 03 18:02:58.257311 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:58.257141    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43fc8368-78b5-4856-a664-dcb061be31a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43fc8368-78b5-4856-a664-dcb061be31a5-plugins-config-volume\") pod \"43fc8368-78b5-4856-a664-dcb061be31a5\" (UID: \"43fc8368-78b5-4856-a664-dcb061be31a5\") "
Nov 03 18:02:58.257311 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:58.257192    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43fc8368-78b5-4856-a664-dcb061be31a5-kube-api-access-stp5m\" (UniqueName: \"kubernetes.io/projected/43fc8368-78b5-4856-a664-dcb061be31a5-kube-api-access-stp5m\") pod \"43fc8368-78b5-4856-a664-dcb061be31a5\" (UID: \"43fc8368-78b5-4856-a664-dcb061be31a5\") "
Nov 03 18:02:58.257311 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:58.257293    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43fc8368-78b5-4856-a664-dcb061be31a5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/43fc8368-78b5-4856-a664-dcb061be31a5-provider-credentials\") pod \"43fc8368-78b5-4856-a664-dcb061be31a5\" (UID: \"43fc8368-78b5-4856-a664-dcb061be31a5\") "
Nov 03 18:02:58.257571 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:58.257544    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43fc8368-78b5-4856-a664-dcb061be31a5-routing-overlay-volume" pod "43fc8368-78b5-4856-a664-dcb061be31a5" (UID: "43fc8368-78b5-4856-a664-dcb061be31a5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:02:58.257690 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:58.257670    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43fc8368-78b5-4856-a664-dcb061be31a5-plugins-config-volume" pod "43fc8368-78b5-4856-a664-dcb061be31a5" (UID: "43fc8368-78b5-4856-a664-dcb061be31a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:02:58.257690 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:58.257684    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43fc8368-78b5-4856-a664-dcb061be31a5-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:02:58.259482 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:58.259462    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43fc8368-78b5-4856-a664-dcb061be31a5-provider-credentials" pod "43fc8368-78b5-4856-a664-dcb061be31a5" (UID: "43fc8368-78b5-4856-a664-dcb061be31a5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:02:58.259554 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:58.259530    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43fc8368-78b5-4856-a664-dcb061be31a5-kube-api-access-stp5m" pod "43fc8368-78b5-4856-a664-dcb061be31a5" (UID: "43fc8368-78b5-4856-a664-dcb061be31a5"). InnerVolumeSpecName "kube-api-access-stp5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:02:58.358184 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:58.358144    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43fc8368-78b5-4856-a664-dcb061be31a5-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:02:58.358184 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:58.358178    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43fc8368-78b5-4856-a664-dcb061be31a5-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:02:58.358184 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:58.358188    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stp5m\" (UniqueName: \"kubernetes.io/projected/43fc8368-78b5-4856-a664-dcb061be31a5-kube-api-access-stp5m\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:02:58.423346 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:58.423307    6059 generic.go:356] "Generic (PLEG): container finished" podID="43fc8368-78b5-4856-a664-dcb061be31a5" containerID="331b334128e98e33d9304c3b0b4b8d1863e853ea03ff079f936a4234144f1202" exitCode=0
Nov 03 18:02:58.423561 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:58.423389    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"
Nov 03 18:02:58.423561 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:58.423394    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7" event={"ID":"43fc8368-78b5-4856-a664-dcb061be31a5","Type":"ContainerDied","Data":"331b334128e98e33d9304c3b0b4b8d1863e853ea03ff079f936a4234144f1202"}
Nov 03 18:02:58.423561 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:58.423449    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7" event={"ID":"43fc8368-78b5-4856-a664-dcb061be31a5","Type":"ContainerDied","Data":"d8bd4260bd12a89ade63869a75f5309cdeae9d52ec3fa7133ef56a699e1d6be1"}
Nov 03 18:02:58.423561 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:58.423473    6059 scope.go:122] "RemoveContainer" containerID="331b334128e98e33d9304c3b0b4b8d1863e853ea03ff079f936a4234144f1202"
Nov 03 18:02:58.432755 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:58.432734    6059 scope.go:122] "RemoveContainer" containerID="331b334128e98e33d9304c3b0b4b8d1863e853ea03ff079f936a4234144f1202"
Nov 03 18:02:58.433046 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:02:58.433029    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"331b334128e98e33d9304c3b0b4b8d1863e853ea03ff079f936a4234144f1202\": container with ID starting with 331b334128e98e33d9304c3b0b4b8d1863e853ea03ff079f936a4234144f1202 not found: ID does not exist" containerID="331b334128e98e33d9304c3b0b4b8d1863e853ea03ff079f936a4234144f1202"
Nov 03 18:02:58.433104 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:58.433054    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"331b334128e98e33d9304c3b0b4b8d1863e853ea03ff079f936a4234144f1202"} err="failed to get container status \"331b334128e98e33d9304c3b0b4b8d1863e853ea03ff079f936a4234144f1202\": rpc error: code = NotFound desc = could not find container \"331b334128e98e33d9304c3b0b4b8d1863e853ea03ff079f936a4234144f1202\": container with ID starting with 331b334128e98e33d9304c3b0b4b8d1863e853ea03ff079f936a4234144f1202 not found: ID does not exist"
Nov 03 18:02:58.595062 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:58.595024    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"]
Nov 03 18:02:58.599394 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:58.599367    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fc8td7"]
Nov 03 18:02:59.335159 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:02:59.335122    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43fc8368-78b5-4856-a664-dcb061be31a5" path="/var/lib/kubelet/pods/43fc8368-78b5-4856-a664-dcb061be31a5/volumes"
Nov 03 18:03:00.005938 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:00.005899    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:04.866917 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:04.866877    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:05.410031 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:05.409996    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"
Nov 03 18:03:06.862133 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:03:06.862094    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:03:09.783033 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:09.782999    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:14.929147 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:14.929115    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:19.878608 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:19.878571    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:24.779810 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:24.779768    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:29.681185 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:29.681127    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:34.579659 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:34.579598    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:36.861255 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:03:36.861212    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:03:39.473729 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:39.473687    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:44.464688 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:44.464651    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:46.110410 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:46.110347    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:49.376222 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:49.376153    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:51.219713 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:51.219671    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"]
Nov 03 18:03:51.220293 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:51.219882    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l" podUID="2861a97b-ab99-4cfc-8a5c-bd2408972572" containerName="istio-proxy" containerID="cri-o://d2aabfa1ba75eb345ceed2e607fe8a7d1149b10cca7651538e77acf962c828b4" gracePeriod=30
Nov 03 18:03:54.296163 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:54.296120    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:56.483098 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.483058    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 18:03:56.560211 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.560122    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-data\") pod \"2861a97b-ab99-4cfc-8a5c-bd2408972572\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") "
Nov 03 18:03:56.560211 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.560177    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2861a97b-ab99-4cfc-8a5c-bd2408972572-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2861a97b-ab99-4cfc-8a5c-bd2408972572-istiod-ca-cert\") pod \"2861a97b-ab99-4cfc-8a5c-bd2408972572\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") "
Nov 03 18:03:56.560486 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.560258    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-workload-certs\") pod \"2861a97b-ab99-4cfc-8a5c-bd2408972572\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") "
Nov 03 18:03:56.560486 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.560313    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-token\" (UniqueName: \"kubernetes.io/projected/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-token\") pod \"2861a97b-ab99-4cfc-8a5c-bd2408972572\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") "
Nov 03 18:03:56.560486 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.560341    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-credential-socket\") pod \"2861a97b-ab99-4cfc-8a5c-bd2408972572\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") "
Nov 03 18:03:56.560486 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.560415    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-podinfo\") pod \"2861a97b-ab99-4cfc-8a5c-bd2408972572\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") "
Nov 03 18:03:56.560727 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.560504    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-envoy\") pod \"2861a97b-ab99-4cfc-8a5c-bd2408972572\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") "
Nov 03 18:03:56.560727 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.560537    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-workload-socket\") pod \"2861a97b-ab99-4cfc-8a5c-bd2408972572\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") "
Nov 03 18:03:56.560727 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.560566    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2861a97b-ab99-4cfc-8a5c-bd2408972572-kube-api-access-trlk6\" (UniqueName: \"kubernetes.io/projected/2861a97b-ab99-4cfc-8a5c-bd2408972572-kube-api-access-trlk6\") pod \"2861a97b-ab99-4cfc-8a5c-bd2408972572\" (UID: \"2861a97b-ab99-4cfc-8a5c-bd2408972572\") "
Nov 03 18:03:56.560727 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.560476    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-data" pod "2861a97b-ab99-4cfc-8a5c-bd2408972572" (UID: "2861a97b-ab99-4cfc-8a5c-bd2408972572"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:03:56.560727 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.560561    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-workload-certs" pod "2861a97b-ab99-4cfc-8a5c-bd2408972572" (UID: "2861a97b-ab99-4cfc-8a5c-bd2408972572"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:03:56.560727 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.560648    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-credential-socket" pod "2861a97b-ab99-4cfc-8a5c-bd2408972572" (UID: "2861a97b-ab99-4cfc-8a5c-bd2408972572"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:03:56.560727 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.560673    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2861a97b-ab99-4cfc-8a5c-bd2408972572-istiod-ca-cert" pod "2861a97b-ab99-4cfc-8a5c-bd2408972572" (UID: "2861a97b-ab99-4cfc-8a5c-bd2408972572"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:03:56.561105 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.560820    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-workload-socket" pod "2861a97b-ab99-4cfc-8a5c-bd2408972572" (UID: "2861a97b-ab99-4cfc-8a5c-bd2408972572"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:03:56.561105 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.560999    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:03:56.561105 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.561016    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:03:56.561105 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.561028    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2861a97b-ab99-4cfc-8a5c-bd2408972572-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:03:56.561105 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.561045    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:03:56.561105 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.561064    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:03:56.562813 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.562785    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-envoy" pod "2861a97b-ab99-4cfc-8a5c-bd2408972572" (UID: "2861a97b-ab99-4cfc-8a5c-bd2408972572"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:03:56.562966 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.562814    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2861a97b-ab99-4cfc-8a5c-bd2408972572-kube-api-access-trlk6" pod "2861a97b-ab99-4cfc-8a5c-bd2408972572" (UID: "2861a97b-ab99-4cfc-8a5c-bd2408972572"). InnerVolumeSpecName "kube-api-access-trlk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:03:56.562966 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.562928    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-podinfo" pod "2861a97b-ab99-4cfc-8a5c-bd2408972572" (UID: "2861a97b-ab99-4cfc-8a5c-bd2408972572"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:03:56.563136 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.563114    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-token" pod "2861a97b-ab99-4cfc-8a5c-bd2408972572" (UID: "2861a97b-ab99-4cfc-8a5c-bd2408972572"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:03:56.661832 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.661791    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:03:56.661832 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.661836    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:03:56.662096 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.661852    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2861a97b-ab99-4cfc-8a5c-bd2408972572-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:03:56.662096 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.661868    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trlk6\" (UniqueName: \"kubernetes.io/projected/2861a97b-ab99-4cfc-8a5c-bd2408972572-kube-api-access-trlk6\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:03:56.742673 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.742642    6059 generic.go:356] "Generic (PLEG): container finished" podID="2861a97b-ab99-4cfc-8a5c-bd2408972572" containerID="d2aabfa1ba75eb345ceed2e607fe8a7d1149b10cca7651538e77acf962c828b4" exitCode=0
Nov 03 18:03:56.742860 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.742684    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l" event={"ID":"2861a97b-ab99-4cfc-8a5c-bd2408972572","Type":"ContainerDied","Data":"d2aabfa1ba75eb345ceed2e607fe8a7d1149b10cca7651538e77acf962c828b4"}
Nov 03 18:03:56.742860 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.742707    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"
Nov 03 18:03:56.742860 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.742726    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l" event={"ID":"2861a97b-ab99-4cfc-8a5c-bd2408972572","Type":"ContainerDied","Data":"004e813e89b5d332c786f0eecf9fe1ce88a0c58ec083ca2145229acf2a64352e"}
Nov 03 18:03:56.742860 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.742746    6059 scope.go:122] "RemoveContainer" containerID="d2aabfa1ba75eb345ceed2e607fe8a7d1149b10cca7651538e77acf962c828b4"
Nov 03 18:03:56.752709 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.752688    6059 scope.go:122] "RemoveContainer" containerID="503cfd967c23e1ec0f4e2fd0cb8b59adcee7f840f752e7aa337c5119f3856750"
Nov 03 18:03:56.762235 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.762213    6059 scope.go:122] "RemoveContainer" containerID="d2aabfa1ba75eb345ceed2e607fe8a7d1149b10cca7651538e77acf962c828b4"
Nov 03 18:03:56.762565 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:03:56.762543    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2aabfa1ba75eb345ceed2e607fe8a7d1149b10cca7651538e77acf962c828b4\": container with ID starting with d2aabfa1ba75eb345ceed2e607fe8a7d1149b10cca7651538e77acf962c828b4 not found: ID does not exist" containerID="d2aabfa1ba75eb345ceed2e607fe8a7d1149b10cca7651538e77acf962c828b4"
Nov 03 18:03:56.762648 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.762574    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2aabfa1ba75eb345ceed2e607fe8a7d1149b10cca7651538e77acf962c828b4"} err="failed to get container status \"d2aabfa1ba75eb345ceed2e607fe8a7d1149b10cca7651538e77acf962c828b4\": rpc error: code = NotFound desc = could not find container \"d2aabfa1ba75eb345ceed2e607fe8a7d1149b10cca7651538e77acf962c828b4\": container with ID starting with d2aabfa1ba75eb345ceed2e607fe8a7d1149b10cca7651538e77acf962c828b4 not found: ID does not exist"
Nov 03 18:03:56.762648 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.762594    6059 scope.go:122] "RemoveContainer" containerID="503cfd967c23e1ec0f4e2fd0cb8b59adcee7f840f752e7aa337c5119f3856750"
Nov 03 18:03:56.762846 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:03:56.762822    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"503cfd967c23e1ec0f4e2fd0cb8b59adcee7f840f752e7aa337c5119f3856750\": container with ID starting with 503cfd967c23e1ec0f4e2fd0cb8b59adcee7f840f752e7aa337c5119f3856750 not found: ID does not exist" containerID="503cfd967c23e1ec0f4e2fd0cb8b59adcee7f840f752e7aa337c5119f3856750"
Nov 03 18:03:56.762899 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.762854    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"503cfd967c23e1ec0f4e2fd0cb8b59adcee7f840f752e7aa337c5119f3856750"} err="failed to get container status \"503cfd967c23e1ec0f4e2fd0cb8b59adcee7f840f752e7aa337c5119f3856750\": rpc error: code = NotFound desc = could not find container \"503cfd967c23e1ec0f4e2fd0cb8b59adcee7f840f752e7aa337c5119f3856750\": container with ID starting with 503cfd967c23e1ec0f4e2fd0cb8b59adcee7f840f752e7aa337c5119f3856750 not found: ID does not exist"
Nov 03 18:03:56.919093 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.918998    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"]
Nov 03 18:03:56.924943 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:56.924913    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-xxr9l"]
Nov 03 18:03:57.335913 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:57.335873    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2861a97b-ab99-4cfc-8a5c-bd2408972572" path="/var/lib/kubelet/pods/2861a97b-ab99-4cfc-8a5c-bd2408972572/volumes"
Nov 03 18:03:59.274843 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:59.274808    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:03:59.310725 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:59.310686    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"]
Nov 03 18:03:59.310925 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:59.310896    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6" podUID="32988d90-426a-4a7b-b034-a4685db935b8" containerName="istio-proxy" containerID="cri-o://fc2a5b83871e84ce9f9c7c75b9e50eb323ccc38e6f041a781aa94eeae78d98fd" gracePeriod=30
Nov 03 18:03:59.684211 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:59.684112    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8kpl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:03:59.684211 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:03:59.684197    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6" podUID="32988d90-426a-4a7b-b034-a4685db935b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:04:04.165263 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.165222    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:04.569452 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.569425    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 18:04:04.744012 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.743977    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-workload-socket\") pod \"32988d90-426a-4a7b-b034-a4685db935b8\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") "
Nov 03 18:04:04.744231 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.744083    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32988d90-426a-4a7b-b034-a4685db935b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32988d90-426a-4a7b-b034-a4685db935b8-istio-podinfo\") pod \"32988d90-426a-4a7b-b034-a4685db935b8\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") "
Nov 03 18:04:04.744231 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.744110    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-istio-envoy\") pod \"32988d90-426a-4a7b-b034-a4685db935b8\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") "
Nov 03 18:04:04.744231 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.744147    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-credential-socket\") pod \"32988d90-426a-4a7b-b034-a4685db935b8\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") "
Nov 03 18:04:04.744231 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.744177    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32988d90-426a-4a7b-b034-a4685db935b8-istio-token\" (UniqueName: \"kubernetes.io/projected/32988d90-426a-4a7b-b034-a4685db935b8-istio-token\") pod \"32988d90-426a-4a7b-b034-a4685db935b8\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") "
Nov 03 18:04:04.744231 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.744221    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32988d90-426a-4a7b-b034-a4685db935b8-kube-api-access-q6pgw\" (UniqueName: \"kubernetes.io/projected/32988d90-426a-4a7b-b034-a4685db935b8-kube-api-access-q6pgw\") pod \"32988d90-426a-4a7b-b034-a4685db935b8\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") "
Nov 03 18:04:04.744559 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.744300    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-istio-data\") pod \"32988d90-426a-4a7b-b034-a4685db935b8\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") "
Nov 03 18:04:04.744559 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.744315    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-workload-socket" pod "32988d90-426a-4a7b-b034-a4685db935b8" (UID: "32988d90-426a-4a7b-b034-a4685db935b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:04.744559 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.744385    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32988d90-426a-4a7b-b034-a4685db935b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32988d90-426a-4a7b-b034-a4685db935b8-istiod-ca-cert\") pod \"32988d90-426a-4a7b-b034-a4685db935b8\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") "
Nov 03 18:04:04.744559 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.744424    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-workload-certs\") pod \"32988d90-426a-4a7b-b034-a4685db935b8\" (UID: \"32988d90-426a-4a7b-b034-a4685db935b8\") "
Nov 03 18:04:04.744559 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.744486    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-credential-socket" pod "32988d90-426a-4a7b-b034-a4685db935b8" (UID: "32988d90-426a-4a7b-b034-a4685db935b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:04.744847 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.744805    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32988d90-426a-4a7b-b034-a4685db935b8-istiod-ca-cert" pod "32988d90-426a-4a7b-b034-a4685db935b8" (UID: "32988d90-426a-4a7b-b034-a4685db935b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:04:04.745016 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.744991    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-istio-data" pod "32988d90-426a-4a7b-b034-a4685db935b8" (UID: "32988d90-426a-4a7b-b034-a4685db935b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:04.745223 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.745190    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-workload-certs" pod "32988d90-426a-4a7b-b034-a4685db935b8" (UID: "32988d90-426a-4a7b-b034-a4685db935b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:04.747042 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.746999    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-istio-envoy" pod "32988d90-426a-4a7b-b034-a4685db935b8" (UID: "32988d90-426a-4a7b-b034-a4685db935b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:04.747042 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.747013    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32988d90-426a-4a7b-b034-a4685db935b8-kube-api-access-q6pgw" pod "32988d90-426a-4a7b-b034-a4685db935b8" (UID: "32988d90-426a-4a7b-b034-a4685db935b8"). InnerVolumeSpecName "kube-api-access-q6pgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:04.747572 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.747546    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32988d90-426a-4a7b-b034-a4685db935b8-istio-podinfo" pod "32988d90-426a-4a7b-b034-a4685db935b8" (UID: "32988d90-426a-4a7b-b034-a4685db935b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:04:04.747670 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.747573    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:04:04.747670 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.747594    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32988d90-426a-4a7b-b034-a4685db935b8-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:04:04.747670 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.747606    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:04:04.747670 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.747628    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:04:04.747670 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.747640    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:04:04.747670 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.747652    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32988d90-426a-4a7b-b034-a4685db935b8-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:04:04.747670 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.747665    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6pgw\" (UniqueName: \"kubernetes.io/projected/32988d90-426a-4a7b-b034-a4685db935b8-kube-api-access-q6pgw\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:04:04.748233 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.748196    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32988d90-426a-4a7b-b034-a4685db935b8-istio-token" pod "32988d90-426a-4a7b-b034-a4685db935b8" (UID: "32988d90-426a-4a7b-b034-a4685db935b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:04.788989 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.788950    6059 generic.go:356] "Generic (PLEG): container finished" podID="32988d90-426a-4a7b-b034-a4685db935b8" containerID="fc2a5b83871e84ce9f9c7c75b9e50eb323ccc38e6f041a781aa94eeae78d98fd" exitCode=0
Nov 03 18:04:04.789190 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.789004    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"
Nov 03 18:04:04.789190 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.789035    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6" event={"ID":"32988d90-426a-4a7b-b034-a4685db935b8","Type":"ContainerDied","Data":"fc2a5b83871e84ce9f9c7c75b9e50eb323ccc38e6f041a781aa94eeae78d98fd"}
Nov 03 18:04:04.789190 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.789092    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6" event={"ID":"32988d90-426a-4a7b-b034-a4685db935b8","Type":"ContainerDied","Data":"2e80d98c8b00eda65ec88dcb26d039e24763827960b2765c6b0be1cccdbfc769"}
Nov 03 18:04:04.789190 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.789115    6059 scope.go:122] "RemoveContainer" containerID="fc2a5b83871e84ce9f9c7c75b9e50eb323ccc38e6f041a781aa94eeae78d98fd"
Nov 03 18:04:04.799325 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.799304    6059 scope.go:122] "RemoveContainer" containerID="fc2a5b83871e84ce9f9c7c75b9e50eb323ccc38e6f041a781aa94eeae78d98fd"
Nov 03 18:04:04.799680 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:04:04.799654    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc2a5b83871e84ce9f9c7c75b9e50eb323ccc38e6f041a781aa94eeae78d98fd\": container with ID starting with fc2a5b83871e84ce9f9c7c75b9e50eb323ccc38e6f041a781aa94eeae78d98fd not found: ID does not exist" containerID="fc2a5b83871e84ce9f9c7c75b9e50eb323ccc38e6f041a781aa94eeae78d98fd"
Nov 03 18:04:04.799744 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.799691    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc2a5b83871e84ce9f9c7c75b9e50eb323ccc38e6f041a781aa94eeae78d98fd"} err="failed to get container status \"fc2a5b83871e84ce9f9c7c75b9e50eb323ccc38e6f041a781aa94eeae78d98fd\": rpc error: code = NotFound desc = could not find container \"fc2a5b83871e84ce9f9c7c75b9e50eb323ccc38e6f041a781aa94eeae78d98fd\": container with ID starting with fc2a5b83871e84ce9f9c7c75b9e50eb323ccc38e6f041a781aa94eeae78d98fd not found: ID does not exist"
Nov 03 18:04:04.848844 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.848809    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32988d90-426a-4a7b-b034-a4685db935b8-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:04:04.848844 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.848839    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32988d90-426a-4a7b-b034-a4685db935b8-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:04:04.959835 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.959797    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"]
Nov 03 18:04:04.963813 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:04.963787    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kpl6"]
Nov 03 18:04:05.341408 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:05.341344    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32988d90-426a-4a7b-b034-a4685db935b8" path="/var/lib/kubelet/pods/32988d90-426a-4a7b-b034-a4685db935b8/volumes"
Nov 03 18:04:06.410581 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:06.410533    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:06.587316 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:06.587285    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:06.861730 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:04:06.861693    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:04:09.058847 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:09.058812    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:14.067256 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:14.067209    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:18.990897 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:18.990783    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:23.872491 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:23.872404    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:28.859773 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:28.859739    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:33.765518 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:33.765478    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:36.860209 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:04:36.860171    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:04:38.674949 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:38.674913    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:43.667107 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:43.667072    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:46.438325 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:46.438294    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:48.594771 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:48.594741    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:53.586077 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:53.586045    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:04:58.463675 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:04:58.463635    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:03.384159 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:03.384119    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:06.863157 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:05:06.863122    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:05:08.261906 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:08.261832    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:13.197057 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:13.197018    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:18.177078 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:18.177041    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:23.112878 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:23.112835    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:27.132894 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:27.132838    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:27.346033 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:27.345988    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:28.077660 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:28.077619    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:31.126665 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.126618    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:31.368490 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.368458    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"]
Nov 03 18:05:31.368882 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.368870    6059 cpu_manager.go:414] "containerMap: removing container" podUID="43fc8368-78b5-4856-a664-dcb061be31a5" containerName="payload-processing"
Nov 03 18:05:31.368927 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.368884    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43fc8368-78b5-4856-a664-dcb061be31a5" containerName="payload-processing"
Nov 03 18:05:31.368927 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.368911    6059 cpu_manager.go:414] "containerMap: removing container" podUID="2861a97b-ab99-4cfc-8a5c-bd2408972572" containerName="istio-proxy"
Nov 03 18:05:31.368927 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.368917    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2861a97b-ab99-4cfc-8a5c-bd2408972572" containerName="istio-proxy"
Nov 03 18:05:31.369046 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.368929    6059 cpu_manager.go:414] "containerMap: removing container" podUID="32988d90-426a-4a7b-b034-a4685db935b8" containerName="istio-proxy"
Nov 03 18:05:31.369046 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.368935    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32988d90-426a-4a7b-b034-a4685db935b8" containerName="istio-proxy"
Nov 03 18:05:31.369046 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.369016    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2861a97b-ab99-4cfc-8a5c-bd2408972572" containerName="istio-proxy"
Nov 03 18:05:31.369046 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.369030    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="32988d90-426a-4a7b-b034-a4685db935b8" containerName="istio-proxy"
Nov 03 18:05:31.369046 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.369037    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2861a97b-ab99-4cfc-8a5c-bd2408972572" containerName="istio-proxy"
Nov 03 18:05:31.369046 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.369045    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="43fc8368-78b5-4856-a664-dcb061be31a5" containerName="payload-processing"
Nov 03 18:05:31.372332 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.372316    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"
Nov 03 18:05:31.401268 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.401238    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a188c3d-95cc-4c28-b885-510a8b8d3f65-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw\" (UID: \"4a188c3d-95cc-4c28-b885-510a8b8d3f65\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"
Nov 03 18:05:31.401410 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.401351    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s84c4\" (UniqueName: \"kubernetes.io/projected/4a188c3d-95cc-4c28-b885-510a8b8d3f65-kube-api-access-s84c4\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw\" (UID: \"4a188c3d-95cc-4c28-b885-510a8b8d3f65\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"
Nov 03 18:05:31.401604 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.401578    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a188c3d-95cc-4c28-b885-510a8b8d3f65-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw\" (UID: \"4a188c3d-95cc-4c28-b885-510a8b8d3f65\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"
Nov 03 18:05:31.401752 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.401724    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a188c3d-95cc-4c28-b885-510a8b8d3f65-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw\" (UID: \"4a188c3d-95cc-4c28-b885-510a8b8d3f65\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"
Nov 03 18:05:31.428544 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.428523    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"agc-cred-a75a150a8e19\""
Nov 03 18:05:31.480607 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.480574    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"]
Nov 03 18:05:31.502688 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.502651    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a188c3d-95cc-4c28-b885-510a8b8d3f65-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw\" (UID: \"4a188c3d-95cc-4c28-b885-510a8b8d3f65\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"
Nov 03 18:05:31.502841 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.502793    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a188c3d-95cc-4c28-b885-510a8b8d3f65-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw\" (UID: \"4a188c3d-95cc-4c28-b885-510a8b8d3f65\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"
Nov 03 18:05:31.502890 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.502859    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a188c3d-95cc-4c28-b885-510a8b8d3f65-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw\" (UID: \"4a188c3d-95cc-4c28-b885-510a8b8d3f65\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"
Nov 03 18:05:31.502927 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.502894    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s84c4\" (UniqueName: \"kubernetes.io/projected/4a188c3d-95cc-4c28-b885-510a8b8d3f65-kube-api-access-s84c4\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw\" (UID: \"4a188c3d-95cc-4c28-b885-510a8b8d3f65\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"
Nov 03 18:05:31.503572 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.503555    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a188c3d-95cc-4c28-b885-510a8b8d3f65-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw\" (UID: \"4a188c3d-95cc-4c28-b885-510a8b8d3f65\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"
Nov 03 18:05:31.503629 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.503564    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a188c3d-95cc-4c28-b885-510a8b8d3f65-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw\" (UID: \"4a188c3d-95cc-4c28-b885-510a8b8d3f65\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"
Nov 03 18:05:31.505318 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.505301    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a188c3d-95cc-4c28-b885-510a8b8d3f65-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw\" (UID: \"4a188c3d-95cc-4c28-b885-510a8b8d3f65\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"
Nov 03 18:05:31.566490 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.566460    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s84c4\" (UniqueName: \"kubernetes.io/projected/4a188c3d-95cc-4c28-b885-510a8b8d3f65-kube-api-access-s84c4\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw\" (UID: \"4a188c3d-95cc-4c28-b885-510a8b8d3f65\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"
Nov 03 18:05:31.685042 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:31.684958    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"
Nov 03 18:05:32.073039 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:05:32.073003    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a188c3d_95cc_4c28_b885_510a8b8d3f65.slice/crio-ffb7caf38f3627b192904a4efa57d0119f73ee34280e212e01e5168164a4bdc9 WatchSource:0}: Error finding container ffb7caf38f3627b192904a4efa57d0119f73ee34280e212e01e5168164a4bdc9: Status 404 returned error can't find the container with id ffb7caf38f3627b192904a4efa57d0119f73ee34280e212e01e5168164a4bdc9
Nov 03 18:05:32.075505 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:32.075428    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"]
Nov 03 18:05:32.251061 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:32.251022    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw" event={"ID":"4a188c3d-95cc-4c28-b885-510a8b8d3f65","Type":"ContainerStarted","Data":"add08f281dce0920895aad182e8d63a1e7b3c8c879831ba14629b10c1a3e4e91"}
Nov 03 18:05:32.251492 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:32.251128    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"
Nov 03 18:05:32.251492 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:32.251148    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw" event={"ID":"4a188c3d-95cc-4c28-b885-510a8b8d3f65","Type":"ContainerStarted","Data":"ffb7caf38f3627b192904a4efa57d0119f73ee34280e212e01e5168164a4bdc9"}
Nov 03 18:05:32.375287 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:32.375147    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw" podStartSLOduration=1.375128148 podStartE2EDuration="1.375128148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:32.373689751 +0000 UTC m=+2873.171402533" watchObservedRunningTime="2026-11-03 18:05:32.375128148 +0000 UTC m=+2873.172840933"
Nov 03 18:05:33.061981 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:33.061941    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:36.860140 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:05:36.860095    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:05:37.854029 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:37.853995    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:42.769471 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:42.769438    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:43.260484 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:43.260445    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"
Nov 03 18:05:43.436497 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:43.436464    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"]
Nov 03 18:05:46.808509 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:46.808478    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:47.572332 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:47.572233    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:48.437149 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:48.437085    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq" podUID="896c29df-4e9f-4b56-afc2-c17fd1d15e2c" containerName="payload-processing" containerID="cri-o://b774c5a0e4f0a8dee5fd428bc004941f965dd163343c04b9de056c63b9b1b966" gracePeriod=25
Nov 03 18:05:48.690979 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:48.690921    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"
Nov 03 18:05:48.777887 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:48.777857    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-plugins-config-volume\") pod \"896c29df-4e9f-4b56-afc2-c17fd1d15e2c\" (UID: \"896c29df-4e9f-4b56-afc2-c17fd1d15e2c\") "
Nov 03 18:05:48.778081 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:48.777924    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-provider-credentials\") pod \"896c29df-4e9f-4b56-afc2-c17fd1d15e2c\" (UID: \"896c29df-4e9f-4b56-afc2-c17fd1d15e2c\") "
Nov 03 18:05:48.778081 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:48.777983    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-routing-overlay-volume\") pod \"896c29df-4e9f-4b56-afc2-c17fd1d15e2c\" (UID: \"896c29df-4e9f-4b56-afc2-c17fd1d15e2c\") "
Nov 03 18:05:48.778215 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:48.778130    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-kube-api-access-lmt5d\" (UniqueName: \"kubernetes.io/projected/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-kube-api-access-lmt5d\") pod \"896c29df-4e9f-4b56-afc2-c17fd1d15e2c\" (UID: \"896c29df-4e9f-4b56-afc2-c17fd1d15e2c\") "
Nov 03 18:05:48.778394 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:48.778347    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-plugins-config-volume" pod "896c29df-4e9f-4b56-afc2-c17fd1d15e2c" (UID: "896c29df-4e9f-4b56-afc2-c17fd1d15e2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:05:48.778491 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:48.778370    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-routing-overlay-volume" pod "896c29df-4e9f-4b56-afc2-c17fd1d15e2c" (UID: "896c29df-4e9f-4b56-afc2-c17fd1d15e2c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:05:48.778550 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:48.778532    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:48.778590 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:48.778553    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:48.780072 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:48.780047    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-provider-credentials" pod "896c29df-4e9f-4b56-afc2-c17fd1d15e2c" (UID: "896c29df-4e9f-4b56-afc2-c17fd1d15e2c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:05:48.780146 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:48.780121    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-kube-api-access-lmt5d" pod "896c29df-4e9f-4b56-afc2-c17fd1d15e2c" (UID: "896c29df-4e9f-4b56-afc2-c17fd1d15e2c"). InnerVolumeSpecName "kube-api-access-lmt5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:05:48.880017 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:48.879983    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmt5d\" (UniqueName: \"kubernetes.io/projected/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-kube-api-access-lmt5d\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:48.880017 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:48.880012    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/896c29df-4e9f-4b56-afc2-c17fd1d15e2c-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:05:49.348204 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:49.348173    6059 generic.go:356] "Generic (PLEG): container finished" podID="896c29df-4e9f-4b56-afc2-c17fd1d15e2c" containerID="b774c5a0e4f0a8dee5fd428bc004941f965dd163343c04b9de056c63b9b1b966" exitCode=0
Nov 03 18:05:49.348385 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:49.348230    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"
Nov 03 18:05:49.348385 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:49.348252    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq" event={"ID":"896c29df-4e9f-4b56-afc2-c17fd1d15e2c","Type":"ContainerDied","Data":"b774c5a0e4f0a8dee5fd428bc004941f965dd163343c04b9de056c63b9b1b966"}
Nov 03 18:05:49.348385 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:49.348317    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq" event={"ID":"896c29df-4e9f-4b56-afc2-c17fd1d15e2c","Type":"ContainerDied","Data":"9b2bf587dcf729b41a33a36a1a06042a70ecf1745bd8765e7916ce37f5287bf1"}
Nov 03 18:05:49.348385 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:49.348347    6059 scope.go:122] "RemoveContainer" containerID="b774c5a0e4f0a8dee5fd428bc004941f965dd163343c04b9de056c63b9b1b966"
Nov 03 18:05:49.356919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:49.356902    6059 scope.go:122] "RemoveContainer" containerID="b774c5a0e4f0a8dee5fd428bc004941f965dd163343c04b9de056c63b9b1b966"
Nov 03 18:05:49.357189 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:05:49.357169    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b774c5a0e4f0a8dee5fd428bc004941f965dd163343c04b9de056c63b9b1b966\": container with ID starting with b774c5a0e4f0a8dee5fd428bc004941f965dd163343c04b9de056c63b9b1b966 not found: ID does not exist" containerID="b774c5a0e4f0a8dee5fd428bc004941f965dd163343c04b9de056c63b9b1b966"
Nov 03 18:05:49.357246 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:49.357197    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b774c5a0e4f0a8dee5fd428bc004941f965dd163343c04b9de056c63b9b1b966"} err="failed to get container status \"b774c5a0e4f0a8dee5fd428bc004941f965dd163343c04b9de056c63b9b1b966\": rpc error: code = NotFound desc = could not find container \"b774c5a0e4f0a8dee5fd428bc004941f965dd163343c04b9de056c63b9b1b966\": container with ID starting with b774c5a0e4f0a8dee5fd428bc004941f965dd163343c04b9de056c63b9b1b966 not found: ID does not exist"
Nov 03 18:05:49.519254 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:49.519220    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"]
Nov 03 18:05:49.523246 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:49.523213    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bqz2tq"]
Nov 03 18:05:51.334332 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:51.334297    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="896c29df-4e9f-4b56-afc2-c17fd1d15e2c" path="/var/lib/kubelet/pods/896c29df-4e9f-4b56-afc2-c17fd1d15e2c/volumes"
Nov 03 18:05:52.356496 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:52.356465    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:05:57.262632 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:05:57.262569    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:01.107701 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:06:01.107665    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:01.285344 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:06:01.285312    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:02.118213 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:06:02.118170    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:06.861441 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:06:06.861406    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:06:06.947969 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:06:06.947931    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:11.804656 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:06:11.804561    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:16.676471 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:06:16.676418    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:21.355133 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:06:21.355095    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:26.183212 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:06:26.183155    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:30.942276 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:06:30.942239    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:35.743010 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:06:35.742973    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:36.860566 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:06:36.860531    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:06:40.655151 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:06:40.655111    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:45.458611 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:06:45.458579    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:47.158884 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:06:47.158845    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:50.257939 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:06:50.257906    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:55.081741 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:06:55.081684    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:06:59.861051 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:06:59.861015    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:04.655872 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:07:04.655839    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:06.860606 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:07:06.860560    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:07:09.551567 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:07:09.551532    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:14.356306 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:07:14.356269    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:19.162841 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:07:19.162737    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:24.046140 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:07:24.046108    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:28.875715 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:07:28.875652    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:30.272592 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:07:30.272560    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:30.454972 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:07:30.454936    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:30.675414 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:07:30.675335    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:30.852141 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:07:30.852108    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:33.671889 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:07:33.671851    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:36.860597 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:07:36.860549    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:07:38.523915 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:07:38.523873    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:39.811755 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:07:39.811726    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:07:39.822771 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:07:39.822744    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:07:39.829500 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:07:39.829478    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:07:39.840147 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:07:39.840130    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:07:43.451784 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:07:43.451751    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:47.491566 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:07:47.491527    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:48.245224 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:07:48.245186    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:53.154060 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:07:53.154018    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:07:57.981931 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:07:57.981896    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:02.769565 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:08:02.769529    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:06.860443 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:08:06.860401    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:08:07.653684 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:08:07.653629    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:12.458552 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:08:12.458522    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:17.369405 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:08:17.369344    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:22.148615 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:08:22.148575    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:26.962080 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:08:26.962044    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:31.762799 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:08:31.762757    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:36.647099 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:08:36.647064    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:36.859923 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:08:36.859885    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:08:41.435773 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:08:41.435739    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:46.235802 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:08:46.235762    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:47.829208 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:08:47.829131    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:51.147310 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:08:51.147260    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:08:55.843604 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:08:55.843563    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:00.648665 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:09:00.648628    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:05.468464 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:09:05.468431    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:06.860907 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:09:06.860865    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:09:10.266097 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:09:10.266059    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:15.059588 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:09:15.059552    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:19.965237 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:09:19.965198    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:24.752000 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:09:24.751966    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:27.860929 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:09:27.860890    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:29.540331 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:09:29.540282    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:34.351665 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:09:34.351625    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:36.860552 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:09:36.860517    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:09:39.149233 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:09:39.149194    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:43.953618 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:09:43.953565    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:48.189867 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:09:48.189829    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:48.373304 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:09:48.373259    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:48.853536 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:09:48.853504    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:53.652001 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:09:53.651969    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:09:58.555500 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:09:58.555461    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:03.457366 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:10:03.457321    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:06.860558 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:10:06.860525    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:10:08.250731 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:10:08.250694    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:13.066176 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:10:13.066136    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:17.955907 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:10:17.955809    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:22.778916 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:10:22.778875    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:27.556328 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:10:27.556283    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:32.451694 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:10:32.451653    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:36.862457 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:10:36.862419    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:10:37.243820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:10:37.243783    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:42.050516 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:10:42.050480    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:46.856868 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:10:46.856831    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:48.530286 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:10:48.530248    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:51.648215 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:10:51.648179    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:10:56.459943 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:10:56.459901    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:01.331776 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:11:01.331737    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:06.175915 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:11:06.175875    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:06.861871 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:11:06.861829    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:11:10.949997 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:11:10.949960    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:15.769992 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:11:15.769953    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:20.578800 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:11:20.578766    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:25.339060 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:11:25.339025    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:30.161351 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:11:30.161310    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:32.198765 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:11:32.198731    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:34.938868 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:11:34.938834    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:36.861836 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:11:36.861800    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:11:39.750633 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:11:39.750593    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:44.576976 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:11:44.576935    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:48.885946 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:11:48.885907    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:49.366412 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:11:49.366348    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:56.625765 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:11:56.625727    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:11:59.154404 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:11:59.154340    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:03.947409 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:03.947340    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:06.861617 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:12:06.861575    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:12:08.752574 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:08.752536    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:10.756328 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:10.756294    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58"]
Nov 03 18:12:10.756808 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:10.756727    6059 cpu_manager.go:414] "containerMap: removing container" podUID="896c29df-4e9f-4b56-afc2-c17fd1d15e2c" containerName="payload-processing"
Nov 03 18:12:10.756808 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:10.756741    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="896c29df-4e9f-4b56-afc2-c17fd1d15e2c" containerName="payload-processing"
Nov 03 18:12:10.756808 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:10.756776    6059 cpu_manager.go:414] "containerMap: removing container" podUID="2861a97b-ab99-4cfc-8a5c-bd2408972572" containerName="istio-proxy"
Nov 03 18:12:10.756808 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:10.756781    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2861a97b-ab99-4cfc-8a5c-bd2408972572" containerName="istio-proxy"
Nov 03 18:12:10.756952 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:10.756857    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="896c29df-4e9f-4b56-afc2-c17fd1d15e2c" containerName="payload-processing"
Nov 03 18:12:10.760087 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:10.760060    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58"
Nov 03 18:12:10.867825 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:10.867780    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:12:10.867825 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:10.867781    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:12:10.867825 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:10.867780    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:12:10.868091 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:10.867781    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 18:12:10.869706 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:10.869683    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58"]
Nov 03 18:12:10.919726 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:10.919695    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcxw9\" (UniqueName: \"kubernetes.io/projected/1aa61d3c-fc87-4b7b-a873-348eb918ffd7-kube-api-access-dcxw9\") pod \"maas-api-revoke-sub-34bf0720-thh58\" (UID: \"1aa61d3c-fc87-4b7b-a873-348eb918ffd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58"
Nov 03 18:12:10.919949 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:10.919746    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa61d3c-fc87-4b7b-a873-348eb918ffd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34bf0720-thh58\" (UID: \"1aa61d3c-fc87-4b7b-a873-348eb918ffd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58"
Nov 03 18:12:10.919949 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:10.919915    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa61d3c-fc87-4b7b-a873-348eb918ffd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34bf0720-thh58\" (UID: \"1aa61d3c-fc87-4b7b-a873-348eb918ffd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58"
Nov 03 18:12:11.028218 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:11.028120    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcxw9\" (UniqueName: \"kubernetes.io/projected/1aa61d3c-fc87-4b7b-a873-348eb918ffd7-kube-api-access-dcxw9\") pod \"maas-api-revoke-sub-34bf0720-thh58\" (UID: \"1aa61d3c-fc87-4b7b-a873-348eb918ffd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58"
Nov 03 18:12:11.028218 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:11.028188    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa61d3c-fc87-4b7b-a873-348eb918ffd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34bf0720-thh58\" (UID: \"1aa61d3c-fc87-4b7b-a873-348eb918ffd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58"
Nov 03 18:12:11.028503 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:11.028275    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa61d3c-fc87-4b7b-a873-348eb918ffd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34bf0720-thh58\" (UID: \"1aa61d3c-fc87-4b7b-a873-348eb918ffd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58"
Nov 03 18:12:11.029106 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:11.029079    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa61d3c-fc87-4b7b-a873-348eb918ffd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34bf0720-thh58\" (UID: \"1aa61d3c-fc87-4b7b-a873-348eb918ffd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58"
Nov 03 18:12:11.031212 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:11.031190    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa61d3c-fc87-4b7b-a873-348eb918ffd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34bf0720-thh58\" (UID: \"1aa61d3c-fc87-4b7b-a873-348eb918ffd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58"
Nov 03 18:12:11.091967 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:11.091923    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcxw9\" (UniqueName: \"kubernetes.io/projected/1aa61d3c-fc87-4b7b-a873-348eb918ffd7-kube-api-access-dcxw9\") pod \"maas-api-revoke-sub-34bf0720-thh58\" (UID: \"1aa61d3c-fc87-4b7b-a873-348eb918ffd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58"
Nov 03 18:12:11.373277 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:11.373184    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58"
Nov 03 18:12:11.746982 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:11.746953    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58"]
Nov 03 18:12:12.355743 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:12.355644    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58" event={"ID":"1aa61d3c-fc87-4b7b-a873-348eb918ffd7","Type":"ContainerStarted","Data":"7cf12d1684f8feee52bd767edcab34433a40159931fa8358e55d2ddccca8699c"}
Nov 03 18:12:12.355743 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:12.355707    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58" event={"ID":"1aa61d3c-fc87-4b7b-a873-348eb918ffd7","Type":"ContainerStarted","Data":"667e26ba5ca20389182d21614a05c9c65ac38a382080518200754a52ad52e1f0"}
Nov 03 18:12:12.476209 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:12.476151    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58" podStartSLOduration=2.476134505 podStartE2EDuration="2.476134505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:12:12.47515141 +0000 UTC m=+3273.272864192" watchObservedRunningTime="2026-11-03 18:12:12.476134505 +0000 UTC m=+3273.273847285"
Nov 03 18:12:13.362191 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:13.362151    6059 generic.go:356] "Generic (PLEG): container finished" podID="1aa61d3c-fc87-4b7b-a873-348eb918ffd7" containerID="7cf12d1684f8feee52bd767edcab34433a40159931fa8358e55d2ddccca8699c" exitCode=0
Nov 03 18:12:13.362615 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:13.362236    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58" event={"ID":"1aa61d3c-fc87-4b7b-a873-348eb918ffd7","Type":"ContainerDied","Data":"7cf12d1684f8feee52bd767edcab34433a40159931fa8358e55d2ddccca8699c"}
Nov 03 18:12:13.661702 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:13.661593    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:14.510247 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:14.510216    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58"
Nov 03 18:12:14.564706 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:14.564666    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa61d3c-fc87-4b7b-a873-348eb918ffd7-kube-api-access-dcxw9\" (UniqueName: \"kubernetes.io/projected/1aa61d3c-fc87-4b7b-a873-348eb918ffd7-kube-api-access-dcxw9\") pod \"1aa61d3c-fc87-4b7b-a873-348eb918ffd7\" (UID: \"1aa61d3c-fc87-4b7b-a873-348eb918ffd7\") "
Nov 03 18:12:14.564903 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:14.564752    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aa61d3c-fc87-4b7b-a873-348eb918ffd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa61d3c-fc87-4b7b-a873-348eb918ffd7-maas-api-openshift-ca\") pod \"1aa61d3c-fc87-4b7b-a873-348eb918ffd7\" (UID: \"1aa61d3c-fc87-4b7b-a873-348eb918ffd7\") "
Nov 03 18:12:14.564903 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:14.564784    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aa61d3c-fc87-4b7b-a873-348eb918ffd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa61d3c-fc87-4b7b-a873-348eb918ffd7-maas-api-xks-ca\") pod \"1aa61d3c-fc87-4b7b-a873-348eb918ffd7\" (UID: \"1aa61d3c-fc87-4b7b-a873-348eb918ffd7\") "
Nov 03 18:12:14.565111 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:14.565088    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa61d3c-fc87-4b7b-a873-348eb918ffd7-maas-api-openshift-ca" pod "1aa61d3c-fc87-4b7b-a873-348eb918ffd7" (UID: "1aa61d3c-fc87-4b7b-a873-348eb918ffd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:14.566861 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:14.566833    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aa61d3c-fc87-4b7b-a873-348eb918ffd7-maas-api-xks-ca" pod "1aa61d3c-fc87-4b7b-a873-348eb918ffd7" (UID: "1aa61d3c-fc87-4b7b-a873-348eb918ffd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:12:14.566977 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:14.566952    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa61d3c-fc87-4b7b-a873-348eb918ffd7-kube-api-access-dcxw9" pod "1aa61d3c-fc87-4b7b-a873-348eb918ffd7" (UID: "1aa61d3c-fc87-4b7b-a873-348eb918ffd7"). InnerVolumeSpecName "kube-api-access-dcxw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:14.666021 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:14.665924    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcxw9\" (UniqueName: \"kubernetes.io/projected/1aa61d3c-fc87-4b7b-a873-348eb918ffd7-kube-api-access-dcxw9\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:14.666021 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:14.665964    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa61d3c-fc87-4b7b-a873-348eb918ffd7-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:14.666021 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:14.665978    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa61d3c-fc87-4b7b-a873-348eb918ffd7-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:15.380388 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:15.380339    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58"
Nov 03 18:12:15.380388 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:15.380342    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58" event={"ID":"1aa61d3c-fc87-4b7b-a873-348eb918ffd7","Type":"ContainerDied","Data":"667e26ba5ca20389182d21614a05c9c65ac38a382080518200754a52ad52e1f0"}
Nov 03 18:12:15.380645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:15.380426    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="667e26ba5ca20389182d21614a05c9c65ac38a382080518200754a52ad52e1f0"
Nov 03 18:12:18.450870 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:18.450832    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:23.438911 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:23.438873    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:23.623121 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:23.623089    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:24.077156 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.077114    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db637277-44w92"]
Nov 03 18:12:24.077676 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.077656    6059 cpu_manager.go:414] "containerMap: removing container" podUID="1aa61d3c-fc87-4b7b-a873-348eb918ffd7" containerName="revoke-keys"
Nov 03 18:12:24.077783 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.077676    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa61d3c-fc87-4b7b-a873-348eb918ffd7" containerName="revoke-keys"
Nov 03 18:12:24.077845 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.077808    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa61d3c-fc87-4b7b-a873-348eb918ffd7" containerName="revoke-keys"
Nov 03 18:12:24.081307 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.081285    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db637277-44w92"
Nov 03 18:12:24.144199 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.144167    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db637277-44w92"]
Nov 03 18:12:24.164958 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.164925    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c306988-214c-4fdc-8548-7b19ca819d66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db637277-44w92\" (UID: \"9c306988-214c-4fdc-8548-7b19ca819d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db637277-44w92"
Nov 03 18:12:24.164958 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.164967    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c306988-214c-4fdc-8548-7b19ca819d66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db637277-44w92\" (UID: \"9c306988-214c-4fdc-8548-7b19ca819d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db637277-44w92"
Nov 03 18:12:24.165217 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.165065    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crjlf\" (UniqueName: \"kubernetes.io/projected/9c306988-214c-4fdc-8548-7b19ca819d66-kube-api-access-crjlf\") pod \"maas-api-revoke-sub-db637277-44w92\" (UID: \"9c306988-214c-4fdc-8548-7b19ca819d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db637277-44w92"
Nov 03 18:12:24.193482 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.193446    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:12:24.193743 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.193718    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:12:24.193856 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.193743    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 18:12:24.193856 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.193748    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:12:24.266124 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.266084    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c306988-214c-4fdc-8548-7b19ca819d66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db637277-44w92\" (UID: \"9c306988-214c-4fdc-8548-7b19ca819d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db637277-44w92"
Nov 03 18:12:24.266332 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.266214    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c306988-214c-4fdc-8548-7b19ca819d66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db637277-44w92\" (UID: \"9c306988-214c-4fdc-8548-7b19ca819d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db637277-44w92"
Nov 03 18:12:24.266332 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.266282    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crjlf\" (UniqueName: \"kubernetes.io/projected/9c306988-214c-4fdc-8548-7b19ca819d66-kube-api-access-crjlf\") pod \"maas-api-revoke-sub-db637277-44w92\" (UID: \"9c306988-214c-4fdc-8548-7b19ca819d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db637277-44w92"
Nov 03 18:12:24.266970 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.266944    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c306988-214c-4fdc-8548-7b19ca819d66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db637277-44w92\" (UID: \"9c306988-214c-4fdc-8548-7b19ca819d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db637277-44w92"
Nov 03 18:12:24.268586 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.268567    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c306988-214c-4fdc-8548-7b19ca819d66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db637277-44w92\" (UID: \"9c306988-214c-4fdc-8548-7b19ca819d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db637277-44w92"
Nov 03 18:12:24.329215 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.329134    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crjlf\" (UniqueName: \"kubernetes.io/projected/9c306988-214c-4fdc-8548-7b19ca819d66-kube-api-access-crjlf\") pod \"maas-api-revoke-sub-db637277-44w92\" (UID: \"9c306988-214c-4fdc-8548-7b19ca819d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db637277-44w92"
Nov 03 18:12:24.392747 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.392711    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"]
Nov 03 18:12:24.394592 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.394570    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db637277-44w92"
Nov 03 18:12:24.396763 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.396742    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"
Nov 03 18:12:24.459019 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.458805    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"]
Nov 03 18:12:24.468766 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.468728    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0480008-8d47-450e-9083-36a9239da0e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n\" (UID: \"e0480008-8d47-450e-9083-36a9239da0e7\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"
Nov 03 18:12:24.468918 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.468794    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0480008-8d47-450e-9083-36a9239da0e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n\" (UID: \"e0480008-8d47-450e-9083-36a9239da0e7\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"
Nov 03 18:12:24.468918 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.468872    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmtt4\" (UniqueName: \"kubernetes.io/projected/e0480008-8d47-450e-9083-36a9239da0e7-kube-api-access-wmtt4\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n\" (UID: \"e0480008-8d47-450e-9083-36a9239da0e7\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"
Nov 03 18:12:24.468993 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.468922    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0480008-8d47-450e-9083-36a9239da0e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n\" (UID: \"e0480008-8d47-450e-9083-36a9239da0e7\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"
Nov 03 18:12:24.569942 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.569905    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0480008-8d47-450e-9083-36a9239da0e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n\" (UID: \"e0480008-8d47-450e-9083-36a9239da0e7\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"
Nov 03 18:12:24.569942 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.569948    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0480008-8d47-450e-9083-36a9239da0e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n\" (UID: \"e0480008-8d47-450e-9083-36a9239da0e7\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"
Nov 03 18:12:24.570173 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.569987    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmtt4\" (UniqueName: \"kubernetes.io/projected/e0480008-8d47-450e-9083-36a9239da0e7-kube-api-access-wmtt4\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n\" (UID: \"e0480008-8d47-450e-9083-36a9239da0e7\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"
Nov 03 18:12:24.570173 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.570015    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0480008-8d47-450e-9083-36a9239da0e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n\" (UID: \"e0480008-8d47-450e-9083-36a9239da0e7\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"
Nov 03 18:12:24.570703 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.570681    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0480008-8d47-450e-9083-36a9239da0e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n\" (UID: \"e0480008-8d47-450e-9083-36a9239da0e7\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"
Nov 03 18:12:24.570821 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.570778    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0480008-8d47-450e-9083-36a9239da0e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n\" (UID: \"e0480008-8d47-450e-9083-36a9239da0e7\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"
Nov 03 18:12:24.572922 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.572891    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0480008-8d47-450e-9083-36a9239da0e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n\" (UID: \"e0480008-8d47-450e-9083-36a9239da0e7\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"
Nov 03 18:12:24.633401 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.633308    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmtt4\" (UniqueName: \"kubernetes.io/projected/e0480008-8d47-450e-9083-36a9239da0e7-kube-api-access-wmtt4\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n\" (UID: \"e0480008-8d47-450e-9083-36a9239da0e7\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"
Nov 03 18:12:24.731097 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.731055    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"
Nov 03 18:12:24.769445 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:12:24.769409    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c306988_214c_4fdc_8548_7b19ca819d66.slice/crio-8103197ed42cbbb0de14643ad447b1180e2a9b81f28c2c3ba5e45694f60f39b2 WatchSource:0}: Error finding container 8103197ed42cbbb0de14643ad447b1180e2a9b81f28c2c3ba5e45694f60f39b2: Status 404 returned error can't find the container with id 8103197ed42cbbb0de14643ad447b1180e2a9b81f28c2c3ba5e45694f60f39b2
Nov 03 18:12:24.770272 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:24.770245    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db637277-44w92"]
Nov 03 18:12:25.110833 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:12:25.110796    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0480008_8d47_450e_9083_36a9239da0e7.slice/crio-c5652898ff3d445322b55282fa19e75fbefe4d76e724afd1337b098c1d909567 WatchSource:0}: Error finding container c5652898ff3d445322b55282fa19e75fbefe4d76e724afd1337b098c1d909567: Status 404 returned error can't find the container with id c5652898ff3d445322b55282fa19e75fbefe4d76e724afd1337b098c1d909567
Nov 03 18:12:25.113021 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:25.112996    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"]
Nov 03 18:12:25.438829 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:25.438792    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n" event={"ID":"e0480008-8d47-450e-9083-36a9239da0e7","Type":"ContainerStarted","Data":"a37af67563e37eb3f20a134341a9aa75fbde35bf06cce84b10ead3f9b570298b"}
Nov 03 18:12:25.439058 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:25.438846    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n" event={"ID":"e0480008-8d47-450e-9083-36a9239da0e7","Type":"ContainerStarted","Data":"c5652898ff3d445322b55282fa19e75fbefe4d76e724afd1337b098c1d909567"}
Nov 03 18:12:25.439058 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:25.438893    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"
Nov 03 18:12:25.440583 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:25.440559    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db637277-44w92" event={"ID":"9c306988-214c-4fdc-8548-7b19ca819d66","Type":"ContainerStarted","Data":"71b8205363f734940986b9f4fc713a96911c449ffa753a49b1e8ca20071d30ac"}
Nov 03 18:12:25.440693 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:25.440598    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db637277-44w92" event={"ID":"9c306988-214c-4fdc-8548-7b19ca819d66","Type":"ContainerStarted","Data":"8103197ed42cbbb0de14643ad447b1180e2a9b81f28c2c3ba5e45694f60f39b2"}
Nov 03 18:12:25.560966 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:25.560908    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n" podStartSLOduration=1.5608917020000002 podStartE2EDuration="1.560891702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:12:25.559630779 +0000 UTC m=+3286.357343561" watchObservedRunningTime="2026-11-03 18:12:25.560891702 +0000 UTC m=+3286.358604480"
Nov 03 18:12:26.448685 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:26.448647    6059 generic.go:356] "Generic (PLEG): container finished" podID="9c306988-214c-4fdc-8548-7b19ca819d66" containerID="71b8205363f734940986b9f4fc713a96911c449ffa753a49b1e8ca20071d30ac" exitCode=0
Nov 03 18:12:26.448892 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:26.448735    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db637277-44w92" event={"ID":"9c306988-214c-4fdc-8548-7b19ca819d66","Type":"ContainerDied","Data":"71b8205363f734940986b9f4fc713a96911c449ffa753a49b1e8ca20071d30ac"}
Nov 03 18:12:27.593463 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:27.593435    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db637277-44w92"
Nov 03 18:12:27.699081 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:27.699036    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c306988-214c-4fdc-8548-7b19ca819d66-kube-api-access-crjlf\" (UniqueName: \"kubernetes.io/projected/9c306988-214c-4fdc-8548-7b19ca819d66-kube-api-access-crjlf\") pod \"9c306988-214c-4fdc-8548-7b19ca819d66\" (UID: \"9c306988-214c-4fdc-8548-7b19ca819d66\") "
Nov 03 18:12:27.699291 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:27.699118    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c306988-214c-4fdc-8548-7b19ca819d66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c306988-214c-4fdc-8548-7b19ca819d66-maas-api-xks-ca\") pod \"9c306988-214c-4fdc-8548-7b19ca819d66\" (UID: \"9c306988-214c-4fdc-8548-7b19ca819d66\") "
Nov 03 18:12:27.699291 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:27.699186    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c306988-214c-4fdc-8548-7b19ca819d66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c306988-214c-4fdc-8548-7b19ca819d66-maas-api-openshift-ca\") pod \"9c306988-214c-4fdc-8548-7b19ca819d66\" (UID: \"9c306988-214c-4fdc-8548-7b19ca819d66\") "
Nov 03 18:12:27.699579 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:27.699555    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c306988-214c-4fdc-8548-7b19ca819d66-maas-api-openshift-ca" pod "9c306988-214c-4fdc-8548-7b19ca819d66" (UID: "9c306988-214c-4fdc-8548-7b19ca819d66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:27.701225 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:27.701203    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c306988-214c-4fdc-8548-7b19ca819d66-maas-api-xks-ca" pod "9c306988-214c-4fdc-8548-7b19ca819d66" (UID: "9c306988-214c-4fdc-8548-7b19ca819d66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:12:27.701368 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:27.701252    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c306988-214c-4fdc-8548-7b19ca819d66-kube-api-access-crjlf" pod "9c306988-214c-4fdc-8548-7b19ca819d66" (UID: "9c306988-214c-4fdc-8548-7b19ca819d66"). InnerVolumeSpecName "kube-api-access-crjlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:27.800450 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:27.800414    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crjlf\" (UniqueName: \"kubernetes.io/projected/9c306988-214c-4fdc-8548-7b19ca819d66-kube-api-access-crjlf\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:27.800450 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:27.800445    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c306988-214c-4fdc-8548-7b19ca819d66-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:27.800450 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:27.800455    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c306988-214c-4fdc-8548-7b19ca819d66-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:28.156135 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:28.156047    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:28.460304 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:28.460265    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db637277-44w92" event={"ID":"9c306988-214c-4fdc-8548-7b19ca819d66","Type":"ContainerDied","Data":"8103197ed42cbbb0de14643ad447b1180e2a9b81f28c2c3ba5e45694f60f39b2"}
Nov 03 18:12:28.460544 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:28.460327    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8103197ed42cbbb0de14643ad447b1180e2a9b81f28c2c3ba5e45694f60f39b2"
Nov 03 18:12:28.460544 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:28.460282    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db637277-44w92"
Nov 03 18:12:32.962107 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:32.962063    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:35.793332 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:35.793295    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aece27a8-jg7b7"]
Nov 03 18:12:35.793767 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:35.793748    6059 cpu_manager.go:414] "containerMap: removing container" podUID="9c306988-214c-4fdc-8548-7b19ca819d66" containerName="revoke-keys"
Nov 03 18:12:35.793767 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:35.793760    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c306988-214c-4fdc-8548-7b19ca819d66" containerName="revoke-keys"
Nov 03 18:12:35.793877 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:35.793865    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c306988-214c-4fdc-8548-7b19ca819d66" containerName="revoke-keys"
Nov 03 18:12:35.798378 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:35.798341    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aece27a8-jg7b7"
Nov 03 18:12:35.854193 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:35.854158    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:12:35.885799 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:35.885756    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ff32de-2054-4ef4-b11a-63d59ebe56bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aece27a8-jg7b7\" (UID: \"b7ff32de-2054-4ef4-b11a-63d59ebe56bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aece27a8-jg7b7"
Nov 03 18:12:35.885799 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:35.885798    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v8sj\" (UniqueName: \"kubernetes.io/projected/b7ff32de-2054-4ef4-b11a-63d59ebe56bf-kube-api-access-8v8sj\") pod \"maas-api-revoke-sub-aece27a8-jg7b7\" (UID: \"b7ff32de-2054-4ef4-b11a-63d59ebe56bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aece27a8-jg7b7"
Nov 03 18:12:35.886077 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:35.886014    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ff32de-2054-4ef4-b11a-63d59ebe56bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aece27a8-jg7b7\" (UID: \"b7ff32de-2054-4ef4-b11a-63d59ebe56bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aece27a8-jg7b7"
Nov 03 18:12:35.905850 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:35.905818    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:12:35.906022 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:35.905891    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:12:35.906022 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:35.905916    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 18:12:35.907956 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:35.907937    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aece27a8-jg7b7"]
Nov 03 18:12:35.987130 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:35.987088    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ff32de-2054-4ef4-b11a-63d59ebe56bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aece27a8-jg7b7\" (UID: \"b7ff32de-2054-4ef4-b11a-63d59ebe56bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aece27a8-jg7b7"
Nov 03 18:12:35.987309 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:35.987145    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v8sj\" (UniqueName: \"kubernetes.io/projected/b7ff32de-2054-4ef4-b11a-63d59ebe56bf-kube-api-access-8v8sj\") pod \"maas-api-revoke-sub-aece27a8-jg7b7\" (UID: \"b7ff32de-2054-4ef4-b11a-63d59ebe56bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aece27a8-jg7b7"
Nov 03 18:12:35.987309 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:35.987285    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ff32de-2054-4ef4-b11a-63d59ebe56bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aece27a8-jg7b7\" (UID: \"b7ff32de-2054-4ef4-b11a-63d59ebe56bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aece27a8-jg7b7"
Nov 03 18:12:35.987809 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:35.987788    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ff32de-2054-4ef4-b11a-63d59ebe56bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aece27a8-jg7b7\" (UID: \"b7ff32de-2054-4ef4-b11a-63d59ebe56bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aece27a8-jg7b7"
Nov 03 18:12:35.989601 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:35.989574    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ff32de-2054-4ef4-b11a-63d59ebe56bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aece27a8-jg7b7\" (UID: \"b7ff32de-2054-4ef4-b11a-63d59ebe56bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aece27a8-jg7b7"
Nov 03 18:12:36.050814 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:36.050740    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v8sj\" (UniqueName: \"kubernetes.io/projected/b7ff32de-2054-4ef4-b11a-63d59ebe56bf-kube-api-access-8v8sj\") pod \"maas-api-revoke-sub-aece27a8-jg7b7\" (UID: \"b7ff32de-2054-4ef4-b11a-63d59ebe56bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aece27a8-jg7b7"
Nov 03 18:12:36.111698 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:36.111655    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aece27a8-jg7b7"
Nov 03 18:12:36.452389 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:36.452285    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n" podUID="e0480008-8d47-450e-9083-36a9239da0e7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:12:36.494458 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:36.494427    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aece27a8-jg7b7"]
Nov 03 18:12:36.505668 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:36.505636    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aece27a8-jg7b7" event={"ID":"b7ff32de-2054-4ef4-b11a-63d59ebe56bf","Type":"ContainerStarted","Data":"79a76966de5c4e1a420638e1349219ee552d603e1c46faac1df0a9c9ae6ce414"}
Nov 03 18:12:36.861044 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:12:36.860998    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:12:37.513121 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:37.513085    6059 generic.go:356] "Generic (PLEG): container finished" podID="b7ff32de-2054-4ef4-b11a-63d59ebe56bf" containerID="ca1790540895224ba1e600ca4d5afdb75b3caacd69c0f1040191f97ac24dd92e" exitCode=0
Nov 03 18:12:37.513317 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:37.513169    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aece27a8-jg7b7" event={"ID":"b7ff32de-2054-4ef4-b11a-63d59ebe56bf","Type":"ContainerDied","Data":"ca1790540895224ba1e600ca4d5afdb75b3caacd69c0f1040191f97ac24dd92e"}
Nov 03 18:12:37.770202 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:37.770109    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:38.658819 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:38.658792    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aece27a8-jg7b7"
Nov 03 18:12:38.814606 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:38.814516    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7ff32de-2054-4ef4-b11a-63d59ebe56bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ff32de-2054-4ef4-b11a-63d59ebe56bf-maas-api-xks-ca\") pod \"b7ff32de-2054-4ef4-b11a-63d59ebe56bf\" (UID: \"b7ff32de-2054-4ef4-b11a-63d59ebe56bf\") "
Nov 03 18:12:38.814606 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:38.814578    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7ff32de-2054-4ef4-b11a-63d59ebe56bf-kube-api-access-8v8sj\" (UniqueName: \"kubernetes.io/projected/b7ff32de-2054-4ef4-b11a-63d59ebe56bf-kube-api-access-8v8sj\") pod \"b7ff32de-2054-4ef4-b11a-63d59ebe56bf\" (UID: \"b7ff32de-2054-4ef4-b11a-63d59ebe56bf\") "
Nov 03 18:12:38.814847 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:38.814621    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7ff32de-2054-4ef4-b11a-63d59ebe56bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ff32de-2054-4ef4-b11a-63d59ebe56bf-maas-api-openshift-ca\") pod \"b7ff32de-2054-4ef4-b11a-63d59ebe56bf\" (UID: \"b7ff32de-2054-4ef4-b11a-63d59ebe56bf\") "
Nov 03 18:12:38.815011 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:38.814987    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7ff32de-2054-4ef4-b11a-63d59ebe56bf-maas-api-openshift-ca" pod "b7ff32de-2054-4ef4-b11a-63d59ebe56bf" (UID: "b7ff32de-2054-4ef4-b11a-63d59ebe56bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:38.816648 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:38.816623    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ff32de-2054-4ef4-b11a-63d59ebe56bf-kube-api-access-8v8sj" pod "b7ff32de-2054-4ef4-b11a-63d59ebe56bf" (UID: "b7ff32de-2054-4ef4-b11a-63d59ebe56bf"). InnerVolumeSpecName "kube-api-access-8v8sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:38.816648 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:38.816634    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7ff32de-2054-4ef4-b11a-63d59ebe56bf-maas-api-xks-ca" pod "b7ff32de-2054-4ef4-b11a-63d59ebe56bf" (UID: "b7ff32de-2054-4ef4-b11a-63d59ebe56bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:12:38.916256 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:38.916216    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ff32de-2054-4ef4-b11a-63d59ebe56bf-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:38.916256 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:38.916250    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v8sj\" (UniqueName: \"kubernetes.io/projected/b7ff32de-2054-4ef4-b11a-63d59ebe56bf-kube-api-access-8v8sj\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:38.916256 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:38.916261    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ff32de-2054-4ef4-b11a-63d59ebe56bf-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:39.526742 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:39.526698    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aece27a8-jg7b7" event={"ID":"b7ff32de-2054-4ef4-b11a-63d59ebe56bf","Type":"ContainerDied","Data":"79a76966de5c4e1a420638e1349219ee552d603e1c46faac1df0a9c9ae6ce414"}
Nov 03 18:12:39.526925 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:39.526758    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79a76966de5c4e1a420638e1349219ee552d603e1c46faac1df0a9c9ae6ce414"
Nov 03 18:12:39.526925 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:39.526720    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aece27a8-jg7b7"
Nov 03 18:12:39.852306 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:39.852280    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:12:39.864985 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:39.864960    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:12:39.870992 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:39.870965    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:12:39.882963 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:39.882940    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:12:41.165973 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:41.165941    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:41.346478 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:41.346446    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:42.587682 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:42.587633    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:46.450543 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:46.450499    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n" podUID="e0480008-8d47-450e-9083-36a9239da0e7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:12:47.379259 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:47.379222    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:48.026681 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:48.026647    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:48.214206 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:48.214177    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:49.087079 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.087045    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"]
Nov 03 18:12:49.088305 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.088276    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n" podUID="e0480008-8d47-450e-9083-36a9239da0e7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:12:49.139128 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.139092    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"]
Nov 03 18:12:49.139795 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.139777    6059 cpu_manager.go:414] "containerMap: removing container" podUID="b7ff32de-2054-4ef4-b11a-63d59ebe56bf" containerName="revoke-keys"
Nov 03 18:12:49.139860 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.139797    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7ff32de-2054-4ef4-b11a-63d59ebe56bf" containerName="revoke-keys"
Nov 03 18:12:49.139898 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.139892    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7ff32de-2054-4ef4-b11a-63d59ebe56bf" containerName="revoke-keys"
Nov 03 18:12:49.143484 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.143466    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"
Nov 03 18:12:49.203223 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.203190    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"]
Nov 03 18:12:49.219618 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.219581    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm\" (UID: \"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"
Nov 03 18:12:49.219831 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.219724    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7cwn\" (UniqueName: \"kubernetes.io/projected/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-kube-api-access-t7cwn\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm\" (UID: \"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"
Nov 03 18:12:49.219896 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.219864    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm\" (UID: \"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"
Nov 03 18:12:49.219937 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.219896    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm\" (UID: \"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"
Nov 03 18:12:49.229147 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.229117    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:49.321004 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.320965    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm\" (UID: \"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"
Nov 03 18:12:49.321335 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.321027    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7cwn\" (UniqueName: \"kubernetes.io/projected/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-kube-api-access-t7cwn\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm\" (UID: \"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"
Nov 03 18:12:49.321335 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.321093    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm\" (UID: \"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"
Nov 03 18:12:49.321335 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.321122    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm\" (UID: \"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"
Nov 03 18:12:49.321812 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.321784    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm\" (UID: \"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"
Nov 03 18:12:49.321936 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.321916    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm\" (UID: \"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"
Nov 03 18:12:49.323920 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.323898    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm\" (UID: \"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"
Nov 03 18:12:49.384721 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.384642    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7cwn\" (UniqueName: \"kubernetes.io/projected/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-kube-api-access-t7cwn\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm\" (UID: \"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"
Nov 03 18:12:49.407461 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.407425    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:49.456280 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.456247    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"
Nov 03 18:12:49.832119 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:12:49.832082    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8a3cd7_23de_4fe4_9a2f_6d0708ecdf3f.slice/crio-6c9a0c586e2201969567b451ee1ad854a0b0b5f511dd518eb636731e27e8ab34 WatchSource:0}: Error finding container 6c9a0c586e2201969567b451ee1ad854a0b0b5f511dd518eb636731e27e8ab34: Status 404 returned error can't find the container with id 6c9a0c586e2201969567b451ee1ad854a0b0b5f511dd518eb636731e27e8ab34
Nov 03 18:12:49.840626 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:49.836119    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"]
Nov 03 18:12:50.298486 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:50.298452    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:50.481878 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:50.481838    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:50.590048 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:50.589964    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm" event={"ID":"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f","Type":"ContainerStarted","Data":"96d0260dc0d694a5ac10ff5fa2b143e02930e83748f7bf8d8e752a32877ca993"}
Nov 03 18:12:50.590048 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:50.590018    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm" event={"ID":"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f","Type":"ContainerStarted","Data":"6c9a0c586e2201969567b451ee1ad854a0b0b5f511dd518eb636731e27e8ab34"}
Nov 03 18:12:50.590250 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:50.590070    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"
Nov 03 18:12:50.635852 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:12:50.635809    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-029ca0/agc-cred-b23a6815d37d: secret "agc-cred-b23a6815d37d" not found
Nov 03 18:12:50.635852 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:12:50.635849    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm: secret "agc-cred-b23a6815d37d" not found
Nov 03 18:12:50.636118 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:12:50.635914    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-provider-credentials podName:ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f nodeName:}" failed. No retries permitted until 2026-11-03 18:12:51.135893745 +0000 UTC m=+3311.933606519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm" (UID: "ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f") : secret "agc-cred-b23a6815d37d" not found
Nov 03 18:12:50.712778 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:50.712726    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm" podStartSLOduration=1.712712807 podStartE2EDuration="1.712712807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:12:50.710604364 +0000 UTC m=+3311.508317145" watchObservedRunningTime="2026-11-03 18:12:50.712712807 +0000 UTC m=+3311.510425611"
Nov 03 18:12:51.090912 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.090876    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q"]
Nov 03 18:12:51.095034 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.095012    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q"
Nov 03 18:12:51.141326 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.141285    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4781b5b6-14d9-4261-8cf3-043e2b912a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26398498-7bj7q\" (UID: \"4781b5b6-14d9-4261-8cf3-043e2b912a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q"
Nov 03 18:12:51.141561 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.141465    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb5mx\" (UniqueName: \"kubernetes.io/projected/4781b5b6-14d9-4261-8cf3-043e2b912a28-kube-api-access-cb5mx\") pod \"maas-api-revoke-sub-26398498-7bj7q\" (UID: \"4781b5b6-14d9-4261-8cf3-043e2b912a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q"
Nov 03 18:12:51.141561 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.141545    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4781b5b6-14d9-4261-8cf3-043e2b912a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26398498-7bj7q\" (UID: \"4781b5b6-14d9-4261-8cf3-043e2b912a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q"
Nov 03 18:12:51.141696 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:12:51.141630    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-029ca0/agc-cred-b23a6815d37d: secret "agc-cred-b23a6815d37d" not found
Nov 03 18:12:51.141696 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:12:51.141651    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm: secret "agc-cred-b23a6815d37d" not found
Nov 03 18:12:51.141808 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:12:51.141718    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-provider-credentials podName:ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f nodeName:}" failed. No retries permitted until 2026-11-03 18:12:52.141695469 +0000 UTC m=+3312.939408230 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm" (UID: "ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f") : secret "agc-cred-b23a6815d37d" not found
Nov 03 18:12:51.152418 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.152384    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:12:51.152872 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.152442    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:12:51.152872 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.152460    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:12:51.152872 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.152511    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 18:12:51.155528 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.155500    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q"]
Nov 03 18:12:51.243235 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.243194    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb5mx\" (UniqueName: \"kubernetes.io/projected/4781b5b6-14d9-4261-8cf3-043e2b912a28-kube-api-access-cb5mx\") pod \"maas-api-revoke-sub-26398498-7bj7q\" (UID: \"4781b5b6-14d9-4261-8cf3-043e2b912a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q"
Nov 03 18:12:51.243235 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.243238    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4781b5b6-14d9-4261-8cf3-043e2b912a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26398498-7bj7q\" (UID: \"4781b5b6-14d9-4261-8cf3-043e2b912a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q"
Nov 03 18:12:51.243520 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.243293    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4781b5b6-14d9-4261-8cf3-043e2b912a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26398498-7bj7q\" (UID: \"4781b5b6-14d9-4261-8cf3-043e2b912a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q"
Nov 03 18:12:51.243951 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.243924    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4781b5b6-14d9-4261-8cf3-043e2b912a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26398498-7bj7q\" (UID: \"4781b5b6-14d9-4261-8cf3-043e2b912a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q"
Nov 03 18:12:51.245824 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.245797    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4781b5b6-14d9-4261-8cf3-043e2b912a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26398498-7bj7q\" (UID: \"4781b5b6-14d9-4261-8cf3-043e2b912a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q"
Nov 03 18:12:51.306982 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.306939    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb5mx\" (UniqueName: \"kubernetes.io/projected/4781b5b6-14d9-4261-8cf3-043e2b912a28-kube-api-access-cb5mx\") pod \"maas-api-revoke-sub-26398498-7bj7q\" (UID: \"4781b5b6-14d9-4261-8cf3-043e2b912a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q"
Nov 03 18:12:51.408368 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.408267    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q"
Nov 03 18:12:51.478560 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.478528    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"]
Nov 03 18:12:51.521317 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.521278    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"]
Nov 03 18:12:51.525368 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.525335    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"
Nov 03 18:12:51.586067 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.586030    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"]
Nov 03 18:12:51.596079 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.596048    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:51.647323 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.647282    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f64e59e2-2593-4a86-9930-bba999ea63c2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp\" (UID: \"f64e59e2-2593-4a86-9930-bba999ea63c2\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"
Nov 03 18:12:51.647553 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.647332    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76v72\" (UniqueName: \"kubernetes.io/projected/f64e59e2-2593-4a86-9930-bba999ea63c2-kube-api-access-76v72\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp\" (UID: \"f64e59e2-2593-4a86-9930-bba999ea63c2\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"
Nov 03 18:12:51.647553 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.647476    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f64e59e2-2593-4a86-9930-bba999ea63c2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp\" (UID: \"f64e59e2-2593-4a86-9930-bba999ea63c2\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"
Nov 03 18:12:51.647805 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.647783    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f64e59e2-2593-4a86-9930-bba999ea63c2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp\" (UID: \"f64e59e2-2593-4a86-9930-bba999ea63c2\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"
Nov 03 18:12:51.748720 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.748677    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f64e59e2-2593-4a86-9930-bba999ea63c2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp\" (UID: \"f64e59e2-2593-4a86-9930-bba999ea63c2\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"
Nov 03 18:12:51.748720 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.748723    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f64e59e2-2593-4a86-9930-bba999ea63c2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp\" (UID: \"f64e59e2-2593-4a86-9930-bba999ea63c2\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"
Nov 03 18:12:51.749021 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.748759    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76v72\" (UniqueName: \"kubernetes.io/projected/f64e59e2-2593-4a86-9930-bba999ea63c2-kube-api-access-76v72\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp\" (UID: \"f64e59e2-2593-4a86-9930-bba999ea63c2\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"
Nov 03 18:12:51.749021 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.748796    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f64e59e2-2593-4a86-9930-bba999ea63c2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp\" (UID: \"f64e59e2-2593-4a86-9930-bba999ea63c2\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"
Nov 03 18:12:51.749540 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.749517    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f64e59e2-2593-4a86-9930-bba999ea63c2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp\" (UID: \"f64e59e2-2593-4a86-9930-bba999ea63c2\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"
Nov 03 18:12:51.749646 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.749593    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f64e59e2-2593-4a86-9930-bba999ea63c2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp\" (UID: \"f64e59e2-2593-4a86-9930-bba999ea63c2\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"
Nov 03 18:12:51.751400 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.751351    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f64e59e2-2593-4a86-9930-bba999ea63c2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp\" (UID: \"f64e59e2-2593-4a86-9930-bba999ea63c2\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"
Nov 03 18:12:51.780621 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:12:51.780587    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4781b5b6_14d9_4261_8cf3_043e2b912a28.slice/crio-de54ec3688bc329459f4744563d98e50357b08e7470d465ef6a5406a9f8b9c15 WatchSource:0}: Error finding container de54ec3688bc329459f4744563d98e50357b08e7470d465ef6a5406a9f8b9c15: Status 404 returned error can't find the container with id de54ec3688bc329459f4744563d98e50357b08e7470d465ef6a5406a9f8b9c15
Nov 03 18:12:51.781817 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.781790    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q"]
Nov 03 18:12:51.812575 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.812542    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76v72\" (UniqueName: \"kubernetes.io/projected/f64e59e2-2593-4a86-9930-bba999ea63c2-kube-api-access-76v72\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp\" (UID: \"f64e59e2-2593-4a86-9930-bba999ea63c2\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"
Nov 03 18:12:51.839093 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:51.839059    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"
Nov 03 18:12:52.152867 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:12:52.152832    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-029ca0/agc-cred-b23a6815d37d: secret "agc-cred-b23a6815d37d" not found
Nov 03 18:12:52.152867 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:12:52.152866    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm: secret "agc-cred-b23a6815d37d" not found
Nov 03 18:12:52.153121 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:12:52.152917    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-provider-credentials podName:ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f nodeName:}" failed. No retries permitted until 2026-11-03 18:12:54.152903265 +0000 UTC m=+3314.950616029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm" (UID: "ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f") : secret "agc-cred-b23a6815d37d" not found
Nov 03 18:12:52.155532 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:52.155493    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:52.212217 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:12:52.212178    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf64e59e2_2593_4a86_9930_bba999ea63c2.slice/crio-3ffa536d224095e286ac1654a07a074e0d956e4a8a69895aa143ad80f85d5600 WatchSource:0}: Error finding container 3ffa536d224095e286ac1654a07a074e0d956e4a8a69895aa143ad80f85d5600: Status 404 returned error can't find the container with id 3ffa536d224095e286ac1654a07a074e0d956e4a8a69895aa143ad80f85d5600
Nov 03 18:12:52.214301 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:52.214279    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"]
Nov 03 18:12:52.603749 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:52.603656    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q" event={"ID":"4781b5b6-14d9-4261-8cf3-043e2b912a28","Type":"ContainerStarted","Data":"0dfee0ded9d1455afc6cf088ee8ce70ccc7ef251d14de8e9379a8a40b0db24d9"}
Nov 03 18:12:52.604179 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:52.603785    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q" event={"ID":"4781b5b6-14d9-4261-8cf3-043e2b912a28","Type":"ContainerStarted","Data":"de54ec3688bc329459f4744563d98e50357b08e7470d465ef6a5406a9f8b9c15"}
Nov 03 18:12:52.605481 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:52.605448    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp" event={"ID":"f64e59e2-2593-4a86-9930-bba999ea63c2","Type":"ContainerStarted","Data":"648e7476aeb9ba5b9397ff5901c74992bada702434f2ebe6f14341b83b15feb3"}
Nov 03 18:12:52.605623 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:52.605551    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"
Nov 03 18:12:52.605623 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:52.605603    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp" event={"ID":"f64e59e2-2593-4a86-9930-bba999ea63c2","Type":"ContainerStarted","Data":"3ffa536d224095e286ac1654a07a074e0d956e4a8a69895aa143ad80f85d5600"}
Nov 03 18:12:52.763809 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:52.763753    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q" podStartSLOduration=1.7637387900000001 podStartE2EDuration="1.76373879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:12:52.761840703 +0000 UTC m=+3313.559553485" watchObservedRunningTime="2026-11-03 18:12:52.76373879 +0000 UTC m=+3313.561451571"
Nov 03 18:12:53.005043 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:53.004975    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp" podStartSLOduration=2.004960234 podStartE2EDuration="2.004960234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:12:53.004223457 +0000 UTC m=+3313.801936239" watchObservedRunningTime="2026-11-03 18:12:53.004960234 +0000 UTC m=+3313.802673046"
Nov 03 18:12:53.612296 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:53.612259    6059 generic.go:356] "Generic (PLEG): container finished" podID="4781b5b6-14d9-4261-8cf3-043e2b912a28" containerID="0dfee0ded9d1455afc6cf088ee8ce70ccc7ef251d14de8e9379a8a40b0db24d9" exitCode=0
Nov 03 18:12:53.612740 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:53.612349    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q" event={"ID":"4781b5b6-14d9-4261-8cf3-043e2b912a28","Type":"ContainerDied","Data":"0dfee0ded9d1455afc6cf088ee8ce70ccc7ef251d14de8e9379a8a40b0db24d9"}
Nov 03 18:12:54.087715 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.087671    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n" podUID="e0480008-8d47-450e-9083-36a9239da0e7" containerName="payload-processing" containerID="cri-o://a37af67563e37eb3f20a134341a9aa75fbde35bf06cce84b10ead3f9b570298b" gracePeriod=25
Nov 03 18:12:54.176002 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:12:54.175966    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-029ca0/agc-cred-b23a6815d37d: secret "agc-cred-b23a6815d37d" not found
Nov 03 18:12:54.176002 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:12:54.176008    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm: secret "agc-cred-b23a6815d37d" not found
Nov 03 18:12:54.176266 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:12:54.176078    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-provider-credentials podName:ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f nodeName:}" failed. No retries permitted until 2026-11-03 18:12:58.176057803 +0000 UTC m=+3318.973770562 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm" (UID: "ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f") : secret "agc-cred-b23a6815d37d" not found
Nov 03 18:12:54.332986 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.332959    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"
Nov 03 18:12:54.378241 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.378149    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0480008-8d47-450e-9083-36a9239da0e7-kube-api-access-wmtt4\" (UniqueName: \"kubernetes.io/projected/e0480008-8d47-450e-9083-36a9239da0e7-kube-api-access-wmtt4\") pod \"e0480008-8d47-450e-9083-36a9239da0e7\" (UID: \"e0480008-8d47-450e-9083-36a9239da0e7\") "
Nov 03 18:12:54.378457 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.378242    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0480008-8d47-450e-9083-36a9239da0e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0480008-8d47-450e-9083-36a9239da0e7-plugins-config-volume\") pod \"e0480008-8d47-450e-9083-36a9239da0e7\" (UID: \"e0480008-8d47-450e-9083-36a9239da0e7\") "
Nov 03 18:12:54.378457 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.378269    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0480008-8d47-450e-9083-36a9239da0e7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0480008-8d47-450e-9083-36a9239da0e7-provider-credentials\") pod \"e0480008-8d47-450e-9083-36a9239da0e7\" (UID: \"e0480008-8d47-450e-9083-36a9239da0e7\") "
Nov 03 18:12:54.378457 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.378339    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0480008-8d47-450e-9083-36a9239da0e7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0480008-8d47-450e-9083-36a9239da0e7-routing-overlay-volume\") pod \"e0480008-8d47-450e-9083-36a9239da0e7\" (UID: \"e0480008-8d47-450e-9083-36a9239da0e7\") "
Nov 03 18:12:54.378769 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.378735    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0480008-8d47-450e-9083-36a9239da0e7-plugins-config-volume" pod "e0480008-8d47-450e-9083-36a9239da0e7" (UID: "e0480008-8d47-450e-9083-36a9239da0e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:54.378886 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.378767    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0480008-8d47-450e-9083-36a9239da0e7-routing-overlay-volume" pod "e0480008-8d47-450e-9083-36a9239da0e7" (UID: "e0480008-8d47-450e-9083-36a9239da0e7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:54.380437 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.380413    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0480008-8d47-450e-9083-36a9239da0e7-provider-credentials" pod "e0480008-8d47-450e-9083-36a9239da0e7" (UID: "e0480008-8d47-450e-9083-36a9239da0e7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:54.380535 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.380514    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0480008-8d47-450e-9083-36a9239da0e7-kube-api-access-wmtt4" pod "e0480008-8d47-450e-9083-36a9239da0e7" (UID: "e0480008-8d47-450e-9083-36a9239da0e7"). InnerVolumeSpecName "kube-api-access-wmtt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:54.479494 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.479460    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0480008-8d47-450e-9083-36a9239da0e7-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:54.479494 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.479490    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0480008-8d47-450e-9083-36a9239da0e7-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:54.479494 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.479499    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0480008-8d47-450e-9083-36a9239da0e7-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:54.479494 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.479508    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmtt4\" (UniqueName: \"kubernetes.io/projected/e0480008-8d47-450e-9083-36a9239da0e7-kube-api-access-wmtt4\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:54.620074 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.620037    6059 generic.go:356] "Generic (PLEG): container finished" podID="e0480008-8d47-450e-9083-36a9239da0e7" containerID="a37af67563e37eb3f20a134341a9aa75fbde35bf06cce84b10ead3f9b570298b" exitCode=0
Nov 03 18:12:54.620550 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.620086    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"
Nov 03 18:12:54.620550 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.620125    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n" event={"ID":"e0480008-8d47-450e-9083-36a9239da0e7","Type":"ContainerDied","Data":"a37af67563e37eb3f20a134341a9aa75fbde35bf06cce84b10ead3f9b570298b"}
Nov 03 18:12:54.620550 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.620175    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n" event={"ID":"e0480008-8d47-450e-9083-36a9239da0e7","Type":"ContainerDied","Data":"c5652898ff3d445322b55282fa19e75fbefe4d76e724afd1337b098c1d909567"}
Nov 03 18:12:54.620550 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.620197    6059 scope.go:122] "RemoveContainer" containerID="a37af67563e37eb3f20a134341a9aa75fbde35bf06cce84b10ead3f9b570298b"
Nov 03 18:12:54.630702 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.630683    6059 scope.go:122] "RemoveContainer" containerID="a37af67563e37eb3f20a134341a9aa75fbde35bf06cce84b10ead3f9b570298b"
Nov 03 18:12:54.631002 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:12:54.630984    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a37af67563e37eb3f20a134341a9aa75fbde35bf06cce84b10ead3f9b570298b\": container with ID starting with a37af67563e37eb3f20a134341a9aa75fbde35bf06cce84b10ead3f9b570298b not found: ID does not exist" containerID="a37af67563e37eb3f20a134341a9aa75fbde35bf06cce84b10ead3f9b570298b"
Nov 03 18:12:54.631070 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.631011    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a37af67563e37eb3f20a134341a9aa75fbde35bf06cce84b10ead3f9b570298b"} err="failed to get container status \"a37af67563e37eb3f20a134341a9aa75fbde35bf06cce84b10ead3f9b570298b\": rpc error: code = NotFound desc = could not find container \"a37af67563e37eb3f20a134341a9aa75fbde35bf06cce84b10ead3f9b570298b\": container with ID starting with a37af67563e37eb3f20a134341a9aa75fbde35bf06cce84b10ead3f9b570298b not found: ID does not exist"
Nov 03 18:12:54.750658 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.750631    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q"
Nov 03 18:12:54.789385 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.789342    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"]
Nov 03 18:12:54.793904 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.793872    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-568bdj86n"]
Nov 03 18:12:54.884064 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.883967    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4781b5b6-14d9-4261-8cf3-043e2b912a28-kube-api-access-cb5mx\" (UniqueName: \"kubernetes.io/projected/4781b5b6-14d9-4261-8cf3-043e2b912a28-kube-api-access-cb5mx\") pod \"4781b5b6-14d9-4261-8cf3-043e2b912a28\" (UID: \"4781b5b6-14d9-4261-8cf3-043e2b912a28\") "
Nov 03 18:12:54.884252 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.884103    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4781b5b6-14d9-4261-8cf3-043e2b912a28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4781b5b6-14d9-4261-8cf3-043e2b912a28-maas-api-xks-ca\") pod \"4781b5b6-14d9-4261-8cf3-043e2b912a28\" (UID: \"4781b5b6-14d9-4261-8cf3-043e2b912a28\") "
Nov 03 18:12:54.884252 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.884132    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4781b5b6-14d9-4261-8cf3-043e2b912a28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4781b5b6-14d9-4261-8cf3-043e2b912a28-maas-api-openshift-ca\") pod \"4781b5b6-14d9-4261-8cf3-043e2b912a28\" (UID: \"4781b5b6-14d9-4261-8cf3-043e2b912a28\") "
Nov 03 18:12:54.884548 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.884521    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4781b5b6-14d9-4261-8cf3-043e2b912a28-maas-api-openshift-ca" pod "4781b5b6-14d9-4261-8cf3-043e2b912a28" (UID: "4781b5b6-14d9-4261-8cf3-043e2b912a28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:54.886121 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.886098    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4781b5b6-14d9-4261-8cf3-043e2b912a28-kube-api-access-cb5mx" pod "4781b5b6-14d9-4261-8cf3-043e2b912a28" (UID: "4781b5b6-14d9-4261-8cf3-043e2b912a28"). InnerVolumeSpecName "kube-api-access-cb5mx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:54.886202 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.886156    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4781b5b6-14d9-4261-8cf3-043e2b912a28-maas-api-xks-ca" pod "4781b5b6-14d9-4261-8cf3-043e2b912a28" (UID: "4781b5b6-14d9-4261-8cf3-043e2b912a28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:12:54.985204 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.985163    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4781b5b6-14d9-4261-8cf3-043e2b912a28-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:54.985204 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.985198    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4781b5b6-14d9-4261-8cf3-043e2b912a28-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:54.985460 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:54.985214    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb5mx\" (UniqueName: \"kubernetes.io/projected/4781b5b6-14d9-4261-8cf3-043e2b912a28-kube-api-access-cb5mx\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:55.334471 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:55.334434    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0480008-8d47-450e-9083-36a9239da0e7" path="/var/lib/kubelet/pods/e0480008-8d47-450e-9083-36a9239da0e7/volumes"
Nov 03 18:12:55.626971 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:55.626887    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q"
Nov 03 18:12:55.626971 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:55.626926    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q" event={"ID":"4781b5b6-14d9-4261-8cf3-043e2b912a28","Type":"ContainerDied","Data":"de54ec3688bc329459f4744563d98e50357b08e7470d465ef6a5406a9f8b9c15"}
Nov 03 18:12:55.627461 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:55.626980    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de54ec3688bc329459f4744563d98e50357b08e7470d465ef6a5406a9f8b9c15"
Nov 03 18:12:57.060615 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:57.060550    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:12:57.605917 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:57.605864    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm" podUID="ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f" containerName="payload-processing" containerID="cri-o://96d0260dc0d694a5ac10ff5fa2b143e02930e83748f7bf8d8e752a32877ca993" gracePeriod=25
Nov 03 18:12:57.866468 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:57.866400    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"
Nov 03 18:12:57.916452 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:57.916414    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-plugins-config-volume\") pod \"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f\" (UID: \"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f\") "
Nov 03 18:12:57.916636 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:57.916458    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-routing-overlay-volume\") pod \"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f\" (UID: \"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f\") "
Nov 03 18:12:57.916636 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:57.916489    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-kube-api-access-t7cwn\" (UniqueName: \"kubernetes.io/projected/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-kube-api-access-t7cwn\") pod \"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f\" (UID: \"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f\") "
Nov 03 18:12:57.916636 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:57.916564    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-provider-credentials\") pod \"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f\" (UID: \"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f\") "
Nov 03 18:12:57.916921 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:57.916893    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-plugins-config-volume" pod "ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f" (UID: "ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:57.916981 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:57.916906    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-routing-overlay-volume" pod "ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f" (UID: "ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:57.918795 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:57.918769    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-kube-api-access-t7cwn" pod "ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f" (UID: "ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f"). InnerVolumeSpecName "kube-api-access-t7cwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:57.918924 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:57.918801    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-provider-credentials" pod "ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f" (UID: "ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:58.017753 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:58.017697    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:58.017753 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:58.017725    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:58.017753 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:58.017734    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:58.017753 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:58.017743    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7cwn\" (UniqueName: \"kubernetes.io/projected/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f-kube-api-access-t7cwn\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:58.646099 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:58.646065    6059 generic.go:356] "Generic (PLEG): container finished" podID="ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f" containerID="96d0260dc0d694a5ac10ff5fa2b143e02930e83748f7bf8d8e752a32877ca993" exitCode=0
Nov 03 18:12:58.646555 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:58.646136    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"
Nov 03 18:12:58.646555 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:58.646152    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm" event={"ID":"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f","Type":"ContainerDied","Data":"96d0260dc0d694a5ac10ff5fa2b143e02930e83748f7bf8d8e752a32877ca993"}
Nov 03 18:12:58.646555 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:58.646210    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm" event={"ID":"ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f","Type":"ContainerDied","Data":"6c9a0c586e2201969567b451ee1ad854a0b0b5f511dd518eb636731e27e8ab34"}
Nov 03 18:12:58.646555 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:58.646232    6059 scope.go:122] "RemoveContainer" containerID="96d0260dc0d694a5ac10ff5fa2b143e02930e83748f7bf8d8e752a32877ca993"
Nov 03 18:12:58.656616 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:58.656594    6059 scope.go:122] "RemoveContainer" containerID="96d0260dc0d694a5ac10ff5fa2b143e02930e83748f7bf8d8e752a32877ca993"
Nov 03 18:12:58.656902 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:12:58.656881    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96d0260dc0d694a5ac10ff5fa2b143e02930e83748f7bf8d8e752a32877ca993\": container with ID starting with 96d0260dc0d694a5ac10ff5fa2b143e02930e83748f7bf8d8e752a32877ca993 not found: ID does not exist" containerID="96d0260dc0d694a5ac10ff5fa2b143e02930e83748f7bf8d8e752a32877ca993"
Nov 03 18:12:58.656975 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:58.656909    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d0260dc0d694a5ac10ff5fa2b143e02930e83748f7bf8d8e752a32877ca993"} err="failed to get container status \"96d0260dc0d694a5ac10ff5fa2b143e02930e83748f7bf8d8e752a32877ca993\": rpc error: code = NotFound desc = could not find container \"96d0260dc0d694a5ac10ff5fa2b143e02930e83748f7bf8d8e752a32877ca993\": container with ID starting with 96d0260dc0d694a5ac10ff5fa2b143e02930e83748f7bf8d8e752a32877ca993 not found: ID does not exist"
Nov 03 18:12:58.815307 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:58.815271    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"]
Nov 03 18:12:58.866844 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:58.866809    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-786fzc5wm"]
Nov 03 18:12:59.335575 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:12:59.335538    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f" path="/var/lib/kubelet/pods/ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f/volumes"
Nov 03 18:13:01.874706 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:01.874664    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:03.615744 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:03.615699    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp" podUID="f64e59e2-2593-4a86-9930-bba999ea63c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:13:04.383711 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:04.383673    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:04.561506 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:04.561470    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:05.169959 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.169917    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x"]
Nov 03 18:13:05.170432 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.170375    6059 cpu_manager.go:414] "containerMap: removing container" podUID="ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f" containerName="payload-processing"
Nov 03 18:13:05.170432 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.170388    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f" containerName="payload-processing"
Nov 03 18:13:05.170432 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.170403    6059 cpu_manager.go:414] "containerMap: removing container" podUID="e0480008-8d47-450e-9083-36a9239da0e7" containerName="payload-processing"
Nov 03 18:13:05.170432 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.170408    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0480008-8d47-450e-9083-36a9239da0e7" containerName="payload-processing"
Nov 03 18:13:05.170432 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.170416    6059 cpu_manager.go:414] "containerMap: removing container" podUID="4781b5b6-14d9-4261-8cf3-043e2b912a28" containerName="revoke-keys"
Nov 03 18:13:05.170432 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.170422    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4781b5b6-14d9-4261-8cf3-043e2b912a28" containerName="revoke-keys"
Nov 03 18:13:05.170674 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.170493    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff8a3cd7-23de-4fe4-9a2f-6d0708ecdf3f" containerName="payload-processing"
Nov 03 18:13:05.170674 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.170504    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0480008-8d47-450e-9083-36a9239da0e7" containerName="payload-processing"
Nov 03 18:13:05.170674 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.170510    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4781b5b6-14d9-4261-8cf3-043e2b912a28" containerName="revoke-keys"
Nov 03 18:13:05.175170 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.175149    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x"
Nov 03 18:13:05.231178 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.231098    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:13:05.282796 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.282759    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:13:05.282796 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.282784    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:13:05.283019 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.282839    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 18:13:05.285192 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.285170    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x"]
Nov 03 18:13:05.291545 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.291522    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f62e4b5c-d858-4d3a-9149-c4c76d6c21c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec2761d7-fhb8x\" (UID: \"f62e4b5c-d858-4d3a-9149-c4c76d6c21c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x"
Nov 03 18:13:05.291698 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.291576    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f62e4b5c-d858-4d3a-9149-c4c76d6c21c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec2761d7-fhb8x\" (UID: \"f62e4b5c-d858-4d3a-9149-c4c76d6c21c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x"
Nov 03 18:13:05.291750 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.291700    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsqvb\" (UniqueName: \"kubernetes.io/projected/f62e4b5c-d858-4d3a-9149-c4c76d6c21c6-kube-api-access-qsqvb\") pod \"maas-api-revoke-sub-ec2761d7-fhb8x\" (UID: \"f62e4b5c-d858-4d3a-9149-c4c76d6c21c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x"
Nov 03 18:13:05.392411 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.392343    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f62e4b5c-d858-4d3a-9149-c4c76d6c21c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec2761d7-fhb8x\" (UID: \"f62e4b5c-d858-4d3a-9149-c4c76d6c21c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x"
Nov 03 18:13:05.392606 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.392439    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsqvb\" (UniqueName: \"kubernetes.io/projected/f62e4b5c-d858-4d3a-9149-c4c76d6c21c6-kube-api-access-qsqvb\") pod \"maas-api-revoke-sub-ec2761d7-fhb8x\" (UID: \"f62e4b5c-d858-4d3a-9149-c4c76d6c21c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x"
Nov 03 18:13:05.392606 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.392514    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f62e4b5c-d858-4d3a-9149-c4c76d6c21c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec2761d7-fhb8x\" (UID: \"f62e4b5c-d858-4d3a-9149-c4c76d6c21c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x"
Nov 03 18:13:05.393128 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.393105    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f62e4b5c-d858-4d3a-9149-c4c76d6c21c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec2761d7-fhb8x\" (UID: \"f62e4b5c-d858-4d3a-9149-c4c76d6c21c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x"
Nov 03 18:13:05.394713 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.394692    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f62e4b5c-d858-4d3a-9149-c4c76d6c21c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec2761d7-fhb8x\" (UID: \"f62e4b5c-d858-4d3a-9149-c4c76d6c21c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x"
Nov 03 18:13:05.455884 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.455809    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsqvb\" (UniqueName: \"kubernetes.io/projected/f62e4b5c-d858-4d3a-9149-c4c76d6c21c6-kube-api-access-qsqvb\") pod \"maas-api-revoke-sub-ec2761d7-fhb8x\" (UID: \"f62e4b5c-d858-4d3a-9149-c4c76d6c21c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x"
Nov 03 18:13:05.488241 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.488208    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x"
Nov 03 18:13:05.857525 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:13:05.857489    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62e4b5c_d858_4d3a_9149_c4c76d6c21c6.slice/crio-789f569166d35b965628405c20592a246e3035bca26e41133acd8c82d76cc6bc WatchSource:0}: Error finding container 789f569166d35b965628405c20592a246e3035bca26e41133acd8c82d76cc6bc: Status 404 returned error can't find the container with id 789f569166d35b965628405c20592a246e3035bca26e41133acd8c82d76cc6bc
Nov 03 18:13:05.859703 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:05.859678    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x"]
Nov 03 18:13:06.354956 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:06.354918    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:06.648231 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:06.648200    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:06.694643 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:06.694607    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x" event={"ID":"f62e4b5c-d858-4d3a-9149-c4c76d6c21c6","Type":"ContainerStarted","Data":"51a5a0eeb0d390e8731b6369e72af375be33d5edb82c5116a1c1dc8e2a02a532"}
Nov 03 18:13:06.694783 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:06.694665    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x" event={"ID":"f62e4b5c-d858-4d3a-9149-c4c76d6c21c6","Type":"ContainerStarted","Data":"789f569166d35b965628405c20592a246e3035bca26e41133acd8c82d76cc6bc"}
Nov 03 18:13:06.814038 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:06.813975    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x" podStartSLOduration=1.813959967 podStartE2EDuration="1.813959967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:06.813320624 +0000 UTC m=+3327.611033408" watchObservedRunningTime="2026-11-03 18:13:06.813959967 +0000 UTC m=+3327.611672744"
Nov 03 18:13:06.863552 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:13:06.863507    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:13:07.647206 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:07.647166    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"]
Nov 03 18:13:07.648595 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:07.648564    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp" podUID="f64e59e2-2593-4a86-9930-bba999ea63c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:13:07.681580 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:07.681541    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"]
Nov 03 18:13:07.685310 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:07.685292    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"
Nov 03 18:13:07.702482 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:07.702451    6059 generic.go:356] "Generic (PLEG): container finished" podID="f62e4b5c-d858-4d3a-9149-c4c76d6c21c6" containerID="51a5a0eeb0d390e8731b6369e72af375be33d5edb82c5116a1c1dc8e2a02a532" exitCode=0
Nov 03 18:13:07.702620 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:07.702524    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x" event={"ID":"f62e4b5c-d858-4d3a-9149-c4c76d6c21c6","Type":"ContainerDied","Data":"51a5a0eeb0d390e8731b6369e72af375be33d5edb82c5116a1c1dc8e2a02a532"}
Nov 03 18:13:07.763229 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:07.763190    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"]
Nov 03 18:13:07.818938 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:07.818898    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d5daa11-5555-4570-9821-8a3cb98df698-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g\" (UID: \"0d5daa11-5555-4570-9821-8a3cb98df698\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"
Nov 03 18:13:07.819130 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:07.818958    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5daa11-5555-4570-9821-8a3cb98df698-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g\" (UID: \"0d5daa11-5555-4570-9821-8a3cb98df698\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"
Nov 03 18:13:07.819130 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:07.819070    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jwx2\" (UniqueName: \"kubernetes.io/projected/0d5daa11-5555-4570-9821-8a3cb98df698-kube-api-access-9jwx2\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g\" (UID: \"0d5daa11-5555-4570-9821-8a3cb98df698\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"
Nov 03 18:13:07.819130 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:07.819122    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d5daa11-5555-4570-9821-8a3cb98df698-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g\" (UID: \"0d5daa11-5555-4570-9821-8a3cb98df698\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"
Nov 03 18:13:07.920434 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:07.920311    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d5daa11-5555-4570-9821-8a3cb98df698-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g\" (UID: \"0d5daa11-5555-4570-9821-8a3cb98df698\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"
Nov 03 18:13:07.920434 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:07.920390    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5daa11-5555-4570-9821-8a3cb98df698-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g\" (UID: \"0d5daa11-5555-4570-9821-8a3cb98df698\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"
Nov 03 18:13:07.920434 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:07.920432    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jwx2\" (UniqueName: \"kubernetes.io/projected/0d5daa11-5555-4570-9821-8a3cb98df698-kube-api-access-9jwx2\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g\" (UID: \"0d5daa11-5555-4570-9821-8a3cb98df698\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"
Nov 03 18:13:07.920740 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:07.920473    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d5daa11-5555-4570-9821-8a3cb98df698-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g\" (UID: \"0d5daa11-5555-4570-9821-8a3cb98df698\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"
Nov 03 18:13:07.921168 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:07.921143    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d5daa11-5555-4570-9821-8a3cb98df698-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g\" (UID: \"0d5daa11-5555-4570-9821-8a3cb98df698\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"
Nov 03 18:13:07.921295 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:07.921215    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5daa11-5555-4570-9821-8a3cb98df698-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g\" (UID: \"0d5daa11-5555-4570-9821-8a3cb98df698\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"
Nov 03 18:13:07.922849 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:07.922825    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d5daa11-5555-4570-9821-8a3cb98df698-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g\" (UID: \"0d5daa11-5555-4570-9821-8a3cb98df698\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"
Nov 03 18:13:07.983911 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:07.983875    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jwx2\" (UniqueName: \"kubernetes.io/projected/0d5daa11-5555-4570-9821-8a3cb98df698-kube-api-access-9jwx2\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g\" (UID: \"0d5daa11-5555-4570-9821-8a3cb98df698\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"
Nov 03 18:13:07.999025 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:07.998991    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"
Nov 03 18:13:08.385299 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:13:08.385259    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5daa11_5555_4570_9821_8a3cb98df698.slice/crio-7504f1686d587213238a4dd805f28d269321657cb71c0f3d3c6cd58d03a0469b WatchSource:0}: Error finding container 7504f1686d587213238a4dd805f28d269321657cb71c0f3d3c6cd58d03a0469b: Status 404 returned error can't find the container with id 7504f1686d587213238a4dd805f28d269321657cb71c0f3d3c6cd58d03a0469b
Nov 03 18:13:08.387780 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:08.387752    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"]
Nov 03 18:13:08.710340 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:08.710304    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g" event={"ID":"0d5daa11-5555-4570-9821-8a3cb98df698","Type":"ContainerStarted","Data":"a4588d15761cfa7eeff656375029a5dae3e235e60ae107e813654305fadc27e8"}
Nov 03 18:13:08.710797 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:08.710372    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g" event={"ID":"0d5daa11-5555-4570-9821-8a3cb98df698","Type":"ContainerStarted","Data":"7504f1686d587213238a4dd805f28d269321657cb71c0f3d3c6cd58d03a0469b"}
Nov 03 18:13:08.710797 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:08.710397    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"
Nov 03 18:13:08.833323 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:08.833270    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g" podStartSLOduration=1.833251984 podStartE2EDuration="1.833251984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:08.830733025 +0000 UTC m=+3329.628445821" watchObservedRunningTime="2026-11-03 18:13:08.833251984 +0000 UTC m=+3329.630964764"
Nov 03 18:13:08.835846 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:08.835827    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x"
Nov 03 18:13:08.932040 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:08.932005    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f62e4b5c-d858-4d3a-9149-c4c76d6c21c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f62e4b5c-d858-4d3a-9149-c4c76d6c21c6-maas-api-xks-ca\") pod \"f62e4b5c-d858-4d3a-9149-c4c76d6c21c6\" (UID: \"f62e4b5c-d858-4d3a-9149-c4c76d6c21c6\") "
Nov 03 18:13:08.932188 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:08.932095    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f62e4b5c-d858-4d3a-9149-c4c76d6c21c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f62e4b5c-d858-4d3a-9149-c4c76d6c21c6-maas-api-openshift-ca\") pod \"f62e4b5c-d858-4d3a-9149-c4c76d6c21c6\" (UID: \"f62e4b5c-d858-4d3a-9149-c4c76d6c21c6\") "
Nov 03 18:13:08.932188 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:08.932153    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f62e4b5c-d858-4d3a-9149-c4c76d6c21c6-kube-api-access-qsqvb\" (UniqueName: \"kubernetes.io/projected/f62e4b5c-d858-4d3a-9149-c4c76d6c21c6-kube-api-access-qsqvb\") pod \"f62e4b5c-d858-4d3a-9149-c4c76d6c21c6\" (UID: \"f62e4b5c-d858-4d3a-9149-c4c76d6c21c6\") "
Nov 03 18:13:08.932538 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:08.932510    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f62e4b5c-d858-4d3a-9149-c4c76d6c21c6-maas-api-openshift-ca" pod "f62e4b5c-d858-4d3a-9149-c4c76d6c21c6" (UID: "f62e4b5c-d858-4d3a-9149-c4c76d6c21c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:08.934186 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:08.934156    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f62e4b5c-d858-4d3a-9149-c4c76d6c21c6-maas-api-xks-ca" pod "f62e4b5c-d858-4d3a-9149-c4c76d6c21c6" (UID: "f62e4b5c-d858-4d3a-9149-c4c76d6c21c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:08.934290 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:08.934224    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62e4b5c-d858-4d3a-9149-c4c76d6c21c6-kube-api-access-qsqvb" pod "f62e4b5c-d858-4d3a-9149-c4c76d6c21c6" (UID: "f62e4b5c-d858-4d3a-9149-c4c76d6c21c6"). InnerVolumeSpecName "kube-api-access-qsqvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:09.033607 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:09.033518    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f62e4b5c-d858-4d3a-9149-c4c76d6c21c6-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:09.033607 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:09.033547    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsqvb\" (UniqueName: \"kubernetes.io/projected/f62e4b5c-d858-4d3a-9149-c4c76d6c21c6-kube-api-access-qsqvb\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:09.033607 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:09.033558    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f62e4b5c-d858-4d3a-9149-c4c76d6c21c6-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:09.717288 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:09.717255    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x"
Nov 03 18:13:09.717723 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:09.717285    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x" event={"ID":"f62e4b5c-d858-4d3a-9149-c4c76d6c21c6","Type":"ContainerDied","Data":"789f569166d35b965628405c20592a246e3035bca26e41133acd8c82d76cc6bc"}
Nov 03 18:13:09.717723 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:09.717340    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="789f569166d35b965628405c20592a246e3035bca26e41133acd8c82d76cc6bc"
Nov 03 18:13:11.546299 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:11.546262    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:12.647992 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:12.647915    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp" podUID="f64e59e2-2593-4a86-9930-bba999ea63c2" containerName="payload-processing" containerID="cri-o://648e7476aeb9ba5b9397ff5901c74992bada702434f2ebe6f14341b83b15feb3" gracePeriod=25
Nov 03 18:13:12.904112 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:12.904045    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"
Nov 03 18:13:13.076556 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:13.076515    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f64e59e2-2593-4a86-9930-bba999ea63c2-kube-api-access-76v72\" (UniqueName: \"kubernetes.io/projected/f64e59e2-2593-4a86-9930-bba999ea63c2-kube-api-access-76v72\") pod \"f64e59e2-2593-4a86-9930-bba999ea63c2\" (UID: \"f64e59e2-2593-4a86-9930-bba999ea63c2\") "
Nov 03 18:13:13.076746 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:13.076582    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f64e59e2-2593-4a86-9930-bba999ea63c2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f64e59e2-2593-4a86-9930-bba999ea63c2-routing-overlay-volume\") pod \"f64e59e2-2593-4a86-9930-bba999ea63c2\" (UID: \"f64e59e2-2593-4a86-9930-bba999ea63c2\") "
Nov 03 18:13:13.076746 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:13.076691    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f64e59e2-2593-4a86-9930-bba999ea63c2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f64e59e2-2593-4a86-9930-bba999ea63c2-provider-credentials\") pod \"f64e59e2-2593-4a86-9930-bba999ea63c2\" (UID: \"f64e59e2-2593-4a86-9930-bba999ea63c2\") "
Nov 03 18:13:13.076746 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:13.076731    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f64e59e2-2593-4a86-9930-bba999ea63c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f64e59e2-2593-4a86-9930-bba999ea63c2-plugins-config-volume\") pod \"f64e59e2-2593-4a86-9930-bba999ea63c2\" (UID: \"f64e59e2-2593-4a86-9930-bba999ea63c2\") "
Nov 03 18:13:13.077073 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:13.076961    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f64e59e2-2593-4a86-9930-bba999ea63c2-routing-overlay-volume" pod "f64e59e2-2593-4a86-9930-bba999ea63c2" (UID: "f64e59e2-2593-4a86-9930-bba999ea63c2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:13.077183 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:13.077149    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f64e59e2-2593-4a86-9930-bba999ea63c2-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:13.077228 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:13.077172    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f64e59e2-2593-4a86-9930-bba999ea63c2-plugins-config-volume" pod "f64e59e2-2593-4a86-9930-bba999ea63c2" (UID: "f64e59e2-2593-4a86-9930-bba999ea63c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:13.078821 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:13.078796    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f64e59e2-2593-4a86-9930-bba999ea63c2-kube-api-access-76v72" pod "f64e59e2-2593-4a86-9930-bba999ea63c2" (UID: "f64e59e2-2593-4a86-9930-bba999ea63c2"). InnerVolumeSpecName "kube-api-access-76v72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:13.078910 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:13.078832    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f64e59e2-2593-4a86-9930-bba999ea63c2-provider-credentials" pod "f64e59e2-2593-4a86-9930-bba999ea63c2" (UID: "f64e59e2-2593-4a86-9930-bba999ea63c2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:13.177943 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:13.177846    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f64e59e2-2593-4a86-9930-bba999ea63c2-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:13.177943 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:13.177883    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f64e59e2-2593-4a86-9930-bba999ea63c2-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:13.177943 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:13.177893    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76v72\" (UniqueName: \"kubernetes.io/projected/f64e59e2-2593-4a86-9930-bba999ea63c2-kube-api-access-76v72\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:13.741596 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:13.741561    6059 generic.go:356] "Generic (PLEG): container finished" podID="f64e59e2-2593-4a86-9930-bba999ea63c2" containerID="648e7476aeb9ba5b9397ff5901c74992bada702434f2ebe6f14341b83b15feb3" exitCode=0
Nov 03 18:13:13.742085 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:13.741611    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"
Nov 03 18:13:13.742085 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:13.741653    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp" event={"ID":"f64e59e2-2593-4a86-9930-bba999ea63c2","Type":"ContainerDied","Data":"648e7476aeb9ba5b9397ff5901c74992bada702434f2ebe6f14341b83b15feb3"}
Nov 03 18:13:13.742085 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:13.741713    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp" event={"ID":"f64e59e2-2593-4a86-9930-bba999ea63c2","Type":"ContainerDied","Data":"3ffa536d224095e286ac1654a07a074e0d956e4a8a69895aa143ad80f85d5600"}
Nov 03 18:13:13.742085 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:13.741747    6059 scope.go:122] "RemoveContainer" containerID="648e7476aeb9ba5b9397ff5901c74992bada702434f2ebe6f14341b83b15feb3"
Nov 03 18:13:13.750990 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:13.750968    6059 scope.go:122] "RemoveContainer" containerID="648e7476aeb9ba5b9397ff5901c74992bada702434f2ebe6f14341b83b15feb3"
Nov 03 18:13:13.751325 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:13:13.751304    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"648e7476aeb9ba5b9397ff5901c74992bada702434f2ebe6f14341b83b15feb3\": container with ID starting with 648e7476aeb9ba5b9397ff5901c74992bada702434f2ebe6f14341b83b15feb3 not found: ID does not exist" containerID="648e7476aeb9ba5b9397ff5901c74992bada702434f2ebe6f14341b83b15feb3"
Nov 03 18:13:13.751428 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:13.751338    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648e7476aeb9ba5b9397ff5901c74992bada702434f2ebe6f14341b83b15feb3"} err="failed to get container status \"648e7476aeb9ba5b9397ff5901c74992bada702434f2ebe6f14341b83b15feb3\": rpc error: code = NotFound desc = could not find container \"648e7476aeb9ba5b9397ff5901c74992bada702434f2ebe6f14341b83b15feb3\": container with ID starting with 648e7476aeb9ba5b9397ff5901c74992bada702434f2ebe6f14341b83b15feb3 not found: ID does not exist"
Nov 03 18:13:13.912286 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:13.912253    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"]
Nov 03 18:13:13.964208 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:13.964168    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5dqbjp"]
Nov 03 18:13:15.336221 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:15.336176    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f64e59e2-2593-4a86-9930-bba999ea63c2" path="/var/lib/kubelet/pods/f64e59e2-2593-4a86-9930-bba999ea63c2/volumes"
Nov 03 18:13:16.371192 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:16.371148    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:17.420921 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:17.420834    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"]
Nov 03 18:13:17.472542 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:17.472499    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"]
Nov 03 18:13:17.475440 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:17.475399    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g" podUID="0d5daa11-5555-4570-9821-8a3cb98df698" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:13:17.591968 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:17.591934    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:17.765284 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:17.765249    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:18.805596 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:18.805556    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:19.267794 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:19.267761    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:21.090583 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.090475    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:21.271752 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.271699    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:21.586051 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.586013    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk"]
Nov 03 18:13:21.586534 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.586518    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f62e4b5c-d858-4d3a-9149-c4c76d6c21c6" containerName="revoke-keys"
Nov 03 18:13:21.586587 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.586539    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f62e4b5c-d858-4d3a-9149-c4c76d6c21c6" containerName="revoke-keys"
Nov 03 18:13:21.586587 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.586567    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f64e59e2-2593-4a86-9930-bba999ea63c2" containerName="payload-processing"
Nov 03 18:13:21.586587 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.586573    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f64e59e2-2593-4a86-9930-bba999ea63c2" containerName="payload-processing"
Nov 03 18:13:21.586736 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.586653    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f62e4b5c-d858-4d3a-9149-c4c76d6c21c6" containerName="revoke-keys"
Nov 03 18:13:21.586736 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.586668    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f64e59e2-2593-4a86-9930-bba999ea63c2" containerName="payload-processing"
Nov 03 18:13:21.597941 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.597899    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk"
Nov 03 18:13:21.650833 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.650796    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk"]
Nov 03 18:13:21.664987 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.664953    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg6lz\" (UniqueName: \"kubernetes.io/projected/c32d3994-e89a-4e65-bf95-4cc5b60630a6-kube-api-access-tg6lz\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk\" (UID: \"c32d3994-e89a-4e65-bf95-4cc5b60630a6\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk"
Nov 03 18:13:21.665223 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.665026    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c32d3994-e89a-4e65-bf95-4cc5b60630a6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk\" (UID: \"c32d3994-e89a-4e65-bf95-4cc5b60630a6\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk"
Nov 03 18:13:21.665223 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.665129    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c32d3994-e89a-4e65-bf95-4cc5b60630a6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk\" (UID: \"c32d3994-e89a-4e65-bf95-4cc5b60630a6\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk"
Nov 03 18:13:21.665375 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.665233    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c32d3994-e89a-4e65-bf95-4cc5b60630a6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk\" (UID: \"c32d3994-e89a-4e65-bf95-4cc5b60630a6\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk"
Nov 03 18:13:21.699323 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.699281    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"agc-cred-4e87610b42ce\""
Nov 03 18:13:21.699556 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.699460    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"payload-processing-external-model-e2e--58d254fa-dockercfg-w78zw\""
Nov 03 18:13:21.766003 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.765968    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c32d3994-e89a-4e65-bf95-4cc5b60630a6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk\" (UID: \"c32d3994-e89a-4e65-bf95-4cc5b60630a6\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk"
Nov 03 18:13:21.766207 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.766022    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c32d3994-e89a-4e65-bf95-4cc5b60630a6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk\" (UID: \"c32d3994-e89a-4e65-bf95-4cc5b60630a6\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk"
Nov 03 18:13:21.766207 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.766092    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg6lz\" (UniqueName: \"kubernetes.io/projected/c32d3994-e89a-4e65-bf95-4cc5b60630a6-kube-api-access-tg6lz\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk\" (UID: \"c32d3994-e89a-4e65-bf95-4cc5b60630a6\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk"
Nov 03 18:13:21.766207 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.766153    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c32d3994-e89a-4e65-bf95-4cc5b60630a6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk\" (UID: \"c32d3994-e89a-4e65-bf95-4cc5b60630a6\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk"
Nov 03 18:13:21.766207 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:13:21.766186    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-029ca0/agc-cred-4e87610b42ce: secret "agc-cred-4e87610b42ce" not found
Nov 03 18:13:21.766207 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:13:21.766205    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk: secret "agc-cred-4e87610b42ce" not found
Nov 03 18:13:21.766500 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:13:21.766263    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c32d3994-e89a-4e65-bf95-4cc5b60630a6-provider-credentials podName:c32d3994-e89a-4e65-bf95-4cc5b60630a6 nodeName:}" failed. No retries permitted until 2026-11-03 18:13:22.266245194 +0000 UTC m=+3343.063957965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c32d3994-e89a-4e65-bf95-4cc5b60630a6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk" (UID: "c32d3994-e89a-4e65-bf95-4cc5b60630a6") : secret "agc-cred-4e87610b42ce" not found
Nov 03 18:13:21.766697 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.766677    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c32d3994-e89a-4e65-bf95-4cc5b60630a6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk\" (UID: \"c32d3994-e89a-4e65-bf95-4cc5b60630a6\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk"
Nov 03 18:13:21.766862 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.766844    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c32d3994-e89a-4e65-bf95-4cc5b60630a6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk\" (UID: \"c32d3994-e89a-4e65-bf95-4cc5b60630a6\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk"
Nov 03 18:13:21.829647 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.829611    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg6lz\" (UniqueName: \"kubernetes.io/projected/c32d3994-e89a-4e65-bf95-4cc5b60630a6-kube-api-access-tg6lz\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk\" (UID: \"c32d3994-e89a-4e65-bf95-4cc5b60630a6\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk"
Nov 03 18:13:21.962283 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:21.962223    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:22.269783 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.269687    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c32d3994-e89a-4e65-bf95-4cc5b60630a6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk\" (UID: \"c32d3994-e89a-4e65-bf95-4cc5b60630a6\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk"
Nov 03 18:13:22.272139 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:13:22.269804    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-029ca0/agc-cred-4e87610b42ce: secret "agc-cred-4e87610b42ce" not found
Nov 03 18:13:22.272139 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:13:22.269819    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk: secret "agc-cred-4e87610b42ce" not found
Nov 03 18:13:22.272139 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:13:22.269888    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c32d3994-e89a-4e65-bf95-4cc5b60630a6-provider-credentials podName:c32d3994-e89a-4e65-bf95-4cc5b60630a6 nodeName:}" failed. No retries permitted until 2026-11-03 18:13:23.269870224 +0000 UTC m=+3344.067582995 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c32d3994-e89a-4e65-bf95-4cc5b60630a6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk" (UID: "c32d3994-e89a-4e65-bf95-4cc5b60630a6") : secret "agc-cred-4e87610b42ce" not found
Nov 03 18:13:22.347731 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.347660    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk"]
Nov 03 18:13:22.348070 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:13:22.348043    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk" podUID="c32d3994-e89a-4e65-bf95-4cc5b60630a6"
Nov 03 18:13:22.421765 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.421697    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw" podUID="4a188c3d-95cc-4c28-b885-510a8b8d3f65" containerName="payload-processing" containerID="cri-o://add08f281dce0920895aad182e8d63a1e7b3c8c879831ba14629b10c1a3e4e91" gracePeriod=25
Nov 03 18:13:22.473067 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.473021    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g" podUID="0d5daa11-5555-4570-9821-8a3cb98df698" containerName="payload-processing" containerID="cri-o://a4588d15761cfa7eeff656375029a5dae3e235e60ae107e813654305fadc27e8" gracePeriod=25
Nov 03 18:13:22.520891 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.520799    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:22.721157 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.721127    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"
Nov 03 18:13:22.737559 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.737526    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"
Nov 03 18:13:22.776275 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.776185    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d5daa11-5555-4570-9821-8a3cb98df698-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d5daa11-5555-4570-9821-8a3cb98df698-provider-credentials\") pod \"0d5daa11-5555-4570-9821-8a3cb98df698\" (UID: \"0d5daa11-5555-4570-9821-8a3cb98df698\") "
Nov 03 18:13:22.776275 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.776229    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d5daa11-5555-4570-9821-8a3cb98df698-kube-api-access-9jwx2\" (UniqueName: \"kubernetes.io/projected/0d5daa11-5555-4570-9821-8a3cb98df698-kube-api-access-9jwx2\") pod \"0d5daa11-5555-4570-9821-8a3cb98df698\" (UID: \"0d5daa11-5555-4570-9821-8a3cb98df698\") "
Nov 03 18:13:22.776617 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.776287    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d5daa11-5555-4570-9821-8a3cb98df698-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5daa11-5555-4570-9821-8a3cb98df698-plugins-config-volume\") pod \"0d5daa11-5555-4570-9821-8a3cb98df698\" (UID: \"0d5daa11-5555-4570-9821-8a3cb98df698\") "
Nov 03 18:13:22.776617 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.776308    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d5daa11-5555-4570-9821-8a3cb98df698-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d5daa11-5555-4570-9821-8a3cb98df698-routing-overlay-volume\") pod \"0d5daa11-5555-4570-9821-8a3cb98df698\" (UID: \"0d5daa11-5555-4570-9821-8a3cb98df698\") "
Nov 03 18:13:22.776617 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.776408    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a188c3d-95cc-4c28-b885-510a8b8d3f65-kube-api-access-s84c4\" (UniqueName: \"kubernetes.io/projected/4a188c3d-95cc-4c28-b885-510a8b8d3f65-kube-api-access-s84c4\") pod \"4a188c3d-95cc-4c28-b885-510a8b8d3f65\" (UID: \"4a188c3d-95cc-4c28-b885-510a8b8d3f65\") "
Nov 03 18:13:22.776617 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.776462    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a188c3d-95cc-4c28-b885-510a8b8d3f65-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a188c3d-95cc-4c28-b885-510a8b8d3f65-provider-credentials\") pod \"4a188c3d-95cc-4c28-b885-510a8b8d3f65\" (UID: \"4a188c3d-95cc-4c28-b885-510a8b8d3f65\") "
Nov 03 18:13:22.776617 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.776502    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a188c3d-95cc-4c28-b885-510a8b8d3f65-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a188c3d-95cc-4c28-b885-510a8b8d3f65-routing-overlay-volume\") pod \"4a188c3d-95cc-4c28-b885-510a8b8d3f65\" (UID: \"4a188c3d-95cc-4c28-b885-510a8b8d3f65\") "
Nov 03 18:13:22.776617 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.776605    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a188c3d-95cc-4c28-b885-510a8b8d3f65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a188c3d-95cc-4c28-b885-510a8b8d3f65-plugins-config-volume\") pod \"4a188c3d-95cc-4c28-b885-510a8b8d3f65\" (UID: \"4a188c3d-95cc-4c28-b885-510a8b8d3f65\") "
Nov 03 18:13:22.776964 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.776748    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d5daa11-5555-4570-9821-8a3cb98df698-routing-overlay-volume" pod "0d5daa11-5555-4570-9821-8a3cb98df698" (UID: "0d5daa11-5555-4570-9821-8a3cb98df698"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:22.777026 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.776985    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d5daa11-5555-4570-9821-8a3cb98df698-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:22.777075 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.777017    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d5daa11-5555-4570-9821-8a3cb98df698-plugins-config-volume" pod "0d5daa11-5555-4570-9821-8a3cb98df698" (UID: "0d5daa11-5555-4570-9821-8a3cb98df698"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:22.777132 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.777105    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a188c3d-95cc-4c28-b885-510a8b8d3f65-routing-overlay-volume" pod "4a188c3d-95cc-4c28-b885-510a8b8d3f65" (UID: "4a188c3d-95cc-4c28-b885-510a8b8d3f65"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:22.777206 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.777153    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a188c3d-95cc-4c28-b885-510a8b8d3f65-plugins-config-volume" pod "4a188c3d-95cc-4c28-b885-510a8b8d3f65" (UID: "4a188c3d-95cc-4c28-b885-510a8b8d3f65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:22.778942 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.778918    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a188c3d-95cc-4c28-b885-510a8b8d3f65-kube-api-access-s84c4" pod "4a188c3d-95cc-4c28-b885-510a8b8d3f65" (UID: "4a188c3d-95cc-4c28-b885-510a8b8d3f65"). InnerVolumeSpecName "kube-api-access-s84c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:22.779066 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.779009    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a188c3d-95cc-4c28-b885-510a8b8d3f65-provider-credentials" pod "4a188c3d-95cc-4c28-b885-510a8b8d3f65" (UID: "4a188c3d-95cc-4c28-b885-510a8b8d3f65"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:22.779066 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.779014    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5daa11-5555-4570-9821-8a3cb98df698-kube-api-access-9jwx2" pod "0d5daa11-5555-4570-9821-8a3cb98df698" (UID: "0d5daa11-5555-4570-9821-8a3cb98df698"). InnerVolumeSpecName "kube-api-access-9jwx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:22.779066 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.779035    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5daa11-5555-4570-9821-8a3cb98df698-provider-credentials" pod "0d5daa11-5555-4570-9821-8a3cb98df698" (UID: "0d5daa11-5555-4570-9821-8a3cb98df698"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:22.797549 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.797515    6059 generic.go:356] "Generic (PLEG): container finished" podID="4a188c3d-95cc-4c28-b885-510a8b8d3f65" containerID="add08f281dce0920895aad182e8d63a1e7b3c8c879831ba14629b10c1a3e4e91" exitCode=0
Nov 03 18:13:22.797743 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.797557    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"
Nov 03 18:13:22.797743 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.797597    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw" event={"ID":"4a188c3d-95cc-4c28-b885-510a8b8d3f65","Type":"ContainerDied","Data":"add08f281dce0920895aad182e8d63a1e7b3c8c879831ba14629b10c1a3e4e91"}
Nov 03 18:13:22.797743 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.797650    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw" event={"ID":"4a188c3d-95cc-4c28-b885-510a8b8d3f65","Type":"ContainerDied","Data":"ffb7caf38f3627b192904a4efa57d0119f73ee34280e212e01e5168164a4bdc9"}
Nov 03 18:13:22.797743 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.797672    6059 scope.go:122] "RemoveContainer" containerID="add08f281dce0920895aad182e8d63a1e7b3c8c879831ba14629b10c1a3e4e91"
Nov 03 18:13:22.799312 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.799288    6059 generic.go:356] "Generic (PLEG): container finished" podID="0d5daa11-5555-4570-9821-8a3cb98df698" containerID="a4588d15761cfa7eeff656375029a5dae3e235e60ae107e813654305fadc27e8" exitCode=0
Nov 03 18:13:22.799438 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.799332    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"
Nov 03 18:13:22.799438 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.799401    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g" event={"ID":"0d5daa11-5555-4570-9821-8a3cb98df698","Type":"ContainerDied","Data":"a4588d15761cfa7eeff656375029a5dae3e235e60ae107e813654305fadc27e8"}
Nov 03 18:13:22.799529 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.799462    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g" event={"ID":"0d5daa11-5555-4570-9821-8a3cb98df698","Type":"ContainerDied","Data":"7504f1686d587213238a4dd805f28d269321657cb71c0f3d3c6cd58d03a0469b"}
Nov 03 18:13:22.799602 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.799586    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk"
Nov 03 18:13:22.807587 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.807548    6059 scope.go:122] "RemoveContainer" containerID="add08f281dce0920895aad182e8d63a1e7b3c8c879831ba14629b10c1a3e4e91"
Nov 03 18:13:22.807890 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:13:22.807867    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"add08f281dce0920895aad182e8d63a1e7b3c8c879831ba14629b10c1a3e4e91\": container with ID starting with add08f281dce0920895aad182e8d63a1e7b3c8c879831ba14629b10c1a3e4e91 not found: ID does not exist" containerID="add08f281dce0920895aad182e8d63a1e7b3c8c879831ba14629b10c1a3e4e91"
Nov 03 18:13:22.807984 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.807903    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"add08f281dce0920895aad182e8d63a1e7b3c8c879831ba14629b10c1a3e4e91"} err="failed to get container status \"add08f281dce0920895aad182e8d63a1e7b3c8c879831ba14629b10c1a3e4e91\": rpc error: code = NotFound desc = could not find container \"add08f281dce0920895aad182e8d63a1e7b3c8c879831ba14629b10c1a3e4e91\": container with ID starting with add08f281dce0920895aad182e8d63a1e7b3c8c879831ba14629b10c1a3e4e91 not found: ID does not exist"
Nov 03 18:13:22.807984 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.807926    6059 scope.go:122] "RemoveContainer" containerID="a4588d15761cfa7eeff656375029a5dae3e235e60ae107e813654305fadc27e8"
Nov 03 18:13:22.809320 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.809299    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk"
Nov 03 18:13:22.817043 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.817020    6059 scope.go:122] "RemoveContainer" containerID="a4588d15761cfa7eeff656375029a5dae3e235e60ae107e813654305fadc27e8"
Nov 03 18:13:22.817407 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:13:22.817378    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4588d15761cfa7eeff656375029a5dae3e235e60ae107e813654305fadc27e8\": container with ID starting with a4588d15761cfa7eeff656375029a5dae3e235e60ae107e813654305fadc27e8 not found: ID does not exist" containerID="a4588d15761cfa7eeff656375029a5dae3e235e60ae107e813654305fadc27e8"
Nov 03 18:13:22.817463 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.817420    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4588d15761cfa7eeff656375029a5dae3e235e60ae107e813654305fadc27e8"} err="failed to get container status \"a4588d15761cfa7eeff656375029a5dae3e235e60ae107e813654305fadc27e8\": rpc error: code = NotFound desc = could not find container \"a4588d15761cfa7eeff656375029a5dae3e235e60ae107e813654305fadc27e8\": container with ID starting with a4588d15761cfa7eeff656375029a5dae3e235e60ae107e813654305fadc27e8 not found: ID does not exist"
Nov 03 18:13:22.878268 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.878227    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c32d3994-e89a-4e65-bf95-4cc5b60630a6-kube-api-access-tg6lz\" (UniqueName: \"kubernetes.io/projected/c32d3994-e89a-4e65-bf95-4cc5b60630a6-kube-api-access-tg6lz\") pod \"c32d3994-e89a-4e65-bf95-4cc5b60630a6\" (UID: \"c32d3994-e89a-4e65-bf95-4cc5b60630a6\") "
Nov 03 18:13:22.878507 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.878344    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c32d3994-e89a-4e65-bf95-4cc5b60630a6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c32d3994-e89a-4e65-bf95-4cc5b60630a6-routing-overlay-volume\") pod \"c32d3994-e89a-4e65-bf95-4cc5b60630a6\" (UID: \"c32d3994-e89a-4e65-bf95-4cc5b60630a6\") "
Nov 03 18:13:22.878507 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.878468    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c32d3994-e89a-4e65-bf95-4cc5b60630a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c32d3994-e89a-4e65-bf95-4cc5b60630a6-plugins-config-volume\") pod \"c32d3994-e89a-4e65-bf95-4cc5b60630a6\" (UID: \"c32d3994-e89a-4e65-bf95-4cc5b60630a6\") "
Nov 03 18:13:22.878726 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.878688    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c32d3994-e89a-4e65-bf95-4cc5b60630a6-routing-overlay-volume" pod "c32d3994-e89a-4e65-bf95-4cc5b60630a6" (UID: "c32d3994-e89a-4e65-bf95-4cc5b60630a6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:22.878831 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.878812    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c32d3994-e89a-4e65-bf95-4cc5b60630a6-plugins-config-volume" pod "c32d3994-e89a-4e65-bf95-4cc5b60630a6" (UID: "c32d3994-e89a-4e65-bf95-4cc5b60630a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:22.878880 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.878830    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s84c4\" (UniqueName: \"kubernetes.io/projected/4a188c3d-95cc-4c28-b885-510a8b8d3f65-kube-api-access-s84c4\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:22.878880 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.878851    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a188c3d-95cc-4c28-b885-510a8b8d3f65-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:22.878880 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.878867    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a188c3d-95cc-4c28-b885-510a8b8d3f65-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:22.878975 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.878882    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a188c3d-95cc-4c28-b885-510a8b8d3f65-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:22.878975 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.878897    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d5daa11-5555-4570-9821-8a3cb98df698-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:22.878975 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.878911    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c32d3994-e89a-4e65-bf95-4cc5b60630a6-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:22.878975 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.878928    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jwx2\" (UniqueName: \"kubernetes.io/projected/0d5daa11-5555-4570-9821-8a3cb98df698-kube-api-access-9jwx2\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:22.878975 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.878944    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5daa11-5555-4570-9821-8a3cb98df698-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:22.880524 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.880506    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c32d3994-e89a-4e65-bf95-4cc5b60630a6-kube-api-access-tg6lz" pod "c32d3994-e89a-4e65-bf95-4cc5b60630a6" (UID: "c32d3994-e89a-4e65-bf95-4cc5b60630a6"). InnerVolumeSpecName "kube-api-access-tg6lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:22.980239 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.980192    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c32d3994-e89a-4e65-bf95-4cc5b60630a6-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:22.980239 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:22.980228    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg6lz\" (UniqueName: \"kubernetes.io/projected/c32d3994-e89a-4e65-bf95-4cc5b60630a6-kube-api-access-tg6lz\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:23.144809 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:23.144718    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"]
Nov 03 18:13:23.150212 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:23.150178    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5778fzm2g"]
Nov 03 18:13:23.265701 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:23.265665    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"]
Nov 03 18:13:23.269836 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:23.269808    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-66d55d9fw"]
Nov 03 18:13:23.284270 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:23.284232    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c32d3994-e89a-4e65-bf95-4cc5b60630a6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk\" (UID: \"c32d3994-e89a-4e65-bf95-4cc5b60630a6\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk"
Nov 03 18:13:23.284425 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:13:23.284411    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-029ca0/agc-cred-4e87610b42ce: secret "agc-cred-4e87610b42ce" not found
Nov 03 18:13:23.284475 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:13:23.284430    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk: secret "agc-cred-4e87610b42ce" not found
Nov 03 18:13:23.284513 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:13:23.284482    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c32d3994-e89a-4e65-bf95-4cc5b60630a6-provider-credentials podName:c32d3994-e89a-4e65-bf95-4cc5b60630a6 nodeName:}" failed. No retries permitted until 2026-11-03 18:13:25.284467656 +0000 UTC m=+3346.082180427 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c32d3994-e89a-4e65-bf95-4cc5b60630a6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk" (UID: "c32d3994-e89a-4e65-bf95-4cc5b60630a6") : secret "agc-cred-4e87610b42ce" not found
Nov 03 18:13:23.334612 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:23.334566    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d5daa11-5555-4570-9821-8a3cb98df698" path="/var/lib/kubelet/pods/0d5daa11-5555-4570-9821-8a3cb98df698/volumes"
Nov 03 18:13:23.335047 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:23.335028    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a188c3d-95cc-4c28-b885-510a8b8d3f65" path="/var/lib/kubelet/pods/4a188c3d-95cc-4c28-b885-510a8b8d3f65/volumes"
Nov 03 18:13:23.806846 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:23.806814    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk"
Nov 03 18:13:24.041981 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:24.041937    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk"]
Nov 03 18:13:24.045867 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:24.045836    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5fc9pg4tk"]
Nov 03 18:13:24.092899 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:24.092809    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c32d3994-e89a-4e65-bf95-4cc5b60630a6-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:25.335537 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:25.335506    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c32d3994-e89a-4e65-bf95-4cc5b60630a6" path="/var/lib/kubelet/pods/c32d3994-e89a-4e65-bf95-4cc5b60630a6/volumes"
Nov 03 18:13:26.054547 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:26.054510    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:27.991222 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:27.991180    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:28.178576 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:28.178544    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:30.862576 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:30.862521    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:34.127208 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:34.127168    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:34.308919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:34.308876    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:35.234930 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:35.234880    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:35.410642 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:35.410596    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:35.653439 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:35.653331    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:36.093620 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:36.093586    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:36.276968 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:36.276927    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:36.861940 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:13:36.861903    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:13:39.050461 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.050406    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:39.540748 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.540710    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"]
Nov 03 18:13:39.541242 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.541223    6059 cpu_manager.go:414] "containerMap: removing container" podUID="4a188c3d-95cc-4c28-b885-510a8b8d3f65" containerName="payload-processing"
Nov 03 18:13:39.541429 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.541245    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a188c3d-95cc-4c28-b885-510a8b8d3f65" containerName="payload-processing"
Nov 03 18:13:39.541429 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.541290    6059 cpu_manager.go:414] "containerMap: removing container" podUID="0d5daa11-5555-4570-9821-8a3cb98df698" containerName="payload-processing"
Nov 03 18:13:39.541429 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.541300    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d5daa11-5555-4570-9821-8a3cb98df698" containerName="payload-processing"
Nov 03 18:13:39.541429 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.541426    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d5daa11-5555-4570-9821-8a3cb98df698" containerName="payload-processing"
Nov 03 18:13:39.541662 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.541449    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a188c3d-95cc-4c28-b885-510a8b8d3f65" containerName="payload-processing"
Nov 03 18:13:39.544699 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.544678    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"
Nov 03 18:13:39.651522 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.651478    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-dade73eb8d42\""
Nov 03 18:13:39.651723 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.651479    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 18:13:39.651723 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.651648    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-dbfpw\""
Nov 03 18:13:39.651822 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.651727    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 18:13:39.651822 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.651732    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 18:13:39.651822 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.651744    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 18:13:39.655282 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.655269    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"]
Nov 03 18:13:39.659494 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.659466    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4vtn\" (UniqueName: \"kubernetes.io/projected/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-kube-api-access-g4vtn\") pod \"payload-processing-external-model-6c78f6b78f-bdd4l\" (UID: \"dd844e4e-e51f-4f3d-85fb-c459a013c4d5\") " pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"
Nov 03 18:13:39.659604 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.659586    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-routing-overlay-volume\") pod \"payload-processing-external-model-6c78f6b78f-bdd4l\" (UID: \"dd844e4e-e51f-4f3d-85fb-c459a013c4d5\") " pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"
Nov 03 18:13:39.659653 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.659620    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-provider-credentials\") pod \"payload-processing-external-model-6c78f6b78f-bdd4l\" (UID: \"dd844e4e-e51f-4f3d-85fb-c459a013c4d5\") " pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"
Nov 03 18:13:39.659759 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.659741    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-plugins-config-volume\") pod \"payload-processing-external-model-6c78f6b78f-bdd4l\" (UID: \"dd844e4e-e51f-4f3d-85fb-c459a013c4d5\") " pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"
Nov 03 18:13:39.761289 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.761250    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-routing-overlay-volume\") pod \"payload-processing-external-model-6c78f6b78f-bdd4l\" (UID: \"dd844e4e-e51f-4f3d-85fb-c459a013c4d5\") " pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"
Nov 03 18:13:39.761529 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.761308    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-provider-credentials\") pod \"payload-processing-external-model-6c78f6b78f-bdd4l\" (UID: \"dd844e4e-e51f-4f3d-85fb-c459a013c4d5\") " pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"
Nov 03 18:13:39.761529 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.761414    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-plugins-config-volume\") pod \"payload-processing-external-model-6c78f6b78f-bdd4l\" (UID: \"dd844e4e-e51f-4f3d-85fb-c459a013c4d5\") " pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"
Nov 03 18:13:39.761529 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.761448    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4vtn\" (UniqueName: \"kubernetes.io/projected/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-kube-api-access-g4vtn\") pod \"payload-processing-external-model-6c78f6b78f-bdd4l\" (UID: \"dd844e4e-e51f-4f3d-85fb-c459a013c4d5\") " pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"
Nov 03 18:13:39.762104 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.762080    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-routing-overlay-volume\") pod \"payload-processing-external-model-6c78f6b78f-bdd4l\" (UID: \"dd844e4e-e51f-4f3d-85fb-c459a013c4d5\") " pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"
Nov 03 18:13:39.762184 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.762172    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-plugins-config-volume\") pod \"payload-processing-external-model-6c78f6b78f-bdd4l\" (UID: \"dd844e4e-e51f-4f3d-85fb-c459a013c4d5\") " pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"
Nov 03 18:13:39.763850 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.763829    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-provider-credentials\") pod \"payload-processing-external-model-6c78f6b78f-bdd4l\" (UID: \"dd844e4e-e51f-4f3d-85fb-c459a013c4d5\") " pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"
Nov 03 18:13:39.859553 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:39.859477    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4vtn\" (UniqueName: \"kubernetes.io/projected/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-kube-api-access-g4vtn\") pod \"payload-processing-external-model-6c78f6b78f-bdd4l\" (UID: \"dd844e4e-e51f-4f3d-85fb-c459a013c4d5\") " pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"
Nov 03 18:13:40.157586 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:40.157490    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"
Nov 03 18:13:40.540962 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:40.540927    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"]
Nov 03 18:13:40.544768 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:13:40.544740    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd844e4e_e51f_4f3d_85fb_c459a013c4d5.slice/crio-bbc5dfa48d58b1ac214d5cf292920fe1a33825972b4c1fa2f70d0dd483ce6c1c WatchSource:0}: Error finding container bbc5dfa48d58b1ac214d5cf292920fe1a33825972b4c1fa2f70d0dd483ce6c1c: Status 404 returned error can't find the container with id bbc5dfa48d58b1ac214d5cf292920fe1a33825972b4c1fa2f70d0dd483ce6c1c
Nov 03 18:13:40.566720 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:40.566690    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:40.910377 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:40.910275    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l" event={"ID":"dd844e4e-e51f-4f3d-85fb-c459a013c4d5","Type":"ContainerStarted","Data":"0dae6cbdf24f647457ad97e22eb0e33eed48e35f66af9e82fca63ab4a669f000"}
Nov 03 18:13:40.910377 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:40.910342    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"
Nov 03 18:13:40.910583 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:40.910395    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l" event={"ID":"dd844e4e-e51f-4f3d-85fb-c459a013c4d5","Type":"ContainerStarted","Data":"bbc5dfa48d58b1ac214d5cf292920fe1a33825972b4c1fa2f70d0dd483ce6c1c"}
Nov 03 18:13:41.032715 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:41.032661    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l" podStartSLOduration=2.032644854 podStartE2EDuration="2.032644854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:41.031245298 +0000 UTC m=+3361.828958078" watchObservedRunningTime="2026-11-03 18:13:41.032644854 +0000 UTC m=+3361.830357635"
Nov 03 18:13:45.359154 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:45.359115    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:49.571594 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:49.571557    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:50.154840 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:50.154807    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:51.919436 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:51.919407    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"
Nov 03 18:13:55.084565 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:55.084524    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:13:59.856524 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:13:59.856487    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:04.772259 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:04.772224    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:06.861777 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:06.861736    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:14:09.666577 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:09.666502    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:11.811121 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:11.811081    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw"]
Nov 03 18:14:11.814944 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:11.814925    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw"
Nov 03 18:14:11.875410 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:11.875373    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw"]
Nov 03 18:14:11.880244 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:11.880207    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jm6w\" (UniqueName: \"kubernetes.io/projected/69377556-aba2-4899-b10d-f6eb868a6ca4-kube-api-access-6jm6w\") pod \"maas-api-revoke-sub-7e029470-vmgvw\" (UID: \"69377556-aba2-4899-b10d-f6eb868a6ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw"
Nov 03 18:14:11.880461 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:11.880272    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69377556-aba2-4899-b10d-f6eb868a6ca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e029470-vmgvw\" (UID: \"69377556-aba2-4899-b10d-f6eb868a6ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw"
Nov 03 18:14:11.880461 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:11.880343    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69377556-aba2-4899-b10d-f6eb868a6ca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e029470-vmgvw\" (UID: \"69377556-aba2-4899-b10d-f6eb868a6ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw"
Nov 03 18:14:11.923775 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:11.923739    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:14:11.923775 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:11.923746    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:14:11.923775 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:11.923768    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:14:11.923775 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:11.923782    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 18:14:11.980999 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:11.980957    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jm6w\" (UniqueName: \"kubernetes.io/projected/69377556-aba2-4899-b10d-f6eb868a6ca4-kube-api-access-6jm6w\") pod \"maas-api-revoke-sub-7e029470-vmgvw\" (UID: \"69377556-aba2-4899-b10d-f6eb868a6ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw"
Nov 03 18:14:11.980999 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:11.981006    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69377556-aba2-4899-b10d-f6eb868a6ca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e029470-vmgvw\" (UID: \"69377556-aba2-4899-b10d-f6eb868a6ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw"
Nov 03 18:14:11.981292 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:11.981023    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69377556-aba2-4899-b10d-f6eb868a6ca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e029470-vmgvw\" (UID: \"69377556-aba2-4899-b10d-f6eb868a6ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw"
Nov 03 18:14:11.981750 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:11.981725    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69377556-aba2-4899-b10d-f6eb868a6ca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e029470-vmgvw\" (UID: \"69377556-aba2-4899-b10d-f6eb868a6ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw"
Nov 03 18:14:11.983441 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:11.983418    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69377556-aba2-4899-b10d-f6eb868a6ca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e029470-vmgvw\" (UID: \"69377556-aba2-4899-b10d-f6eb868a6ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw"
Nov 03 18:14:12.044581 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:12.044549    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jm6w\" (UniqueName: \"kubernetes.io/projected/69377556-aba2-4899-b10d-f6eb868a6ca4-kube-api-access-6jm6w\") pod \"maas-api-revoke-sub-7e029470-vmgvw\" (UID: \"69377556-aba2-4899-b10d-f6eb868a6ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw"
Nov 03 18:14:12.128488 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:12.128395    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw"
Nov 03 18:14:12.499523 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:14:12.499487    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69377556_aba2_4899_b10d_f6eb868a6ca4.slice/crio-03621f0b81799829756edc99f5b163f3d2e6ed20a5772f7f7bb544c92319fb8e WatchSource:0}: Error finding container 03621f0b81799829756edc99f5b163f3d2e6ed20a5772f7f7bb544c92319fb8e: Status 404 returned error can't find the container with id 03621f0b81799829756edc99f5b163f3d2e6ed20a5772f7f7bb544c92319fb8e
Nov 03 18:14:12.501490 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:12.501467    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw"]
Nov 03 18:14:13.101612 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:13.101510    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw" event={"ID":"69377556-aba2-4899-b10d-f6eb868a6ca4","Type":"ContainerStarted","Data":"0d0130c4187bc93736064b94afd7acc376a4bf6c37c52496906583f67a0d32e1"}
Nov 03 18:14:13.101612 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:13.101568    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw" event={"ID":"69377556-aba2-4899-b10d-f6eb868a6ca4","Type":"ContainerStarted","Data":"03621f0b81799829756edc99f5b163f3d2e6ed20a5772f7f7bb544c92319fb8e"}
Nov 03 18:14:13.223502 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:13.223442    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw" podStartSLOduration=2.223427303 podStartE2EDuration="2.223427303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:13.221800168 +0000 UTC m=+3394.019512950" watchObservedRunningTime="2026-11-03 18:14:13.223427303 +0000 UTC m=+3394.021140083"
Nov 03 18:14:14.109277 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:14.109235    6059 generic.go:356] "Generic (PLEG): container finished" podID="69377556-aba2-4899-b10d-f6eb868a6ca4" containerID="0d0130c4187bc93736064b94afd7acc376a4bf6c37c52496906583f67a0d32e1" exitCode=0
Nov 03 18:14:14.109728 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:14.109329    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw" event={"ID":"69377556-aba2-4899-b10d-f6eb868a6ca4","Type":"ContainerDied","Data":"0d0130c4187bc93736064b94afd7acc376a4bf6c37c52496906583f67a0d32e1"}
Nov 03 18:14:14.454099 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:14.453998    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:15.252797 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.252771    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw"
Nov 03 18:14:15.361657 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.361627    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"]
Nov 03 18:14:15.362065 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.362051    6059 cpu_manager.go:414] "containerMap: removing container" podUID="69377556-aba2-4899-b10d-f6eb868a6ca4" containerName="revoke-keys"
Nov 03 18:14:15.362118 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.362066    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69377556-aba2-4899-b10d-f6eb868a6ca4" containerName="revoke-keys"
Nov 03 18:14:15.362163 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.362154    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="69377556-aba2-4899-b10d-f6eb868a6ca4" containerName="revoke-keys"
Nov 03 18:14:15.365303 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.365278    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.369200 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.369131    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"]
Nov 03 18:14:15.372917 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.372854    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.417662 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.417625    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69377556-aba2-4899-b10d-f6eb868a6ca4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69377556-aba2-4899-b10d-f6eb868a6ca4-maas-api-xks-ca\") pod \"69377556-aba2-4899-b10d-f6eb868a6ca4\" (UID: \"69377556-aba2-4899-b10d-f6eb868a6ca4\") "
Nov 03 18:14:15.417934 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.417712    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69377556-aba2-4899-b10d-f6eb868a6ca4-kube-api-access-6jm6w\" (UniqueName: \"kubernetes.io/projected/69377556-aba2-4899-b10d-f6eb868a6ca4-kube-api-access-6jm6w\") pod \"69377556-aba2-4899-b10d-f6eb868a6ca4\" (UID: \"69377556-aba2-4899-b10d-f6eb868a6ca4\") "
Nov 03 18:14:15.417934 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.417790    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69377556-aba2-4899-b10d-f6eb868a6ca4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69377556-aba2-4899-b10d-f6eb868a6ca4-maas-api-openshift-ca\") pod \"69377556-aba2-4899-b10d-f6eb868a6ca4\" (UID: \"69377556-aba2-4899-b10d-f6eb868a6ca4\") "
Nov 03 18:14:15.418187 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.418161    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69377556-aba2-4899-b10d-f6eb868a6ca4-maas-api-openshift-ca" pod "69377556-aba2-4899-b10d-f6eb868a6ca4" (UID: "69377556-aba2-4899-b10d-f6eb868a6ca4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:15.419903 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.419873    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-t4h2g\""
Nov 03 18:14:15.420010 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.419910    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69377556-aba2-4899-b10d-f6eb868a6ca4-maas-api-xks-ca" pod "69377556-aba2-4899-b10d-f6eb868a6ca4" (UID: "69377556-aba2-4899-b10d-f6eb868a6ca4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:14:15.420054 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.420007    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69377556-aba2-4899-b10d-f6eb868a6ca4-kube-api-access-6jm6w" pod "69377556-aba2-4899-b10d-f6eb868a6ca4" (UID: "69377556-aba2-4899-b10d-f6eb868a6ca4"). InnerVolumeSpecName "kube-api-access-6jm6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:15.449048 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.449019    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"]
Nov 03 18:14:15.501188 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.501149    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"]
Nov 03 18:14:15.519120 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.519080    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.519120 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.519133    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.519385 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.519227    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.519385 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.519279    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.519385 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.519345    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.519506 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.519413    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.519506 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.519446    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8cc50b-9b04-4398-a692-a75d4125fba8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.519506 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.519470    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.519627 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.519560    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.519627 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.519579    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.519627 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.519596    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.519754 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.519660    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.519754 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.519697    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.519824 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.519784    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tzkm\" (UniqueName: \"kubernetes.io/projected/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-kube-api-access-2tzkm\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.519863 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.519822    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tqsd\" (UniqueName: \"kubernetes.io/projected/8c8cc50b-9b04-4398-a692-a75d4125fba8-kube-api-access-9tqsd\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.519911 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.519891    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.519965 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.519950    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.520015 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.519998    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.520091 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.520076    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jm6w\" (UniqueName: \"kubernetes.io/projected/69377556-aba2-4899-b10d-f6eb868a6ca4-kube-api-access-6jm6w\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:15.520127 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.520100    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69377556-aba2-4899-b10d-f6eb868a6ca4-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:15.520127 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.520116    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69377556-aba2-4899-b10d-f6eb868a6ca4-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:15.620914 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.620816    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.620914 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.620868    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.620914 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.620891    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.620914 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.620920    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.621292 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.620940    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.621292 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.620978    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tzkm\" (UniqueName: \"kubernetes.io/projected/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-kube-api-access-2tzkm\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.621292 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.621002    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tqsd\" (UniqueName: \"kubernetes.io/projected/8c8cc50b-9b04-4398-a692-a75d4125fba8-kube-api-access-9tqsd\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.621292 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.621032    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.621292 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.621063    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.621292 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.621101    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.621292 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.621146    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.621292 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.621184    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.621292 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.621238    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.621292 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.621267    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.621904 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.621305    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.621904 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.621329    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.621904 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.621305    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.621904 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.621615    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.621904 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.621641    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.621904 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.621649    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.621904 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.621882    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.621904 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.621902    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.622386 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.621912    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.622386 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.621921    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8cc50b-9b04-4398-a692-a75d4125fba8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.622386 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.621959    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.622386 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.621966    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.622386 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.622148    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.622677 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.622500    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8cc50b-9b04-4398-a692-a75d4125fba8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.623555 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.623525    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.623955 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.623934    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.624048 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.624028    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.624221 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.624205    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.684551 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.684513    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.684705 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.684559    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tqsd\" (UniqueName: \"kubernetes.io/projected/8c8cc50b-9b04-4398-a692-a75d4125fba8-kube-api-access-9tqsd\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.684752 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.684735    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-kjgtk\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:15.684866 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.684849    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tzkm\" (UniqueName: \"kubernetes.io/projected/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-kube-api-access-2tzkm\") pod \"maas-default-gateway-openshift-default-58c6585447-6trbr\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.687719 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.687699    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:15.979032 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:15.978994    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:16.110619 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:14:16.110577    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af1e128_ecfb_4449_a2ab_0d6a3732ca71.slice/crio-bff2c8909dee02818829e9729e9a2ff184eccc294eb9f78505c178b1487169d3 WatchSource:0}: Error finding container bff2c8909dee02818829e9729e9a2ff184eccc294eb9f78505c178b1487169d3: Status 404 returned error can't find the container with id bff2c8909dee02818829e9729e9a2ff184eccc294eb9f78505c178b1487169d3
Nov 03 18:14:16.112982 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:16.112950    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:14:16.113090 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:16.113016    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:14:16.113090 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:16.113052    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:14:16.113206 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:16.113182    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"]
Nov 03 18:14:16.122858 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:16.122834    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw"
Nov 03 18:14:16.122858 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:16.122845    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw" event={"ID":"69377556-aba2-4899-b10d-f6eb868a6ca4","Type":"ContainerDied","Data":"03621f0b81799829756edc99f5b163f3d2e6ed20a5772f7f7bb544c92319fb8e"}
Nov 03 18:14:16.123064 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:16.122887    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03621f0b81799829756edc99f5b163f3d2e6ed20a5772f7f7bb544c92319fb8e"
Nov 03 18:14:16.124311 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:16.124281    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr" event={"ID":"6af1e128-ecfb-4449-a2ab-0d6a3732ca71","Type":"ContainerStarted","Data":"bff2c8909dee02818829e9729e9a2ff184eccc294eb9f78505c178b1487169d3"}
Nov 03 18:14:16.376542 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:16.374925    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:14:16.376542 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:16.375096    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:14:16.376542 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:16.375274    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:14:16.379716 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:16.379684    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"]
Nov 03 18:14:17.147380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:17.143540    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk" event={"ID":"8c8cc50b-9b04-4398-a692-a75d4125fba8","Type":"ContainerStarted","Data":"904ba53e5789cc5d699261e4ba685e181143a96e15ec956dcd3082f9bdbca422"}
Nov 03 18:14:17.173175 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:17.173136    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk" event={"ID":"8c8cc50b-9b04-4398-a692-a75d4125fba8","Type":"ContainerStarted","Data":"8bf4d16abd2092c105bc18d56456a165cd8366639fdbd4a4926dffb94922e2db"}
Nov 03 18:14:17.173519 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:17.173492    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr" event={"ID":"6af1e128-ecfb-4449-a2ab-0d6a3732ca71","Type":"ContainerStarted","Data":"8b96819365fa52902fa2c94775775b9586c3d3d17815337171acb8006bbccb06"}
Nov 03 18:14:17.286101 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:17.286032    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk" podStartSLOduration=2.286016236 podStartE2EDuration="2.286016236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:17.283796128 +0000 UTC m=+3398.081508907" watchObservedRunningTime="2026-11-03 18:14:17.286016236 +0000 UTC m=+3398.083729082"
Nov 03 18:14:17.411284 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:17.411176    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr" podStartSLOduration=2.411161192 podStartE2EDuration="2.411161192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:17.409286189 +0000 UTC m=+3398.206998969" watchObservedRunningTime="2026-11-03 18:14:17.411161192 +0000 UTC m=+3398.208873973"
Nov 03 18:14:17.688477 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:17.688380    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:17.690409 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:17.690376    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6trbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Nov 03 18:14:17.690515 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:17.690459    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr" podUID="6af1e128-ecfb-4449-a2ab-0d6a3732ca71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Nov 03 18:14:17.979837 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:17.979765    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:17.981906 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:17.981630    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-kjgtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Nov 03 18:14:17.981906 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:17.981674    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk" podUID="8c8cc50b-9b04-4398-a692-a75d4125fba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Nov 03 18:14:18.688225 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:18.688176    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6trbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Nov 03 18:14:18.688677 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:18.688242    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr" podUID="6af1e128-ecfb-4449-a2ab-0d6a3732ca71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Nov 03 18:14:18.980057 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:18.980009    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-kjgtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Nov 03 18:14:18.980238 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:18.980100    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk" podUID="8c8cc50b-9b04-4398-a692-a75d4125fba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Nov 03 18:14:19.303761 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:19.303633    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:19.688374 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:19.688267    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6trbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Nov 03 18:14:19.688374 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:19.688325    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr" podUID="6af1e128-ecfb-4449-a2ab-0d6a3732ca71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Nov 03 18:14:19.980380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:19.979798    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-kjgtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Nov 03 18:14:19.980380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:19.979859    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk" podUID="8c8cc50b-9b04-4398-a692-a75d4125fba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Nov 03 18:14:20.688275 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:20.688228    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6trbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Nov 03 18:14:20.688464 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:20.688283    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr" podUID="6af1e128-ecfb-4449-a2ab-0d6a3732ca71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Nov 03 18:14:20.981123 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:20.980785    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-kjgtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Nov 03 18:14:20.981123 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:20.980849    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk" podUID="8c8cc50b-9b04-4398-a692-a75d4125fba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Nov 03 18:14:21.688608 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:21.688560    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6trbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Nov 03 18:14:21.689025 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:21.688616    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr" podUID="6af1e128-ecfb-4449-a2ab-0d6a3732ca71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Nov 03 18:14:21.990222 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:21.981883    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-kjgtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Nov 03 18:14:21.990222 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:21.982017    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk" podUID="8c8cc50b-9b04-4398-a692-a75d4125fba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Nov 03 18:14:22.689202 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:22.689156    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6trbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Nov 03 18:14:22.689748 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:22.689238    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr" podUID="6af1e128-ecfb-4449-a2ab-0d6a3732ca71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Nov 03 18:14:22.979725 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:22.979676    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-kjgtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Nov 03 18:14:22.979986 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:22.979751    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk" podUID="8c8cc50b-9b04-4398-a692-a75d4125fba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Nov 03 18:14:23.688775 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:23.688736    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6trbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Nov 03 18:14:23.689068 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:23.688797    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr" podUID="6af1e128-ecfb-4449-a2ab-0d6a3732ca71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Nov 03 18:14:23.979781 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:23.979742    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-kjgtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Nov 03 18:14:23.980214 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:23.979830    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk" podUID="8c8cc50b-9b04-4398-a692-a75d4125fba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Nov 03 18:14:24.255034 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:24.254942    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:24.603509 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:24.603411    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"]
Nov 03 18:14:24.683435 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:24.683397    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:24.859482 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:24.859349    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:25.730562 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:25.730520    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6trbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:14:25.731187 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:25.731142    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr" podUID="6af1e128-ecfb-4449-a2ab-0d6a3732ca71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:14:25.815041 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:25.814991    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:25.815497 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:25.815473    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:25.816607 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:25.816582    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:14:25.981082 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:25.980974    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-kjgtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:14:25.981082 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:25.981058    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk" podUID="8c8cc50b-9b04-4398-a692-a75d4125fba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:14:26.045417 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:26.045378    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:26.045798 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:26.045778    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:26.046573 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:26.046552    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:14:27.355978 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:27.355927    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:27.527502 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:27.527465    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:29.157905 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:29.157860    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:29.604700 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:29.604637    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l" podUID="dd844e4e-e51f-4f3d-85fb-c459a013c4d5" containerName="payload-processing" containerID="cri-o://0dae6cbdf24f647457ad97e22eb0e33eed48e35f66af9e82fca63ab4a669f000" gracePeriod=25
Nov 03 18:14:29.883450 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:29.883415    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"
Nov 03 18:14:29.990063 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:29.990018    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-plugins-config-volume\") pod \"dd844e4e-e51f-4f3d-85fb-c459a013c4d5\" (UID: \"dd844e4e-e51f-4f3d-85fb-c459a013c4d5\") "
Nov 03 18:14:29.990270 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:29.990164    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-provider-credentials\") pod \"dd844e4e-e51f-4f3d-85fb-c459a013c4d5\" (UID: \"dd844e4e-e51f-4f3d-85fb-c459a013c4d5\") "
Nov 03 18:14:29.990270 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:29.990248    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-routing-overlay-volume\") pod \"dd844e4e-e51f-4f3d-85fb-c459a013c4d5\" (UID: \"dd844e4e-e51f-4f3d-85fb-c459a013c4d5\") "
Nov 03 18:14:29.990428 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:29.990313    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-kube-api-access-g4vtn\" (UniqueName: \"kubernetes.io/projected/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-kube-api-access-g4vtn\") pod \"dd844e4e-e51f-4f3d-85fb-c459a013c4d5\" (UID: \"dd844e4e-e51f-4f3d-85fb-c459a013c4d5\") "
Nov 03 18:14:29.990674 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:29.990491    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-plugins-config-volume" pod "dd844e4e-e51f-4f3d-85fb-c459a013c4d5" (UID: "dd844e4e-e51f-4f3d-85fb-c459a013c4d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:29.990813 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:29.990786    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:29.990964 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:29.990933    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-routing-overlay-volume" pod "dd844e4e-e51f-4f3d-85fb-c459a013c4d5" (UID: "dd844e4e-e51f-4f3d-85fb-c459a013c4d5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:29.992907 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:29.992879    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-provider-credentials" pod "dd844e4e-e51f-4f3d-85fb-c459a013c4d5" (UID: "dd844e4e-e51f-4f3d-85fb-c459a013c4d5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:29.992907 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:29.992885    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-kube-api-access-g4vtn" pod "dd844e4e-e51f-4f3d-85fb-c459a013c4d5" (UID: "dd844e4e-e51f-4f3d-85fb-c459a013c4d5"). InnerVolumeSpecName "kube-api-access-g4vtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:30.091720 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:30.091679    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:30.091720 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:30.091713    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:30.091720 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:30.091723    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4vtn\" (UniqueName: \"kubernetes.io/projected/dd844e4e-e51f-4f3d-85fb-c459a013c4d5-kube-api-access-g4vtn\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:30.270036 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:30.269993    6059 generic.go:356] "Generic (PLEG): container finished" podID="dd844e4e-e51f-4f3d-85fb-c459a013c4d5" containerID="0dae6cbdf24f647457ad97e22eb0e33eed48e35f66af9e82fca63ab4a669f000" exitCode=0
Nov 03 18:14:30.270676 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:30.270064    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"
Nov 03 18:14:30.270676 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:30.270077    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l" event={"ID":"dd844e4e-e51f-4f3d-85fb-c459a013c4d5","Type":"ContainerDied","Data":"0dae6cbdf24f647457ad97e22eb0e33eed48e35f66af9e82fca63ab4a669f000"}
Nov 03 18:14:30.270676 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:30.270140    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l" event={"ID":"dd844e4e-e51f-4f3d-85fb-c459a013c4d5","Type":"ContainerDied","Data":"bbc5dfa48d58b1ac214d5cf292920fe1a33825972b4c1fa2f70d0dd483ce6c1c"}
Nov 03 18:14:30.270676 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:30.270170    6059 scope.go:122] "RemoveContainer" containerID="0dae6cbdf24f647457ad97e22eb0e33eed48e35f66af9e82fca63ab4a669f000"
Nov 03 18:14:30.282435 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:30.282402    6059 scope.go:122] "RemoveContainer" containerID="0dae6cbdf24f647457ad97e22eb0e33eed48e35f66af9e82fca63ab4a669f000"
Nov 03 18:14:30.282816 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:30.282787    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dae6cbdf24f647457ad97e22eb0e33eed48e35f66af9e82fca63ab4a669f000\": container with ID starting with 0dae6cbdf24f647457ad97e22eb0e33eed48e35f66af9e82fca63ab4a669f000 not found: ID does not exist" containerID="0dae6cbdf24f647457ad97e22eb0e33eed48e35f66af9e82fca63ab4a669f000"
Nov 03 18:14:30.282878 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:30.282831    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dae6cbdf24f647457ad97e22eb0e33eed48e35f66af9e82fca63ab4a669f000"} err="failed to get container status \"0dae6cbdf24f647457ad97e22eb0e33eed48e35f66af9e82fca63ab4a669f000\": rpc error: code = NotFound desc = could not find container \"0dae6cbdf24f647457ad97e22eb0e33eed48e35f66af9e82fca63ab4a669f000\": container with ID starting with 0dae6cbdf24f647457ad97e22eb0e33eed48e35f66af9e82fca63ab4a669f000 not found: ID does not exist"
Nov 03 18:14:30.442578 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:30.442537    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"]
Nov 03 18:14:30.494153 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:30.494113    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c78f6b78f-bdd4l"]
Nov 03 18:14:30.651447 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:30.651325    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:31.123686 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.123640    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"]
Nov 03 18:14:31.124158 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.124138    6059 cpu_manager.go:414] "containerMap: removing container" podUID="dd844e4e-e51f-4f3d-85fb-c459a013c4d5" containerName="payload-processing"
Nov 03 18:14:31.124262 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.124159    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd844e4e-e51f-4f3d-85fb-c459a013c4d5" containerName="payload-processing"
Nov 03 18:14:31.124262 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.124248    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd844e4e-e51f-4f3d-85fb-c459a013c4d5" containerName="payload-processing"
Nov 03 18:14:31.128787 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.128766    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"
Nov 03 18:14:31.185702 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.185663    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b76e1d4d12c7\""
Nov 03 18:14:31.186124 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.186102    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 18:14:31.186502 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.186484    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 18:14:31.186722 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.186707    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 18:14:31.188767 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.188750    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 18:14:31.202243 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.202208    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eb41f807-0b83-4ace-ae43-662b644901e2-routing-overlay-volume\") pod \"payload-processing-external-model-69d86fd57c-4hmx4\" (UID: \"eb41f807-0b83-4ace-ae43-662b644901e2\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"
Nov 03 18:14:31.202453 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.202426    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eb41f807-0b83-4ace-ae43-662b644901e2-provider-credentials\") pod \"payload-processing-external-model-69d86fd57c-4hmx4\" (UID: \"eb41f807-0b83-4ace-ae43-662b644901e2\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"
Nov 03 18:14:31.202507 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.202471    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94ghj\" (UniqueName: \"kubernetes.io/projected/eb41f807-0b83-4ace-ae43-662b644901e2-kube-api-access-94ghj\") pod \"payload-processing-external-model-69d86fd57c-4hmx4\" (UID: \"eb41f807-0b83-4ace-ae43-662b644901e2\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"
Nov 03 18:14:31.202555 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.202535    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb41f807-0b83-4ace-ae43-662b644901e2-plugins-config-volume\") pod \"payload-processing-external-model-69d86fd57c-4hmx4\" (UID: \"eb41f807-0b83-4ace-ae43-662b644901e2\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"
Nov 03 18:14:31.252383 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.237585    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-489sh\""
Nov 03 18:14:31.252383 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.241034    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"]
Nov 03 18:14:31.303194 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.303146    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eb41f807-0b83-4ace-ae43-662b644901e2-provider-credentials\") pod \"payload-processing-external-model-69d86fd57c-4hmx4\" (UID: \"eb41f807-0b83-4ace-ae43-662b644901e2\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"
Nov 03 18:14:31.303194 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.303190    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94ghj\" (UniqueName: \"kubernetes.io/projected/eb41f807-0b83-4ace-ae43-662b644901e2-kube-api-access-94ghj\") pod \"payload-processing-external-model-69d86fd57c-4hmx4\" (UID: \"eb41f807-0b83-4ace-ae43-662b644901e2\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"
Nov 03 18:14:31.303736 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.303215    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb41f807-0b83-4ace-ae43-662b644901e2-plugins-config-volume\") pod \"payload-processing-external-model-69d86fd57c-4hmx4\" (UID: \"eb41f807-0b83-4ace-ae43-662b644901e2\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"
Nov 03 18:14:31.303736 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.303248    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eb41f807-0b83-4ace-ae43-662b644901e2-routing-overlay-volume\") pod \"payload-processing-external-model-69d86fd57c-4hmx4\" (UID: \"eb41f807-0b83-4ace-ae43-662b644901e2\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"
Nov 03 18:14:31.304005 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.303983    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eb41f807-0b83-4ace-ae43-662b644901e2-routing-overlay-volume\") pod \"payload-processing-external-model-69d86fd57c-4hmx4\" (UID: \"eb41f807-0b83-4ace-ae43-662b644901e2\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"
Nov 03 18:14:31.304045 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.304002    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb41f807-0b83-4ace-ae43-662b644901e2-plugins-config-volume\") pod \"payload-processing-external-model-69d86fd57c-4hmx4\" (UID: \"eb41f807-0b83-4ace-ae43-662b644901e2\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"
Nov 03 18:14:31.305699 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.305681    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eb41f807-0b83-4ace-ae43-662b644901e2-provider-credentials\") pod \"payload-processing-external-model-69d86fd57c-4hmx4\" (UID: \"eb41f807-0b83-4ace-ae43-662b644901e2\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"
Nov 03 18:14:31.368518 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.368479    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd844e4e-e51f-4f3d-85fb-c459a013c4d5" path="/var/lib/kubelet/pods/dd844e4e-e51f-4f3d-85fb-c459a013c4d5/volumes"
Nov 03 18:14:31.379625 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.379499    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94ghj\" (UniqueName: \"kubernetes.io/projected/eb41f807-0b83-4ace-ae43-662b644901e2-kube-api-access-94ghj\") pod \"payload-processing-external-model-69d86fd57c-4hmx4\" (UID: \"eb41f807-0b83-4ace-ae43-662b644901e2\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"
Nov 03 18:14:31.450080 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.449467    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"
Nov 03 18:14:31.894814 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:31.894775    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"]
Nov 03 18:14:31.896297 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:14:31.896237    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb41f807_0b83_4ace_ae43_662b644901e2.slice/crio-e2c05433ac4e362eb1f3a17b1522e253a7f4df62609f25e58cb4d22f2ca8f949 WatchSource:0}: Error finding container e2c05433ac4e362eb1f3a17b1522e253a7f4df62609f25e58cb4d22f2ca8f949: Status 404 returned error can't find the container with id e2c05433ac4e362eb1f3a17b1522e253a7f4df62609f25e58cb4d22f2ca8f949
Nov 03 18:14:32.296380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:32.295450    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4" event={"ID":"eb41f807-0b83-4ace-ae43-662b644901e2","Type":"ContainerStarted","Data":"81a2e166553adcea02af41b8e90832e7b50a52513f820fa49af10c2e8f304830"}
Nov 03 18:14:32.296380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:32.295524    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4" event={"ID":"eb41f807-0b83-4ace-ae43-662b644901e2","Type":"ContainerStarted","Data":"e2c05433ac4e362eb1f3a17b1522e253a7f4df62609f25e58cb4d22f2ca8f949"}
Nov 03 18:14:32.296380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:32.295729    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"
Nov 03 18:14:32.417470 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:32.417417    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4" podStartSLOduration=1.417404771 podStartE2EDuration="1.417404771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:32.417021667 +0000 UTC m=+3413.214734449" watchObservedRunningTime="2026-11-03 18:14:32.417404771 +0000 UTC m=+3413.215117551"
Nov 03 18:14:33.426258 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:33.426218    6059 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b76e1d4d12c7: secret "agc-cred-b76e1d4d12c7" not found
Nov 03 18:14:33.426258 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:33.426246    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4: secret "agc-cred-b76e1d4d12c7" not found
Nov 03 18:14:33.426775 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:33.426305    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb41f807-0b83-4ace-ae43-662b644901e2-provider-credentials podName:eb41f807-0b83-4ace-ae43-662b644901e2 nodeName:}" failed. No retries permitted until 2026-11-03 18:14:33.926284744 +0000 UTC m=+3414.723997525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/eb41f807-0b83-4ace-ae43-662b644901e2-provider-credentials") pod "payload-processing-external-model-69d86fd57c-4hmx4" (UID: "eb41f807-0b83-4ace-ae43-662b644901e2") : secret "agc-cred-b76e1d4d12c7" not found
Nov 03 18:14:33.616083 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:33.616051    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:33.793229 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:33.793190    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:33.932211 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:33.932174    6059 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b76e1d4d12c7: secret "agc-cred-b76e1d4d12c7" not found
Nov 03 18:14:33.932211 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:33.932204    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4: secret "agc-cred-b76e1d4d12c7" not found
Nov 03 18:14:33.932491 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:33.932253    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb41f807-0b83-4ace-ae43-662b644901e2-provider-credentials podName:eb41f807-0b83-4ace-ae43-662b644901e2 nodeName:}" failed. No retries permitted until 2026-11-03 18:14:34.932239563 +0000 UTC m=+3415.729952335 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/eb41f807-0b83-4ace-ae43-662b644901e2-provider-credentials") pod "payload-processing-external-model-69d86fd57c-4hmx4" (UID: "eb41f807-0b83-4ace-ae43-662b644901e2") : secret "agc-cred-b76e1d4d12c7" not found
Nov 03 18:14:34.116797 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:34.116708    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:34.333056 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:34.333017    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"]
Nov 03 18:14:34.501866 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:34.501831    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:34.944133 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:34.944037    6059 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b76e1d4d12c7: secret "agc-cred-b76e1d4d12c7" not found
Nov 03 18:14:34.944133 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:34.944069    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4: secret "agc-cred-b76e1d4d12c7" not found
Nov 03 18:14:34.944133 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:34.944123    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb41f807-0b83-4ace-ae43-662b644901e2-provider-credentials podName:eb41f807-0b83-4ace-ae43-662b644901e2 nodeName:}" failed. No retries permitted until 2026-11-03 18:14:36.944107831 +0000 UTC m=+3417.741820603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/eb41f807-0b83-4ace-ae43-662b644901e2-provider-credentials") pod "payload-processing-external-model-69d86fd57c-4hmx4" (UID: "eb41f807-0b83-4ace-ae43-662b644901e2") : secret "agc-cred-b76e1d4d12c7" not found
Nov 03 18:14:36.862119 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:36.862082    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:14:36.966336 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:36.966298    6059 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b76e1d4d12c7: secret "agc-cred-b76e1d4d12c7" not found
Nov 03 18:14:36.966336 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:36.966331    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4: secret "agc-cred-b76e1d4d12c7" not found
Nov 03 18:14:36.966587 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:36.966403    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb41f807-0b83-4ace-ae43-662b644901e2-provider-credentials podName:eb41f807-0b83-4ace-ae43-662b644901e2 nodeName:}" failed. No retries permitted until 2026-11-03 18:14:40.96638358 +0000 UTC m=+3421.764096358 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/eb41f807-0b83-4ace-ae43-662b644901e2-provider-credentials") pod "payload-processing-external-model-69d86fd57c-4hmx4" (UID: "eb41f807-0b83-4ace-ae43-662b644901e2") : secret "agc-cred-b76e1d4d12c7" not found
Nov 03 18:14:38.956331 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:38.956296    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:39.333732 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:39.333628    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4" podUID="eb41f807-0b83-4ace-ae43-662b644901e2" containerName="payload-processing" containerID="cri-o://81a2e166553adcea02af41b8e90832e7b50a52513f820fa49af10c2e8f304830" gracePeriod=25
Nov 03 18:14:39.590566 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:39.590496    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"
Nov 03 18:14:39.703229 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:39.703191    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb41f807-0b83-4ace-ae43-662b644901e2-kube-api-access-94ghj\" (UniqueName: \"kubernetes.io/projected/eb41f807-0b83-4ace-ae43-662b644901e2-kube-api-access-94ghj\") pod \"eb41f807-0b83-4ace-ae43-662b644901e2\" (UID: \"eb41f807-0b83-4ace-ae43-662b644901e2\") "
Nov 03 18:14:39.703451 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:39.703286    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb41f807-0b83-4ace-ae43-662b644901e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb41f807-0b83-4ace-ae43-662b644901e2-plugins-config-volume\") pod \"eb41f807-0b83-4ace-ae43-662b644901e2\" (UID: \"eb41f807-0b83-4ace-ae43-662b644901e2\") "
Nov 03 18:14:39.703451 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:39.703323    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb41f807-0b83-4ace-ae43-662b644901e2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/eb41f807-0b83-4ace-ae43-662b644901e2-provider-credentials\") pod \"eb41f807-0b83-4ace-ae43-662b644901e2\" (UID: \"eb41f807-0b83-4ace-ae43-662b644901e2\") "
Nov 03 18:14:39.703451 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:39.703373    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb41f807-0b83-4ace-ae43-662b644901e2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eb41f807-0b83-4ace-ae43-662b644901e2-routing-overlay-volume\") pod \"eb41f807-0b83-4ace-ae43-662b644901e2\" (UID: \"eb41f807-0b83-4ace-ae43-662b644901e2\") "
Nov 03 18:14:39.703757 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:39.703720    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb41f807-0b83-4ace-ae43-662b644901e2-plugins-config-volume" pod "eb41f807-0b83-4ace-ae43-662b644901e2" (UID: "eb41f807-0b83-4ace-ae43-662b644901e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:39.703757 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:39.703748    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb41f807-0b83-4ace-ae43-662b644901e2-routing-overlay-volume" pod "eb41f807-0b83-4ace-ae43-662b644901e2" (UID: "eb41f807-0b83-4ace-ae43-662b644901e2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:39.705522 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:39.705501    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb41f807-0b83-4ace-ae43-662b644901e2-kube-api-access-94ghj" pod "eb41f807-0b83-4ace-ae43-662b644901e2" (UID: "eb41f807-0b83-4ace-ae43-662b644901e2"). InnerVolumeSpecName "kube-api-access-94ghj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:39.705589 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:39.705570    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb41f807-0b83-4ace-ae43-662b644901e2-provider-credentials" pod "eb41f807-0b83-4ace-ae43-662b644901e2" (UID: "eb41f807-0b83-4ace-ae43-662b644901e2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:39.805027 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:39.804981    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94ghj\" (UniqueName: \"kubernetes.io/projected/eb41f807-0b83-4ace-ae43-662b644901e2-kube-api-access-94ghj\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:39.805027 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:39.805018    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb41f807-0b83-4ace-ae43-662b644901e2-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:39.805027 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:39.805028    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eb41f807-0b83-4ace-ae43-662b644901e2-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:39.805027 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:39.805037    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eb41f807-0b83-4ace-ae43-662b644901e2-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:39.915636 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:39.915538    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:40.092112 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:40.092065    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:40.353420 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:40.353385    6059 generic.go:356] "Generic (PLEG): container finished" podID="eb41f807-0b83-4ace-ae43-662b644901e2" containerID="81a2e166553adcea02af41b8e90832e7b50a52513f820fa49af10c2e8f304830" exitCode=0
Nov 03 18:14:40.353603 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:40.353460    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4" event={"ID":"eb41f807-0b83-4ace-ae43-662b644901e2","Type":"ContainerDied","Data":"81a2e166553adcea02af41b8e90832e7b50a52513f820fa49af10c2e8f304830"}
Nov 03 18:14:40.353603 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:40.353545    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4" event={"ID":"eb41f807-0b83-4ace-ae43-662b644901e2","Type":"ContainerDied","Data":"e2c05433ac4e362eb1f3a17b1522e253a7f4df62609f25e58cb4d22f2ca8f949"}
Nov 03 18:14:40.353603 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:40.353587    6059 scope.go:122] "RemoveContainer" containerID="81a2e166553adcea02af41b8e90832e7b50a52513f820fa49af10c2e8f304830"
Nov 03 18:14:40.354604 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:40.354581    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"
Nov 03 18:14:40.373328 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:40.373303    6059 scope.go:122] "RemoveContainer" containerID="81a2e166553adcea02af41b8e90832e7b50a52513f820fa49af10c2e8f304830"
Nov 03 18:14:40.373707 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:40.373687    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a2e166553adcea02af41b8e90832e7b50a52513f820fa49af10c2e8f304830\": container with ID starting with 81a2e166553adcea02af41b8e90832e7b50a52513f820fa49af10c2e8f304830 not found: ID does not exist" containerID="81a2e166553adcea02af41b8e90832e7b50a52513f820fa49af10c2e8f304830"
Nov 03 18:14:40.373781 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:40.373718    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a2e166553adcea02af41b8e90832e7b50a52513f820fa49af10c2e8f304830"} err="failed to get container status \"81a2e166553adcea02af41b8e90832e7b50a52513f820fa49af10c2e8f304830\": rpc error: code = NotFound desc = could not find container \"81a2e166553adcea02af41b8e90832e7b50a52513f820fa49af10c2e8f304830\": container with ID starting with 81a2e166553adcea02af41b8e90832e7b50a52513f820fa49af10c2e8f304830 not found: ID does not exist"
Nov 03 18:14:40.527591 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:40.527551    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"]
Nov 03 18:14:40.531727 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:40.531697    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d86fd57c-4hmx4"]
Nov 03 18:14:41.336318 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:41.336280    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb41f807-0b83-4ace-ae43-662b644901e2" path="/var/lib/kubelet/pods/eb41f807-0b83-4ace-ae43-662b644901e2/volumes"
Nov 03 18:14:41.572837 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:41.572800    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:42.069526 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.069483    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"]
Nov 03 18:14:42.069955 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.069940    6059 cpu_manager.go:414] "containerMap: removing container" podUID="eb41f807-0b83-4ace-ae43-662b644901e2" containerName="payload-processing"
Nov 03 18:14:42.070008 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.069956    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb41f807-0b83-4ace-ae43-662b644901e2" containerName="payload-processing"
Nov 03 18:14:42.070072 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.070063    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb41f807-0b83-4ace-ae43-662b644901e2" containerName="payload-processing"
Nov 03 18:14:42.074940 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.074911    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"
Nov 03 18:14:42.132842 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.132804    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667a5568-953f-47cf-8098-5e8c5e1062b6-plugins-config-volume\") pod \"payload-processing-external-model-69d86fd57c-cg7sz\" (UID: \"667a5568-953f-47cf-8098-5e8c5e1062b6\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"
Nov 03 18:14:42.133018 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.132966    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdxkb\" (UniqueName: \"kubernetes.io/projected/667a5568-953f-47cf-8098-5e8c5e1062b6-kube-api-access-mdxkb\") pod \"payload-processing-external-model-69d86fd57c-cg7sz\" (UID: \"667a5568-953f-47cf-8098-5e8c5e1062b6\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"
Nov 03 18:14:42.133079 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.133063    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/667a5568-953f-47cf-8098-5e8c5e1062b6-routing-overlay-volume\") pod \"payload-processing-external-model-69d86fd57c-cg7sz\" (UID: \"667a5568-953f-47cf-8098-5e8c5e1062b6\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"
Nov 03 18:14:42.133235 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.133201    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/667a5568-953f-47cf-8098-5e8c5e1062b6-provider-credentials\") pod \"payload-processing-external-model-69d86fd57c-cg7sz\" (UID: \"667a5568-953f-47cf-8098-5e8c5e1062b6\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"
Nov 03 18:14:42.183105 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.183061    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 18:14:42.183105 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.183061    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b76e1d4d12c7\""
Nov 03 18:14:42.183395 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.183118    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 18:14:42.183395 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.183118    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 18:14:42.183395 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.183146    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-5c56x\""
Nov 03 18:14:42.183395 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.183122    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 18:14:42.185465 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.185443    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"]
Nov 03 18:14:42.234520 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.234486    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/667a5568-953f-47cf-8098-5e8c5e1062b6-provider-credentials\") pod \"payload-processing-external-model-69d86fd57c-cg7sz\" (UID: \"667a5568-953f-47cf-8098-5e8c5e1062b6\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"
Nov 03 18:14:42.234801 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.234560    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667a5568-953f-47cf-8098-5e8c5e1062b6-plugins-config-volume\") pod \"payload-processing-external-model-69d86fd57c-cg7sz\" (UID: \"667a5568-953f-47cf-8098-5e8c5e1062b6\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"
Nov 03 18:14:42.234801 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.234635    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdxkb\" (UniqueName: \"kubernetes.io/projected/667a5568-953f-47cf-8098-5e8c5e1062b6-kube-api-access-mdxkb\") pod \"payload-processing-external-model-69d86fd57c-cg7sz\" (UID: \"667a5568-953f-47cf-8098-5e8c5e1062b6\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"
Nov 03 18:14:42.234801 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.234679    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/667a5568-953f-47cf-8098-5e8c5e1062b6-routing-overlay-volume\") pod \"payload-processing-external-model-69d86fd57c-cg7sz\" (UID: \"667a5568-953f-47cf-8098-5e8c5e1062b6\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"
Nov 03 18:14:42.235456 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.235432    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/667a5568-953f-47cf-8098-5e8c5e1062b6-routing-overlay-volume\") pod \"payload-processing-external-model-69d86fd57c-cg7sz\" (UID: \"667a5568-953f-47cf-8098-5e8c5e1062b6\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"
Nov 03 18:14:42.235555 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.235497    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667a5568-953f-47cf-8098-5e8c5e1062b6-plugins-config-volume\") pod \"payload-processing-external-model-69d86fd57c-cg7sz\" (UID: \"667a5568-953f-47cf-8098-5e8c5e1062b6\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"
Nov 03 18:14:42.237224 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.237188    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/667a5568-953f-47cf-8098-5e8c5e1062b6-provider-credentials\") pod \"payload-processing-external-model-69d86fd57c-cg7sz\" (UID: \"667a5568-953f-47cf-8098-5e8c5e1062b6\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"
Nov 03 18:14:42.297911 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.297872    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdxkb\" (UniqueName: \"kubernetes.io/projected/667a5568-953f-47cf-8098-5e8c5e1062b6-kube-api-access-mdxkb\") pod \"payload-processing-external-model-69d86fd57c-cg7sz\" (UID: \"667a5568-953f-47cf-8098-5e8c5e1062b6\") " pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"
Nov 03 18:14:42.388611 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.388494    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"
Nov 03 18:14:42.774799 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:14:42.774762    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667a5568_953f_47cf_8098_5e8c5e1062b6.slice/crio-120336e1337c636e48ba77e1c3cd51422486a531919092fa532cd52ff9d6313f WatchSource:0}: Error finding container 120336e1337c636e48ba77e1c3cd51422486a531919092fa532cd52ff9d6313f: Status 404 returned error can't find the container with id 120336e1337c636e48ba77e1c3cd51422486a531919092fa532cd52ff9d6313f
Nov 03 18:14:42.777154 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:42.777129    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"]
Nov 03 18:14:43.391537 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:43.391502    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz" event={"ID":"667a5568-953f-47cf-8098-5e8c5e1062b6","Type":"ContainerStarted","Data":"22b8c718d073f39acb3a03905e20e9d6f2c885db7611bef0cc5fcf89b80bbe16"}
Nov 03 18:14:43.391973 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:43.391554    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz" event={"ID":"667a5568-953f-47cf-8098-5e8c5e1062b6","Type":"ContainerStarted","Data":"120336e1337c636e48ba77e1c3cd51422486a531919092fa532cd52ff9d6313f"}
Nov 03 18:14:43.391973 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:43.391576    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"
Nov 03 18:14:43.513458 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:43.513386    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz" podStartSLOduration=1.513365475 podStartE2EDuration="1.513365475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:43.511032776 +0000 UTC m=+3424.308745557" watchObservedRunningTime="2026-11-03 18:14:43.513365475 +0000 UTC m=+3424.311078251"
Nov 03 18:14:43.856401 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:43.856335    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:46.689753 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:46.689715    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"]
Nov 03 18:14:46.699388 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:46.699347    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:46.876729 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:46.876689    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:48.670984 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:48.670941    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:48.858261 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:48.858217    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:49.057634 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:49.057585    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:49.913045 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:49.913005    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:51.690191 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:51.690137    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz" podUID="667a5568-953f-47cf-8098-5e8c5e1062b6" containerName="payload-processing" containerID="cri-o://22b8c718d073f39acb3a03905e20e9d6f2c885db7611bef0cc5fcf89b80bbe16" gracePeriod=25
Nov 03 18:14:51.942175 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:51.942110    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"
Nov 03 18:14:52.034724 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:52.034691    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/667a5568-953f-47cf-8098-5e8c5e1062b6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/667a5568-953f-47cf-8098-5e8c5e1062b6-provider-credentials\") pod \"667a5568-953f-47cf-8098-5e8c5e1062b6\" (UID: \"667a5568-953f-47cf-8098-5e8c5e1062b6\") "
Nov 03 18:14:52.034925 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:52.034765    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/667a5568-953f-47cf-8098-5e8c5e1062b6-kube-api-access-mdxkb\" (UniqueName: \"kubernetes.io/projected/667a5568-953f-47cf-8098-5e8c5e1062b6-kube-api-access-mdxkb\") pod \"667a5568-953f-47cf-8098-5e8c5e1062b6\" (UID: \"667a5568-953f-47cf-8098-5e8c5e1062b6\") "
Nov 03 18:14:52.034925 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:52.034890    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/667a5568-953f-47cf-8098-5e8c5e1062b6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/667a5568-953f-47cf-8098-5e8c5e1062b6-routing-overlay-volume\") pod \"667a5568-953f-47cf-8098-5e8c5e1062b6\" (UID: \"667a5568-953f-47cf-8098-5e8c5e1062b6\") "
Nov 03 18:14:52.035028 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:52.034983    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/667a5568-953f-47cf-8098-5e8c5e1062b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667a5568-953f-47cf-8098-5e8c5e1062b6-plugins-config-volume\") pod \"667a5568-953f-47cf-8098-5e8c5e1062b6\" (UID: \"667a5568-953f-47cf-8098-5e8c5e1062b6\") "
Nov 03 18:14:52.035282 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:52.035251    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/667a5568-953f-47cf-8098-5e8c5e1062b6-routing-overlay-volume" pod "667a5568-953f-47cf-8098-5e8c5e1062b6" (UID: "667a5568-953f-47cf-8098-5e8c5e1062b6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:52.035476 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:52.035450    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/667a5568-953f-47cf-8098-5e8c5e1062b6-plugins-config-volume" pod "667a5568-953f-47cf-8098-5e8c5e1062b6" (UID: "667a5568-953f-47cf-8098-5e8c5e1062b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:52.035577 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:52.035501    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/667a5568-953f-47cf-8098-5e8c5e1062b6-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.037029 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:52.037002    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667a5568-953f-47cf-8098-5e8c5e1062b6-provider-credentials" pod "667a5568-953f-47cf-8098-5e8c5e1062b6" (UID: "667a5568-953f-47cf-8098-5e8c5e1062b6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:52.037131 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:52.037040    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667a5568-953f-47cf-8098-5e8c5e1062b6-kube-api-access-mdxkb" pod "667a5568-953f-47cf-8098-5e8c5e1062b6" (UID: "667a5568-953f-47cf-8098-5e8c5e1062b6"). InnerVolumeSpecName "kube-api-access-mdxkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:52.136651 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:52.136604    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/667a5568-953f-47cf-8098-5e8c5e1062b6-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.136651 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:52.136642    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdxkb\" (UniqueName: \"kubernetes.io/projected/667a5568-953f-47cf-8098-5e8c5e1062b6-kube-api-access-mdxkb\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.136651 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:52.136656    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667a5568-953f-47cf-8098-5e8c5e1062b6-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.452625 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:52.452592    6059 generic.go:356] "Generic (PLEG): container finished" podID="667a5568-953f-47cf-8098-5e8c5e1062b6" containerID="22b8c718d073f39acb3a03905e20e9d6f2c885db7611bef0cc5fcf89b80bbe16" exitCode=0
Nov 03 18:14:52.452787 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:52.452642    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz" event={"ID":"667a5568-953f-47cf-8098-5e8c5e1062b6","Type":"ContainerDied","Data":"22b8c718d073f39acb3a03905e20e9d6f2c885db7611bef0cc5fcf89b80bbe16"}
Nov 03 18:14:52.452787 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:52.452647    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"
Nov 03 18:14:52.452787 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:52.452686    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz" event={"ID":"667a5568-953f-47cf-8098-5e8c5e1062b6","Type":"ContainerDied","Data":"120336e1337c636e48ba77e1c3cd51422486a531919092fa532cd52ff9d6313f"}
Nov 03 18:14:52.452787 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:52.452708    6059 scope.go:122] "RemoveContainer" containerID="22b8c718d073f39acb3a03905e20e9d6f2c885db7611bef0cc5fcf89b80bbe16"
Nov 03 18:14:52.463661 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:52.463640    6059 scope.go:122] "RemoveContainer" containerID="22b8c718d073f39acb3a03905e20e9d6f2c885db7611bef0cc5fcf89b80bbe16"
Nov 03 18:14:52.464002 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:52.463979    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b8c718d073f39acb3a03905e20e9d6f2c885db7611bef0cc5fcf89b80bbe16\": container with ID starting with 22b8c718d073f39acb3a03905e20e9d6f2c885db7611bef0cc5fcf89b80bbe16 not found: ID does not exist" containerID="22b8c718d073f39acb3a03905e20e9d6f2c885db7611bef0cc5fcf89b80bbe16"
Nov 03 18:14:52.464118 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:52.464008    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b8c718d073f39acb3a03905e20e9d6f2c885db7611bef0cc5fcf89b80bbe16"} err="failed to get container status \"22b8c718d073f39acb3a03905e20e9d6f2c885db7611bef0cc5fcf89b80bbe16\": rpc error: code = NotFound desc = could not find container \"22b8c718d073f39acb3a03905e20e9d6f2c885db7611bef0cc5fcf89b80bbe16\": container with ID starting with 22b8c718d073f39acb3a03905e20e9d6f2c885db7611bef0cc5fcf89b80bbe16 not found: ID does not exist"
Nov 03 18:14:52.515549 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:52.515513    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:52.623129 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:52.623087    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"]
Nov 03 18:14:52.674802 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:52.674763    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d86fd57c-cg7sz"]
Nov 03 18:14:53.014791 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.014753    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"]
Nov 03 18:14:53.015226 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.015213    6059 cpu_manager.go:414] "containerMap: removing container" podUID="667a5568-953f-47cf-8098-5e8c5e1062b6" containerName="payload-processing"
Nov 03 18:14:53.015276 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.015226    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="667a5568-953f-47cf-8098-5e8c5e1062b6" containerName="payload-processing"
Nov 03 18:14:53.015310 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.015294    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="667a5568-953f-47cf-8098-5e8c5e1062b6" containerName="payload-processing"
Nov 03 18:14:53.019908 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.019881    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"
Nov 03 18:14:53.045227 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.045192    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-routing-overlay-volume\") pod \"payload-processing-external-model-748d54b9b-xfd9p\" (UID: \"2a7a8d36-2439-48b1-987f-f0fe02c4ade4\") " pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"
Nov 03 18:14:53.045431 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.045235    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgp74\" (UniqueName: \"kubernetes.io/projected/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-kube-api-access-rgp74\") pod \"payload-processing-external-model-748d54b9b-xfd9p\" (UID: \"2a7a8d36-2439-48b1-987f-f0fe02c4ade4\") " pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"
Nov 03 18:14:53.045431 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.045325    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-plugins-config-volume\") pod \"payload-processing-external-model-748d54b9b-xfd9p\" (UID: \"2a7a8d36-2439-48b1-987f-f0fe02c4ade4\") " pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"
Nov 03 18:14:53.045505 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.045469    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-provider-credentials\") pod \"payload-processing-external-model-748d54b9b-xfd9p\" (UID: \"2a7a8d36-2439-48b1-987f-f0fe02c4ade4\") " pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"
Nov 03 18:14:53.077121 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.076925    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 18:14:53.077121 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.077034    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 18:14:53.077121 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.077091    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-850dcd82d83c\""
Nov 03 18:14:53.077121 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.077092    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 18:14:53.080151 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.080115    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"]
Nov 03 18:14:53.128165 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.128120    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-gbrqq\""
Nov 03 18:14:53.128165 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.128127    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 18:14:53.146844 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.146808    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-provider-credentials\") pod \"payload-processing-external-model-748d54b9b-xfd9p\" (UID: \"2a7a8d36-2439-48b1-987f-f0fe02c4ade4\") " pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"
Nov 03 18:14:53.147073 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.146915    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-routing-overlay-volume\") pod \"payload-processing-external-model-748d54b9b-xfd9p\" (UID: \"2a7a8d36-2439-48b1-987f-f0fe02c4ade4\") " pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"
Nov 03 18:14:53.147073 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.146950    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgp74\" (UniqueName: \"kubernetes.io/projected/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-kube-api-access-rgp74\") pod \"payload-processing-external-model-748d54b9b-xfd9p\" (UID: \"2a7a8d36-2439-48b1-987f-f0fe02c4ade4\") " pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"
Nov 03 18:14:53.147073 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.146997    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-plugins-config-volume\") pod \"payload-processing-external-model-748d54b9b-xfd9p\" (UID: \"2a7a8d36-2439-48b1-987f-f0fe02c4ade4\") " pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"
Nov 03 18:14:53.147628 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.147603    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-routing-overlay-volume\") pod \"payload-processing-external-model-748d54b9b-xfd9p\" (UID: \"2a7a8d36-2439-48b1-987f-f0fe02c4ade4\") " pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"
Nov 03 18:14:53.148138 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.148080    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-plugins-config-volume\") pod \"payload-processing-external-model-748d54b9b-xfd9p\" (UID: \"2a7a8d36-2439-48b1-987f-f0fe02c4ade4\") " pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"
Nov 03 18:14:53.149851 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.149831    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-provider-credentials\") pod \"payload-processing-external-model-748d54b9b-xfd9p\" (UID: \"2a7a8d36-2439-48b1-987f-f0fe02c4ade4\") " pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"
Nov 03 18:14:53.211050 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.211012    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgp74\" (UniqueName: \"kubernetes.io/projected/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-kube-api-access-rgp74\") pod \"payload-processing-external-model-748d54b9b-xfd9p\" (UID: \"2a7a8d36-2439-48b1-987f-f0fe02c4ade4\") " pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"
Nov 03 18:14:53.333284 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.333165    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"
Nov 03 18:14:53.335789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.335755    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="667a5568-953f-47cf-8098-5e8c5e1062b6" path="/var/lib/kubelet/pods/667a5568-953f-47cf-8098-5e8c5e1062b6/volumes"
Nov 03 18:14:53.663769 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.663680    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:53.707962 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:14:53.707929    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7a8d36_2439_48b1_987f_f0fe02c4ade4.slice/crio-bdeff8db1da6ef51c951f54c064fffa65551f4c825af377e8f05f60fc1208f7c WatchSource:0}: Error finding container bdeff8db1da6ef51c951f54c064fffa65551f4c825af377e8f05f60fc1208f7c: Status 404 returned error can't find the container with id bdeff8db1da6ef51c951f54c064fffa65551f4c825af377e8f05f60fc1208f7c
Nov 03 18:14:53.709166 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:53.709137    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"]
Nov 03 18:14:54.481690 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:54.481648    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p" event={"ID":"2a7a8d36-2439-48b1-987f-f0fe02c4ade4","Type":"ContainerStarted","Data":"94c8d2bc9596cc97642b441b2025a2345135d3936b2ab1e47d5b7df8587cca5a"}
Nov 03 18:14:54.482244 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:54.481715    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p" event={"ID":"2a7a8d36-2439-48b1-987f-f0fe02c4ade4","Type":"ContainerStarted","Data":"bdeff8db1da6ef51c951f54c064fffa65551f4c825af377e8f05f60fc1208f7c"}
Nov 03 18:14:54.482244 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:54.481879    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"
Nov 03 18:14:54.602870 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:54.602798    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p" podStartSLOduration=2.602780395 podStartE2EDuration="2.602780395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:54.601103077 +0000 UTC m=+3435.398815879" watchObservedRunningTime="2026-11-03 18:14:54.602780395 +0000 UTC m=+3435.400493205"
Nov 03 18:14:55.400767 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:55.400706    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:55.582546 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:55.582507    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:55.600489 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:55.600444    6059 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-850dcd82d83c: secret "agc-cred-850dcd82d83c" not found
Nov 03 18:14:55.600489 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:55.600484    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p: secret "agc-cred-850dcd82d83c" not found
Nov 03 18:14:55.600763 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:55.600561    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-provider-credentials podName:2a7a8d36-2439-48b1-987f-f0fe02c4ade4 nodeName:}" failed. No retries permitted until 2026-11-03 18:14:56.100536822 +0000 UTC m=+3436.898249601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-provider-credentials") pod "payload-processing-external-model-748d54b9b-xfd9p" (UID: "2a7a8d36-2439-48b1-987f-f0fe02c4ade4") : secret "agc-cred-850dcd82d83c" not found
Nov 03 18:14:56.105461 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:56.105424    6059 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-850dcd82d83c: secret "agc-cred-850dcd82d83c" not found
Nov 03 18:14:56.105461 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:56.105456    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p: secret "agc-cred-850dcd82d83c" not found
Nov 03 18:14:56.105689 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:56.105505    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-provider-credentials podName:2a7a8d36-2439-48b1-987f-f0fe02c4ade4 nodeName:}" failed. No retries permitted until 2026-11-03 18:14:57.105489525 +0000 UTC m=+3437.903202298 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-provider-credentials") pod "payload-processing-external-model-748d54b9b-xfd9p" (UID: "2a7a8d36-2439-48b1-987f-f0fe02c4ade4") : secret "agc-cred-850dcd82d83c" not found
Nov 03 18:14:56.149249 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:56.149187    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"]
Nov 03 18:14:56.315887 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:56.315793    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:56.856612 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:56.856561    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6trbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:14:56.857087 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:56.856664    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr" podUID="6af1e128-ecfb-4449-a2ab-0d6a3732ca71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:14:57.120385 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:57.120265    6059 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-850dcd82d83c: secret "agc-cred-850dcd82d83c" not found
Nov 03 18:14:57.120385 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:57.120297    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p: secret "agc-cred-850dcd82d83c" not found
Nov 03 18:14:57.120385 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:57.120348    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-provider-credentials podName:2a7a8d36-2439-48b1-987f-f0fe02c4ade4 nodeName:}" failed. No retries permitted until 2026-11-03 18:14:59.120334223 +0000 UTC m=+3439.918046994 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-provider-credentials") pod "payload-processing-external-model-748d54b9b-xfd9p" (UID: "2a7a8d36-2439-48b1-987f-f0fe02c4ade4") : secret "agc-cred-850dcd82d83c" not found
Nov 03 18:14:58.219076 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:58.219038    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:58.410939 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:58.410898    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:58.593759 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:58.593655    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:58.776391 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:58.776331    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:59.142683 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:59.142637    6059 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-850dcd82d83c: secret "agc-cred-850dcd82d83c" not found
Nov 03 18:14:59.142683 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:59.142668    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p: secret "agc-cred-850dcd82d83c" not found
Nov 03 18:14:59.142975 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:14:59.142737    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-provider-credentials podName:2a7a8d36-2439-48b1-987f-f0fe02c4ade4 nodeName:}" failed. No retries permitted until 2026-11-03 18:15:03.142715589 +0000 UTC m=+3443.940428360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-provider-credentials") pod "payload-processing-external-model-748d54b9b-xfd9p" (UID: "2a7a8d36-2439-48b1-987f-f0fe02c4ade4") : secret "agc-cred-850dcd82d83c" not found
Nov 03 18:14:59.583085 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:59.583052    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:14:59.770795 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:14:59.770760    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:00.165719 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.165680    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd"]
Nov 03 18:15:00.170138 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.170113    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd"
Nov 03 18:15:00.170320 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.170120    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs"]
Nov 03 18:15:00.174005 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.173981    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs"
Nov 03 18:15:00.224782 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.224750    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:15:00.224972 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.224862    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:15:00.254632 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.254594    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1cd9469-3e37-4b68-9cd5-9936f7dcff6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd\" (UID: \"b1cd9469-3e37-4b68-9cd5-9936f7dcff6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd"
Nov 03 18:15:00.254849 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.254667    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895495-v5mvs\" (UID: \"38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs"
Nov 03 18:15:00.254919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.254879    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b67kb\" (UniqueName: \"kubernetes.io/projected/b1cd9469-3e37-4b68-9cd5-9936f7dcff6d-kube-api-access-b67kb\") pod \"maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd\" (UID: \"b1cd9469-3e37-4b68-9cd5-9936f7dcff6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd"
Nov 03 18:15:00.254979 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.254928    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5z9d\" (UniqueName: \"kubernetes.io/projected/38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7-kube-api-access-l5z9d\") pod \"maas-api-key-cleanup-29895495-v5mvs\" (UID: \"38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs"
Nov 03 18:15:00.276406 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.276322    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 18:15:00.279174 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.279138    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd"]
Nov 03 18:15:00.283203 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.283166    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs"]
Nov 03 18:15:00.356263 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.356224    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b67kb\" (UniqueName: \"kubernetes.io/projected/b1cd9469-3e37-4b68-9cd5-9936f7dcff6d-kube-api-access-b67kb\") pod \"maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd\" (UID: \"b1cd9469-3e37-4b68-9cd5-9936f7dcff6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd"
Nov 03 18:15:00.356469 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.356275    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5z9d\" (UniqueName: \"kubernetes.io/projected/38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7-kube-api-access-l5z9d\") pod \"maas-api-key-cleanup-29895495-v5mvs\" (UID: \"38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs"
Nov 03 18:15:00.356469 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.356321    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1cd9469-3e37-4b68-9cd5-9936f7dcff6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd\" (UID: \"b1cd9469-3e37-4b68-9cd5-9936f7dcff6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd"
Nov 03 18:15:00.356469 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.356395    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895495-v5mvs\" (UID: \"38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs"
Nov 03 18:15:00.357178 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.357154    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1cd9469-3e37-4b68-9cd5-9936f7dcff6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd\" (UID: \"b1cd9469-3e37-4b68-9cd5-9936f7dcff6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd"
Nov 03 18:15:00.357259 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.357154    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895495-v5mvs\" (UID: \"38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs"
Nov 03 18:15:00.383566 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.383531    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"]
Nov 03 18:15:00.387590 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.387565    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.420126 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.420041    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5z9d\" (UniqueName: \"kubernetes.io/projected/38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7-kube-api-access-l5z9d\") pod \"maas-api-key-cleanup-29895495-v5mvs\" (UID: \"38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs"
Nov 03 18:15:00.420709 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.420681    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b67kb\" (UniqueName: \"kubernetes.io/projected/b1cd9469-3e37-4b68-9cd5-9936f7dcff6d-kube-api-access-b67kb\") pod \"maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd\" (UID: \"b1cd9469-3e37-4b68-9cd5-9936f7dcff6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd"
Nov 03 18:15:00.452063 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.452019    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"]
Nov 03 18:15:00.457475 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.457442    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.457652 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.457502    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.457652 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.457596    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.457771 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.457749    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.457822 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.457807    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.457923 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.457907    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.458058 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.458038    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.458147 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.458125    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc2qp\" (UniqueName: \"kubernetes.io/projected/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-kube-api-access-gc2qp\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.458212 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.458187    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.486537 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.486500    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd"
Nov 03 18:15:00.493017 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.492985    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs"
Nov 03 18:15:00.560602 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.560565    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.560737 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.560613    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.560737 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.560662    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.560737 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.560701    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.560737 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.560726    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.560991 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.560821    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.560991 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.560857    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc2qp\" (UniqueName: \"kubernetes.io/projected/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-kube-api-access-gc2qp\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.560991 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.560902    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.560991 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.560939    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.563579 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.562104    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.569740 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.569501    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.571305 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.570489    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.571305 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.570955    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.571305 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.571015    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.571305 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.571260    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.580749 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.580673    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.641583 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.641551    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.641956 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.641932    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc2qp\" (UniqueName: \"kubernetes.io/projected/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-kube-api-access-gc2qp\") pod \"maas-default-gateway-openshift-default-58c6585447-8kglw\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.701172 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.701081    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:00.873249 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:15:00.873208    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cd9469_3e37_4b68_9cd5_9936f7dcff6d.slice/crio-4a6249ae530d7d7b506a58bd1a83c9817a032c55cd944116cfa51553adcef327 WatchSource:0}: Error finding container 4a6249ae530d7d7b506a58bd1a83c9817a032c55cd944116cfa51553adcef327: Status 404 returned error can't find the container with id 4a6249ae530d7d7b506a58bd1a83c9817a032c55cd944116cfa51553adcef327
Nov 03 18:15:00.875205 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.875181    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd"]
Nov 03 18:15:00.924555 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:15:00.924512    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e9e6f1_bf5b_49c6_a28b_5e223c62e9a7.slice/crio-1f129d39368f5bc808e842025018367b3b15a352f546f62d384c742829d49fe5 WatchSource:0}: Error finding container 1f129d39368f5bc808e842025018367b3b15a352f546f62d384c742829d49fe5: Status 404 returned error can't find the container with id 1f129d39368f5bc808e842025018367b3b15a352f546f62d384c742829d49fe5
Nov 03 18:15:00.927256 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:00.927229    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs"]
Nov 03 18:15:01.082409 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.082373    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"]
Nov 03 18:15:01.131898 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:15:01.131852    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbcbf9e3_2774_4872_95d5_4f27b7ad17cc.slice/crio-eae33fa97d3d1c4d15316ea9ac2900433f74b81af91f08701fde5390cb047918 WatchSource:0}: Error finding container eae33fa97d3d1c4d15316ea9ac2900433f74b81af91f08701fde5390cb047918: Status 404 returned error can't find the container with id eae33fa97d3d1c4d15316ea9ac2900433f74b81af91f08701fde5390cb047918
Nov 03 18:15:01.134680 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.134647    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:15:01.134797 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.134713    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:15:01.134797 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.134746    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:15:01.500655 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.500560    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p" podUID="2a7a8d36-2439-48b1-987f-f0fe02c4ade4" containerName="payload-processing" containerID="cri-o://94c8d2bc9596cc97642b441b2025a2345135d3936b2ab1e47d5b7df8587cca5a" gracePeriod=25
Nov 03 18:15:01.538010 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.537973    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd" event={"ID":"b1cd9469-3e37-4b68-9cd5-9936f7dcff6d","Type":"ContainerStarted","Data":"012b183c9bc1c4926ec69c2de9eaabd9ba5510738b96bd2c152c522efa4825de"}
Nov 03 18:15:01.538210 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.538032    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd" event={"ID":"b1cd9469-3e37-4b68-9cd5-9936f7dcff6d","Type":"ContainerStarted","Data":"4a6249ae530d7d7b506a58bd1a83c9817a032c55cd944116cfa51553adcef327"}
Nov 03 18:15:01.539696 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.539668    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs" event={"ID":"38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7","Type":"ContainerStarted","Data":"275345e4675e0b9be1d85acd95983b6ea81ee2850845c261709f725b711db1db"}
Nov 03 18:15:01.539824 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.539722    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs" event={"ID":"38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7","Type":"ContainerStarted","Data":"1f129d39368f5bc808e842025018367b3b15a352f546f62d384c742829d49fe5"}
Nov 03 18:15:01.541427 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.541399    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw" event={"ID":"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc","Type":"ContainerStarted","Data":"aacbd52827741c953232fcb544c82cb856e25476e3b584675e16631e8b7382b7"}
Nov 03 18:15:01.541553 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.541445    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw" event={"ID":"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc","Type":"ContainerStarted","Data":"eae33fa97d3d1c4d15316ea9ac2900433f74b81af91f08701fde5390cb047918"}
Nov 03 18:15:01.702015 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.701986    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:01.786989 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.786866    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd" podStartSLOduration=1.786847005 podStartE2EDuration="1.786847005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:01.661164933 +0000 UTC m=+3442.458877747" watchObservedRunningTime="2026-11-03 18:15:01.786847005 +0000 UTC m=+3442.584559786"
Nov 03 18:15:01.789079 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.789034    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw" podStartSLOduration=1.789023696 podStartE2EDuration="1.789023696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:01.785956174 +0000 UTC m=+3442.583668949" watchObservedRunningTime="2026-11-03 18:15:01.789023696 +0000 UTC m=+3442.586736476"
Nov 03 18:15:01.813264 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.813234    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"
Nov 03 18:15:01.876329 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.876294    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-provider-credentials\") pod \"2a7a8d36-2439-48b1-987f-f0fe02c4ade4\" (UID: \"2a7a8d36-2439-48b1-987f-f0fe02c4ade4\") "
Nov 03 18:15:01.876523 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.876398    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-routing-overlay-volume\") pod \"2a7a8d36-2439-48b1-987f-f0fe02c4ade4\" (UID: \"2a7a8d36-2439-48b1-987f-f0fe02c4ade4\") "
Nov 03 18:15:01.876523 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.876421    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-kube-api-access-rgp74\" (UniqueName: \"kubernetes.io/projected/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-kube-api-access-rgp74\") pod \"2a7a8d36-2439-48b1-987f-f0fe02c4ade4\" (UID: \"2a7a8d36-2439-48b1-987f-f0fe02c4ade4\") "
Nov 03 18:15:01.876523 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.876519    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-plugins-config-volume\") pod \"2a7a8d36-2439-48b1-987f-f0fe02c4ade4\" (UID: \"2a7a8d36-2439-48b1-987f-f0fe02c4ade4\") "
Nov 03 18:15:01.876816 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.876791    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-routing-overlay-volume" pod "2a7a8d36-2439-48b1-987f-f0fe02c4ade4" (UID: "2a7a8d36-2439-48b1-987f-f0fe02c4ade4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:01.876930 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.876905    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-plugins-config-volume" pod "2a7a8d36-2439-48b1-987f-f0fe02c4ade4" (UID: "2a7a8d36-2439-48b1-987f-f0fe02c4ade4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:01.878572 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.878546    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-provider-credentials" pod "2a7a8d36-2439-48b1-987f-f0fe02c4ade4" (UID: "2a7a8d36-2439-48b1-987f-f0fe02c4ade4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:01.878572 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.878568    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-kube-api-access-rgp74" pod "2a7a8d36-2439-48b1-987f-f0fe02c4ade4" (UID: "2a7a8d36-2439-48b1-987f-f0fe02c4ade4"). InnerVolumeSpecName "kube-api-access-rgp74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:01.922323 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.922269    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs" podStartSLOduration=1.922253056 podStartE2EDuration="1.922253056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:01.921536332 +0000 UTC m=+3442.719249114" watchObservedRunningTime="2026-11-03 18:15:01.922253056 +0000 UTC m=+3442.719965831"
Nov 03 18:15:01.978097 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.978059    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.978097 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.978091    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.978097 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.978103    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.978385 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:01.978114    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgp74\" (UniqueName: \"kubernetes.io/projected/2a7a8d36-2439-48b1-987f-f0fe02c4ade4-kube-api-access-rgp74\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:02.099188 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.099064    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:02.560299 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.560256    6059 generic.go:356] "Generic (PLEG): container finished" podID="2a7a8d36-2439-48b1-987f-f0fe02c4ade4" containerID="94c8d2bc9596cc97642b441b2025a2345135d3936b2ab1e47d5b7df8587cca5a" exitCode=0
Nov 03 18:15:02.560299 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.560286    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p" event={"ID":"2a7a8d36-2439-48b1-987f-f0fe02c4ade4","Type":"ContainerDied","Data":"94c8d2bc9596cc97642b441b2025a2345135d3936b2ab1e47d5b7df8587cca5a"}
Nov 03 18:15:02.560618 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.560337    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p" event={"ID":"2a7a8d36-2439-48b1-987f-f0fe02c4ade4","Type":"ContainerDied","Data":"bdeff8db1da6ef51c951f54c064fffa65551f4c825af377e8f05f60fc1208f7c"}
Nov 03 18:15:02.560618 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.560340    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"
Nov 03 18:15:02.560618 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.560381    6059 scope.go:122] "RemoveContainer" containerID="94c8d2bc9596cc97642b441b2025a2345135d3936b2ab1e47d5b7df8587cca5a"
Nov 03 18:15:02.562204 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.562181    6059 generic.go:356] "Generic (PLEG): container finished" podID="b1cd9469-3e37-4b68-9cd5-9936f7dcff6d" containerID="012b183c9bc1c4926ec69c2de9eaabd9ba5510738b96bd2c152c522efa4825de" exitCode=60
Nov 03 18:15:02.562331 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.562265    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd" event={"ID":"b1cd9469-3e37-4b68-9cd5-9936f7dcff6d","Type":"ContainerDied","Data":"012b183c9bc1c4926ec69c2de9eaabd9ba5510738b96bd2c152c522efa4825de"}
Nov 03 18:15:02.562578 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.562559    6059 scope.go:122] "RemoveContainer" containerID="012b183c9bc1c4926ec69c2de9eaabd9ba5510738b96bd2c152c522efa4825de"
Nov 03 18:15:02.564699 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.564676    6059 generic.go:356] "Generic (PLEG): container finished" podID="38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7" containerID="275345e4675e0b9be1d85acd95983b6ea81ee2850845c261709f725b711db1db" exitCode=60
Nov 03 18:15:02.564793 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.564762    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs" event={"ID":"38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7","Type":"ContainerDied","Data":"275345e4675e0b9be1d85acd95983b6ea81ee2850845c261709f725b711db1db"}
Nov 03 18:15:02.564951 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.564936    6059 scope.go:122] "RemoveContainer" containerID="275345e4675e0b9be1d85acd95983b6ea81ee2850845c261709f725b711db1db"
Nov 03 18:15:02.579786 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.579759    6059 scope.go:122] "RemoveContainer" containerID="94c8d2bc9596cc97642b441b2025a2345135d3936b2ab1e47d5b7df8587cca5a"
Nov 03 18:15:02.580198 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:15:02.580171    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c8d2bc9596cc97642b441b2025a2345135d3936b2ab1e47d5b7df8587cca5a\": container with ID starting with 94c8d2bc9596cc97642b441b2025a2345135d3936b2ab1e47d5b7df8587cca5a not found: ID does not exist" containerID="94c8d2bc9596cc97642b441b2025a2345135d3936b2ab1e47d5b7df8587cca5a"
Nov 03 18:15:02.580335 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.580209    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c8d2bc9596cc97642b441b2025a2345135d3936b2ab1e47d5b7df8587cca5a"} err="failed to get container status \"94c8d2bc9596cc97642b441b2025a2345135d3936b2ab1e47d5b7df8587cca5a\": rpc error: code = NotFound desc = could not find container \"94c8d2bc9596cc97642b441b2025a2345135d3936b2ab1e47d5b7df8587cca5a\": container with ID starting with 94c8d2bc9596cc97642b441b2025a2345135d3936b2ab1e47d5b7df8587cca5a not found: ID does not exist"
Nov 03 18:15:02.584571 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.584546    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"]
Nov 03 18:15:02.585100 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.585080    6059 cpu_manager.go:414] "containerMap: removing container" podUID="2a7a8d36-2439-48b1-987f-f0fe02c4ade4" containerName="payload-processing"
Nov 03 18:15:02.585100 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.585097    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a7a8d36-2439-48b1-987f-f0fe02c4ade4" containerName="payload-processing"
Nov 03 18:15:02.585301 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.585219    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a7a8d36-2439-48b1-987f-f0fe02c4ade4" containerName="payload-processing"
Nov 03 18:15:02.590236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.590215    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"
Nov 03 18:15:02.655381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.654579    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"openshift-service-ca.crt\""
Nov 03 18:15:02.655381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.654663    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"agc-cred-7e9aa47b0ebd\""
Nov 03 18:15:02.657462 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.657420    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"]
Nov 03 18:15:02.686664 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.686629    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43e8a2ca-2687-475e-8b25-0d04734a5314-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm\" (UID: \"43e8a2ca-2687-475e-8b25-0d04734a5314\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"
Nov 03 18:15:02.686849 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.686678    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e8a2ca-2687-475e-8b25-0d04734a5314-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm\" (UID: \"43e8a2ca-2687-475e-8b25-0d04734a5314\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"
Nov 03 18:15:02.686849 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.686772    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvmd5\" (UniqueName: \"kubernetes.io/projected/43e8a2ca-2687-475e-8b25-0d04734a5314-kube-api-access-zvmd5\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm\" (UID: \"43e8a2ca-2687-475e-8b25-0d04734a5314\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"
Nov 03 18:15:02.686924 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.686864    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43e8a2ca-2687-475e-8b25-0d04734a5314-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm\" (UID: \"43e8a2ca-2687-475e-8b25-0d04734a5314\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"
Nov 03 18:15:02.703042 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.702983    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8kglw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 18:15:02.703467 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.703059    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw" podUID="bbcbf9e3-2774-4872-95d5-4f27b7ad17cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 18:15:02.735033 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.734997    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"]
Nov 03 18:15:02.749997 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.749964    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"payload-processing-external-model-e2e--58d254fa-dockercfg-vmp26\""
Nov 03 18:15:02.752005 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.751814    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"kube-root-ca.crt\""
Nov 03 18:15:02.752005 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.751955    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-029ca0\""
Nov 03 18:15:02.752749 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.752727    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"routing-overlay\""
Nov 03 18:15:02.756763 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.756651    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-748d54b9b-xfd9p"]
Nov 03 18:15:02.787790 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.787754    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43e8a2ca-2687-475e-8b25-0d04734a5314-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm\" (UID: \"43e8a2ca-2687-475e-8b25-0d04734a5314\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"
Nov 03 18:15:02.787960 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.787817    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e8a2ca-2687-475e-8b25-0d04734a5314-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm\" (UID: \"43e8a2ca-2687-475e-8b25-0d04734a5314\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"
Nov 03 18:15:02.787960 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.787872    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvmd5\" (UniqueName: \"kubernetes.io/projected/43e8a2ca-2687-475e-8b25-0d04734a5314-kube-api-access-zvmd5\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm\" (UID: \"43e8a2ca-2687-475e-8b25-0d04734a5314\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"
Nov 03 18:15:02.787960 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.787939    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43e8a2ca-2687-475e-8b25-0d04734a5314-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm\" (UID: \"43e8a2ca-2687-475e-8b25-0d04734a5314\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"
Nov 03 18:15:02.788564 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.788544    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e8a2ca-2687-475e-8b25-0d04734a5314-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm\" (UID: \"43e8a2ca-2687-475e-8b25-0d04734a5314\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"
Nov 03 18:15:02.788630 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.788552    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43e8a2ca-2687-475e-8b25-0d04734a5314-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm\" (UID: \"43e8a2ca-2687-475e-8b25-0d04734a5314\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"
Nov 03 18:15:02.790548 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.790531    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43e8a2ca-2687-475e-8b25-0d04734a5314-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm\" (UID: \"43e8a2ca-2687-475e-8b25-0d04734a5314\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"
Nov 03 18:15:02.857648 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.857570    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvmd5\" (UniqueName: \"kubernetes.io/projected/43e8a2ca-2687-475e-8b25-0d04734a5314-kube-api-access-zvmd5\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm\" (UID: \"43e8a2ca-2687-475e-8b25-0d04734a5314\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"
Nov 03 18:15:02.966753 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:02.966717    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"
Nov 03 18:15:03.358757 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:03.358719    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a7a8d36-2439-48b1-987f-f0fe02c4ade4" path="/var/lib/kubelet/pods/2a7a8d36-2439-48b1-987f-f0fe02c4ade4/volumes"
Nov 03 18:15:03.380774 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:15:03.380730    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e8a2ca_2687_475e_8b25_0d04734a5314.slice/crio-4328fb3ae3a98dd2dcb6efc738e3536d684eed462aed83d7dc72d015c66cadb6 WatchSource:0}: Error finding container 4328fb3ae3a98dd2dcb6efc738e3536d684eed462aed83d7dc72d015c66cadb6: Status 404 returned error can't find the container with id 4328fb3ae3a98dd2dcb6efc738e3536d684eed462aed83d7dc72d015c66cadb6
Nov 03 18:15:03.383450 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:03.383425    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"]
Nov 03 18:15:03.418478 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:03.418447    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:03.580005 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:03.579966    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm" event={"ID":"43e8a2ca-2687-475e-8b25-0d04734a5314","Type":"ContainerStarted","Data":"d1eaf49764981d51e7aad4d9f2cce86f864af212155a7d060143644e04e436f2"}
Nov 03 18:15:03.580312 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:03.580294    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm" event={"ID":"43e8a2ca-2687-475e-8b25-0d04734a5314","Type":"ContainerStarted","Data":"4328fb3ae3a98dd2dcb6efc738e3536d684eed462aed83d7dc72d015c66cadb6"}
Nov 03 18:15:03.580481 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:03.580468    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"
Nov 03 18:15:03.582840 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:03.582814    6059 generic.go:356] "Generic (PLEG): container finished" podID="b1cd9469-3e37-4b68-9cd5-9936f7dcff6d" containerID="b33e93ba4aadc1aa3f86219745193286e4408ac02a536861e092ddbf765ddb53" exitCode=60
Nov 03 18:15:03.582961 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:03.582898    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd" event={"ID":"b1cd9469-3e37-4b68-9cd5-9936f7dcff6d","Type":"ContainerDied","Data":"b33e93ba4aadc1aa3f86219745193286e4408ac02a536861e092ddbf765ddb53"}
Nov 03 18:15:03.582961 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:03.582955    6059 scope.go:122] "RemoveContainer" containerID="012b183c9bc1c4926ec69c2de9eaabd9ba5510738b96bd2c152c522efa4825de"
Nov 03 18:15:03.584816 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:03.584795    6059 scope.go:122] "RemoveContainer" containerID="b33e93ba4aadc1aa3f86219745193286e4408ac02a536861e092ddbf765ddb53"
Nov 03 18:15:03.585022 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:03.585000    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs" event={"ID":"38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7","Type":"ContainerStarted","Data":"f19c732dc8c4d72fb1254b24b623b4be62bf111a0d7e4c8cefe54cb24b0a6ac7"}
Nov 03 18:15:03.585092 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:15:03.585048    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd_odh-ai-gateway-infra(b1cd9469-3e37-4b68-9cd5-9936f7dcff6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd" podUID="b1cd9469-3e37-4b68-9cd5-9936f7dcff6d"
Nov 03 18:15:03.702299 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:03.702215    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8kglw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 18:15:03.702299 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:03.702275    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw" podUID="bbcbf9e3-2774-4872-95d5-4f27b7ad17cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 18:15:03.723018 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:03.722961    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm" podStartSLOduration=1.7229475829999998 podStartE2EDuration="1.722947583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:03.72055549 +0000 UTC m=+3444.518268272" watchObservedRunningTime="2026-11-03 18:15:03.722947583 +0000 UTC m=+3444.520660363"
Nov 03 18:15:04.592855 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:04.592822    6059 scope.go:122] "RemoveContainer" containerID="b33e93ba4aadc1aa3f86219745193286e4408ac02a536861e092ddbf765ddb53"
Nov 03 18:15:04.593132 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:15:04.593111    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd_odh-ai-gateway-infra(b1cd9469-3e37-4b68-9cd5-9936f7dcff6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd" podUID="b1cd9469-3e37-4b68-9cd5-9936f7dcff6d"
Nov 03 18:15:04.594525 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:04.594502    6059 generic.go:356] "Generic (PLEG): container finished" podID="38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7" containerID="f19c732dc8c4d72fb1254b24b623b4be62bf111a0d7e4c8cefe54cb24b0a6ac7" exitCode=60
Nov 03 18:15:04.594638 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:04.594589    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs" event={"ID":"38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7","Type":"ContainerDied","Data":"f19c732dc8c4d72fb1254b24b623b4be62bf111a0d7e4c8cefe54cb24b0a6ac7"}
Nov 03 18:15:04.594682 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:04.594648    6059 scope.go:122] "RemoveContainer" containerID="275345e4675e0b9be1d85acd95983b6ea81ee2850845c261709f725b711db1db"
Nov 03 18:15:04.594805 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:04.594781    6059 scope.go:122] "RemoveContainer" containerID="f19c732dc8c4d72fb1254b24b623b4be62bf111a0d7e4c8cefe54cb24b0a6ac7"
Nov 03 18:15:04.595064 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:15:04.595043    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895495-v5mvs_odh-ai-gateway-infra(38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs" podUID="38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7"
Nov 03 18:15:04.702574 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:04.702534    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8kglw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 18:15:04.702756 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:04.702595    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw" podUID="bbcbf9e3-2774-4872-95d5-4f27b7ad17cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 18:15:05.604404 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:05.604348    6059 scope.go:122] "RemoveContainer" containerID="f19c732dc8c4d72fb1254b24b623b4be62bf111a0d7e4c8cefe54cb24b0a6ac7"
Nov 03 18:15:05.604906 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:15:05.604693    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895495-v5mvs_odh-ai-gateway-infra(38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs" podUID="38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7"
Nov 03 18:15:05.701659 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:05.701622    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8kglw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 18:15:05.701846 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:05.701690    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw" podUID="bbcbf9e3-2774-4872-95d5-4f27b7ad17cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 18:15:06.702338 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:06.702293    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8kglw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 18:15:06.702798 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:06.702390    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw" podUID="bbcbf9e3-2774-4872-95d5-4f27b7ad17cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 18:15:06.881694 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:15:06.881655    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:15:07.702535 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:07.702486    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8kglw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 18:15:07.703076 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:07.702546    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw" podUID="bbcbf9e3-2774-4872-95d5-4f27b7ad17cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 18:15:08.452107 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:08.452069    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:08.701892 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:08.701841    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8kglw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 18:15:08.702145 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:08.701896    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw" podUID="bbcbf9e3-2774-4872-95d5-4f27b7ad17cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 18:15:10.536671 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:10.536627    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:10.537180 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:10.536949    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:10.537845 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:10.537826    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:15:13.339597 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:13.339566    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:14.328661 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:14.328628    6059 scope.go:122] "RemoveContainer" containerID="b33e93ba4aadc1aa3f86219745193286e4408ac02a536861e092ddbf765ddb53"
Nov 03 18:15:14.658690 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:14.658658    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd" event={"ID":"b1cd9469-3e37-4b68-9cd5-9936f7dcff6d","Type":"ContainerStarted","Data":"2cddd85fca1866058ff3eea9a8e439a5b45500806e4e3c32b53c97130c29487f"}
Nov 03 18:15:15.355939 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.355896    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"]
Nov 03 18:15:15.360465 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.360439    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.453532 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.453442    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.453721 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.453602    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.453857 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.453824    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565e1871-9784-4483-ad19-17413b47017e-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.454074 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.454053    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565e1871-9784-4483-ad19-17413b47017e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.454183 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.454161    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565e1871-9784-4483-ad19-17413b47017e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.454262 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.454244    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8tpr\" (UniqueName: \"kubernetes.io/projected/565e1871-9784-4483-ad19-17413b47017e-kube-api-access-q8tpr\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.454329 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.454312    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.454416 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.454345    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.454614 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.454592    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.469679 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.469638    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"]
Nov 03 18:15:15.556229 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.556183    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.556229 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.556232    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565e1871-9784-4483-ad19-17413b47017e-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.556536 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.556376    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565e1871-9784-4483-ad19-17413b47017e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.556536 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.556430    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565e1871-9784-4483-ad19-17413b47017e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.556536 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.556463    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8tpr\" (UniqueName: \"kubernetes.io/projected/565e1871-9784-4483-ad19-17413b47017e-kube-api-access-q8tpr\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.556536 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.556488    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.556536 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.556517    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.556823 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.556690    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.556823 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.556743    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.556972 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.556942    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.557142 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.557116    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.557225 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.557140    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.557225 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.557172    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565e1871-9784-4483-ad19-17413b47017e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.557225 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.557195    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.558608 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.558586    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.559023 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.559001    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565e1871-9784-4483-ad19-17413b47017e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.607971 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.607939    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"
Nov 03 18:15:15.619437 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.619403    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565e1871-9784-4483-ad19-17413b47017e-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.619597 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.619522    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8tpr\" (UniqueName: \"kubernetes.io/projected/565e1871-9784-4483-ad19-17413b47017e-kube-api-access-q8tpr\") pod \"maas-default-gateway-openshift-default-58c6585447-2qd5j\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.666757 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.666724    6059 generic.go:356] "Generic (PLEG): container finished" podID="b1cd9469-3e37-4b68-9cd5-9936f7dcff6d" containerID="2cddd85fca1866058ff3eea9a8e439a5b45500806e4e3c32b53c97130c29487f" exitCode=60
Nov 03 18:15:15.667156 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.666798    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd" event={"ID":"b1cd9469-3e37-4b68-9cd5-9936f7dcff6d","Type":"ContainerDied","Data":"2cddd85fca1866058ff3eea9a8e439a5b45500806e4e3c32b53c97130c29487f"}
Nov 03 18:15:15.667156 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.666865    6059 scope.go:122] "RemoveContainer" containerID="b33e93ba4aadc1aa3f86219745193286e4408ac02a536861e092ddbf765ddb53"
Nov 03 18:15:15.667251 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.667201    6059 scope.go:122] "RemoveContainer" containerID="2cddd85fca1866058ff3eea9a8e439a5b45500806e4e3c32b53c97130c29487f"
Nov 03 18:15:15.667512 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:15:15.667490    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd_odh-ai-gateway-infra(b1cd9469-3e37-4b68-9cd5-9936f7dcff6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd" podUID="b1cd9469-3e37-4b68-9cd5-9936f7dcff6d"
Nov 03 18:15:15.674100 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.674073    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:15.792841 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:15.792807    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd"]
Nov 03 18:15:16.102437 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:15:16.102402    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565e1871_9784_4483_ad19_17413b47017e.slice/crio-ef6df904df8df53fba1b1c4d85f2d99b4c6fdda6309de3a918ef6180749c09a0 WatchSource:0}: Error finding container ef6df904df8df53fba1b1c4d85f2d99b4c6fdda6309de3a918ef6180749c09a0: Status 404 returned error can't find the container with id ef6df904df8df53fba1b1c4d85f2d99b4c6fdda6309de3a918ef6180749c09a0
Nov 03 18:15:16.104808 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:16.104784    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"]
Nov 03 18:15:16.104929 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:16.104900    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:15:16.104975 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:16.104963    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:15:16.105021 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:16.104998    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:15:16.676915 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:16.676876    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j" event={"ID":"565e1871-9784-4483-ad19-17413b47017e","Type":"ContainerStarted","Data":"07beb1a50ed37b27bf44b14902c2205eeedce86545d9e24ae8ac98bbf768bb49"}
Nov 03 18:15:16.677290 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:16.676957    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j" event={"ID":"565e1871-9784-4483-ad19-17413b47017e","Type":"ContainerStarted","Data":"ef6df904df8df53fba1b1c4d85f2d99b4c6fdda6309de3a918ef6180749c09a0"}
Nov 03 18:15:16.869477 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:16.869451    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd"
Nov 03 18:15:16.922677 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:16.922614    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j" podStartSLOduration=1.922597459 podStartE2EDuration="1.922597459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:16.920781486 +0000 UTC m=+3457.718494269" watchObservedRunningTime="2026-11-03 18:15:16.922597459 +0000 UTC m=+3457.720310240"
Nov 03 18:15:16.973722 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:16.973686    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1cd9469-3e37-4b68-9cd5-9936f7dcff6d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1cd9469-3e37-4b68-9cd5-9936f7dcff6d-maas-api-service-ca\") pod \"b1cd9469-3e37-4b68-9cd5-9936f7dcff6d\" (UID: \"b1cd9469-3e37-4b68-9cd5-9936f7dcff6d\") "
Nov 03 18:15:16.973952 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:16.973795    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1cd9469-3e37-4b68-9cd5-9936f7dcff6d-kube-api-access-b67kb\" (UniqueName: \"kubernetes.io/projected/b1cd9469-3e37-4b68-9cd5-9936f7dcff6d-kube-api-access-b67kb\") pod \"b1cd9469-3e37-4b68-9cd5-9936f7dcff6d\" (UID: \"b1cd9469-3e37-4b68-9cd5-9936f7dcff6d\") "
Nov 03 18:15:16.974132 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:16.974108    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1cd9469-3e37-4b68-9cd5-9936f7dcff6d-maas-api-service-ca" pod "b1cd9469-3e37-4b68-9cd5-9936f7dcff6d" (UID: "b1cd9469-3e37-4b68-9cd5-9936f7dcff6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:16.974220 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:16.974202    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1cd9469-3e37-4b68-9cd5-9936f7dcff6d-maas-api-service-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:16.976121 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:16.976095    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1cd9469-3e37-4b68-9cd5-9936f7dcff6d-kube-api-access-b67kb" pod "b1cd9469-3e37-4b68-9cd5-9936f7dcff6d" (UID: "b1cd9469-3e37-4b68-9cd5-9936f7dcff6d"). InnerVolumeSpecName "kube-api-access-b67kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:17.075124 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:17.075075    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b67kb\" (UniqueName: \"kubernetes.io/projected/b1cd9469-3e37-4b68-9cd5-9936f7dcff6d-kube-api-access-b67kb\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:17.328924 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:17.328842    6059 scope.go:122] "RemoveContainer" containerID="f19c732dc8c4d72fb1254b24b623b4be62bf111a0d7e4c8cefe54cb24b0a6ac7"
Nov 03 18:15:17.674776 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:17.674736    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:17.676647 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:17.676620    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2qd5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 18:15:17.676782 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:17.676671    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j" podUID="565e1871-9784-4483-ad19-17413b47017e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 18:15:17.683746 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:17.683712    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd"
Nov 03 18:15:17.684198 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:17.683753    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd" event={"ID":"b1cd9469-3e37-4b68-9cd5-9936f7dcff6d","Type":"ContainerDied","Data":"4a6249ae530d7d7b506a58bd1a83c9817a032c55cd944116cfa51553adcef327"}
Nov 03 18:15:17.684198 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:17.683814    6059 scope.go:122] "RemoveContainer" containerID="2cddd85fca1866058ff3eea9a8e439a5b45500806e4e3c32b53c97130c29487f"
Nov 03 18:15:17.878656 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:17.878598    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd"]
Nov 03 18:15:17.881682 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:17.881650    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895495-jd9hd"]
Nov 03 18:15:18.155322 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:18.155280    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:18.674502 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:18.674462    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2qd5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 18:15:18.674695 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:18.674520    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j" podUID="565e1871-9784-4483-ad19-17413b47017e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 18:15:18.692707 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:18.692672    6059 generic.go:356] "Generic (PLEG): container finished" podID="38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7" containerID="1b8a7ce1ad763cca666d7b340b3ffe2b17b709e16c7960c140d25824ec5610d3" exitCode=60
Nov 03 18:15:18.693200 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:18.692750    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs" event={"ID":"38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7","Type":"ContainerDied","Data":"1b8a7ce1ad763cca666d7b340b3ffe2b17b709e16c7960c140d25824ec5610d3"}
Nov 03 18:15:18.693200 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:18.692873    6059 scope.go:122] "RemoveContainer" containerID="f19c732dc8c4d72fb1254b24b623b4be62bf111a0d7e4c8cefe54cb24b0a6ac7"
Nov 03 18:15:18.693343 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:18.693328    6059 scope.go:122] "RemoveContainer" containerID="1b8a7ce1ad763cca666d7b340b3ffe2b17b709e16c7960c140d25824ec5610d3"
Nov 03 18:15:18.693619 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:15:18.693596    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895495-v5mvs_odh-ai-gateway-infra(38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs" podUID="38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7"
Nov 03 18:15:19.359084 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:19.359047    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1cd9469-3e37-4b68-9cd5-9936f7dcff6d" path="/var/lib/kubelet/pods/b1cd9469-3e37-4b68-9cd5-9936f7dcff6d/volumes"
Nov 03 18:15:19.675591 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:19.675497    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2qd5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 18:15:19.675591 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:19.675554    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j" podUID="565e1871-9784-4483-ad19-17413b47017e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 18:15:19.840382 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:19.829385    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs"]
Nov 03 18:15:20.016655 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:20.016628    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs"
Nov 03 18:15:20.111133 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:20.111093    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7-maas-api-service-ca\") pod \"38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7\" (UID: \"38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7\") "
Nov 03 18:15:20.111328 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:20.111268    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7-kube-api-access-l5z9d\" (UniqueName: \"kubernetes.io/projected/38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7-kube-api-access-l5z9d\") pod \"38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7\" (UID: \"38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7\") "
Nov 03 18:15:20.111470 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:20.111445    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7-maas-api-service-ca" pod "38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7" (UID: "38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:20.111601 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:20.111584    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7-maas-api-service-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:20.113492 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:20.113466    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7-kube-api-access-l5z9d" pod "38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7" (UID: "38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7"). InnerVolumeSpecName "kube-api-access-l5z9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:20.212336 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:20.212303    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5z9d\" (UniqueName: \"kubernetes.io/projected/38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7-kube-api-access-l5z9d\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:20.674583 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:20.674549    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2qd5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 18:15:20.674781 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:20.674613    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j" podUID="565e1871-9784-4483-ad19-17413b47017e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 18:15:20.707330 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:20.707295    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs" event={"ID":"38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7","Type":"ContainerDied","Data":"1f129d39368f5bc808e842025018367b3b15a352f546f62d384c742829d49fe5"}
Nov 03 18:15:20.707330 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:20.707330    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs"
Nov 03 18:15:20.707619 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:20.707376    6059 scope.go:122] "RemoveContainer" containerID="1b8a7ce1ad763cca666d7b340b3ffe2b17b709e16c7960c140d25824ec5610d3"
Nov 03 18:15:20.891689 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:20.891643    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs"]
Nov 03 18:15:20.895929 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:20.895890    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895495-v5mvs"]
Nov 03 18:15:21.359230 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:21.359194    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7" path="/var/lib/kubelet/pods/38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7/volumes"
Nov 03 18:15:21.675213 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:21.675120    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2qd5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 18:15:21.675213 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:21.675178    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j" podUID="565e1871-9784-4483-ad19-17413b47017e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 18:15:22.675474 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:22.675427    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2qd5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 18:15:22.675891 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:22.675516    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j" podUID="565e1871-9784-4483-ad19-17413b47017e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 18:15:22.971784 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:22.971749    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:23.674801 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:23.674756    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-2qd5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 18:15:23.675012 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:23.674830    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j" podUID="565e1871-9784-4483-ad19-17413b47017e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 18:15:25.511457 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:25.511426    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:25.511899 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:25.511794    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:25.512992 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:25.512969    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:15:27.764165 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:27.764124    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:28.054297 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:28.054206    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:28.236203 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:28.236172    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:31.835969 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:31.835926    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:32.329425 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.329385    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"]
Nov 03 18:15:32.329849 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.329836    6059 cpu_manager.go:414] "containerMap: removing container" podUID="38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7" containerName="cleanup"
Nov 03 18:15:32.329898 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.329850    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7" containerName="cleanup"
Nov 03 18:15:32.329898 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.329861    6059 cpu_manager.go:414] "containerMap: removing container" podUID="38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7" containerName="cleanup"
Nov 03 18:15:32.329898 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.329866    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7" containerName="cleanup"
Nov 03 18:15:32.329898 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.329893    6059 cpu_manager.go:414] "containerMap: removing container" podUID="b1cd9469-3e37-4b68-9cd5-9936f7dcff6d" containerName="cleanup"
Nov 03 18:15:32.329898 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.329899    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1cd9469-3e37-4b68-9cd5-9936f7dcff6d" containerName="cleanup"
Nov 03 18:15:32.330059 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.329909    6059 cpu_manager.go:414] "containerMap: removing container" podUID="b1cd9469-3e37-4b68-9cd5-9936f7dcff6d" containerName="cleanup"
Nov 03 18:15:32.330059 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.329914    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1cd9469-3e37-4b68-9cd5-9936f7dcff6d" containerName="cleanup"
Nov 03 18:15:32.330059 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.329923    6059 cpu_manager.go:414] "containerMap: removing container" podUID="b1cd9469-3e37-4b68-9cd5-9936f7dcff6d" containerName="cleanup"
Nov 03 18:15:32.330059 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.329927    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1cd9469-3e37-4b68-9cd5-9936f7dcff6d" containerName="cleanup"
Nov 03 18:15:32.330059 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.330004    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1cd9469-3e37-4b68-9cd5-9936f7dcff6d" containerName="cleanup"
Nov 03 18:15:32.330059 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.330015    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1cd9469-3e37-4b68-9cd5-9936f7dcff6d" containerName="cleanup"
Nov 03 18:15:32.330059 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.330024    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1cd9469-3e37-4b68-9cd5-9936f7dcff6d" containerName="cleanup"
Nov 03 18:15:32.330059 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.330036    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7" containerName="cleanup"
Nov 03 18:15:32.330059 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.330041    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7" containerName="cleanup"
Nov 03 18:15:32.330059 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.330050    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7" containerName="cleanup"
Nov 03 18:15:32.334686 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.334666    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"
Nov 03 18:15:32.380399 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.380343    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8ljs\" (UniqueName: \"kubernetes.io/projected/25657ef4-b7fb-4fe6-848c-24dd44c22d62-kube-api-access-t8ljs\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8\" (UID: \"25657ef4-b7fb-4fe6-848c-24dd44c22d62\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"
Nov 03 18:15:32.380605 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.380488    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25657ef4-b7fb-4fe6-848c-24dd44c22d62-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8\" (UID: \"25657ef4-b7fb-4fe6-848c-24dd44c22d62\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"
Nov 03 18:15:32.380605 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.380556    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25657ef4-b7fb-4fe6-848c-24dd44c22d62-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8\" (UID: \"25657ef4-b7fb-4fe6-848c-24dd44c22d62\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"
Nov 03 18:15:32.380697 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.380634    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25657ef4-b7fb-4fe6-848c-24dd44c22d62-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8\" (UID: \"25657ef4-b7fb-4fe6-848c-24dd44c22d62\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"
Nov 03 18:15:32.442090 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.442056    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"kube-root-ca.crt\""
Nov 03 18:15:32.442308 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.442056    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-310ab5d0c87c\""
Nov 03 18:15:32.442308 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.442069    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"routing-overlay\""
Nov 03 18:15:32.442308 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.442132    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"openshift-service-ca.crt\""
Nov 03 18:15:32.442308 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.442075    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-8c9b0f\""
Nov 03 18:15:32.442308 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.442178    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"payload-processing-external-model-e2e--b603cb9e-dockercfg-drjz5\""
Nov 03 18:15:32.444555 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.444535    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"]
Nov 03 18:15:32.481756 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.481718    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25657ef4-b7fb-4fe6-848c-24dd44c22d62-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8\" (UID: \"25657ef4-b7fb-4fe6-848c-24dd44c22d62\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"
Nov 03 18:15:32.481756 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.481760    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25657ef4-b7fb-4fe6-848c-24dd44c22d62-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8\" (UID: \"25657ef4-b7fb-4fe6-848c-24dd44c22d62\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"
Nov 03 18:15:32.482044 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.481896    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25657ef4-b7fb-4fe6-848c-24dd44c22d62-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8\" (UID: \"25657ef4-b7fb-4fe6-848c-24dd44c22d62\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"
Nov 03 18:15:32.482044 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.482033    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8ljs\" (UniqueName: \"kubernetes.io/projected/25657ef4-b7fb-4fe6-848c-24dd44c22d62-kube-api-access-t8ljs\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8\" (UID: \"25657ef4-b7fb-4fe6-848c-24dd44c22d62\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"
Nov 03 18:15:32.482551 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.482518    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25657ef4-b7fb-4fe6-848c-24dd44c22d62-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8\" (UID: \"25657ef4-b7fb-4fe6-848c-24dd44c22d62\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"
Nov 03 18:15:32.482551 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.482545    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25657ef4-b7fb-4fe6-848c-24dd44c22d62-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8\" (UID: \"25657ef4-b7fb-4fe6-848c-24dd44c22d62\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"
Nov 03 18:15:32.484186 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.484162    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25657ef4-b7fb-4fe6-848c-24dd44c22d62-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8\" (UID: \"25657ef4-b7fb-4fe6-848c-24dd44c22d62\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"
Nov 03 18:15:32.543625 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.543590    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:32.547053 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.547026    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8ljs\" (UniqueName: \"kubernetes.io/projected/25657ef4-b7fb-4fe6-848c-24dd44c22d62-kube-api-access-t8ljs\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8\" (UID: \"25657ef4-b7fb-4fe6-848c-24dd44c22d62\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"
Nov 03 18:15:32.648387 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:32.648250    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"
Nov 03 18:15:33.081988 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:15:33.081948    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25657ef4_b7fb_4fe6_848c_24dd44c22d62.slice/crio-311ae539c5c7a725d38013fff4a0825d3eaf5cb61b67ddc470f4fcd2e07d3cfc WatchSource:0}: Error finding container 311ae539c5c7a725d38013fff4a0825d3eaf5cb61b67ddc470f4fcd2e07d3cfc: Status 404 returned error can't find the container with id 311ae539c5c7a725d38013fff4a0825d3eaf5cb61b67ddc470f4fcd2e07d3cfc
Nov 03 18:15:33.084995 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:33.084968    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"]
Nov 03 18:15:33.791426 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:33.791386    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8" event={"ID":"25657ef4-b7fb-4fe6-848c-24dd44c22d62","Type":"ContainerStarted","Data":"077d2e73bc11bc84485e47957030f5943498e7764d13d792f91131d577593e4f"}
Nov 03 18:15:33.791615 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:33.791461    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"
Nov 03 18:15:33.791615 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:33.791486    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8" event={"ID":"25657ef4-b7fb-4fe6-848c-24dd44c22d62","Type":"ContainerStarted","Data":"311ae539c5c7a725d38013fff4a0825d3eaf5cb61b67ddc470f4fcd2e07d3cfc"}
Nov 03 18:15:33.913973 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:33.913921    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8" podStartSLOduration=1.913906607 podStartE2EDuration="1.913906607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:33.91120308 +0000 UTC m=+3474.708915860" watchObservedRunningTime="2026-11-03 18:15:33.913906607 +0000 UTC m=+3474.711619387"
Nov 03 18:15:36.861475 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:15:36.861437    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:15:37.376500 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:37.376459    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:42.150171 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:42.150136    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:44.802089 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:44.802062    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"
Nov 03 18:15:46.966118 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:46.966079    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:50.249863 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:50.249830    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:50.435805 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:50.435769    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:51.780598 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:51.780557    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:15:56.659066 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:15:56.659029    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:01.457469 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:01.457432    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:04.766840 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:04.766796    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"]
Nov 03 18:16:04.809275 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:04.809243    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:04.992218 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:04.990937    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:06.355193 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:06.355130    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:06.861782 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:16:06.861743    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:16:09.768208 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:09.768158    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8" podUID="25657ef4-b7fb-4fe6-848c-24dd44c22d62" containerName="payload-processing" containerID="cri-o://077d2e73bc11bc84485e47957030f5943498e7764d13d792f91131d577593e4f" gracePeriod=25
Nov 03 18:16:10.027402 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:10.027331    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"
Nov 03 18:16:10.034123 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:10.034083    6059 generic.go:356] "Generic (PLEG): container finished" podID="25657ef4-b7fb-4fe6-848c-24dd44c22d62" containerID="077d2e73bc11bc84485e47957030f5943498e7764d13d792f91131d577593e4f" exitCode=0
Nov 03 18:16:10.034266 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:10.034126    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"
Nov 03 18:16:10.034266 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:10.034137    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8" event={"ID":"25657ef4-b7fb-4fe6-848c-24dd44c22d62","Type":"ContainerDied","Data":"077d2e73bc11bc84485e47957030f5943498e7764d13d792f91131d577593e4f"}
Nov 03 18:16:10.034266 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:10.034237    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8" event={"ID":"25657ef4-b7fb-4fe6-848c-24dd44c22d62","Type":"ContainerDied","Data":"311ae539c5c7a725d38013fff4a0825d3eaf5cb61b67ddc470f4fcd2e07d3cfc"}
Nov 03 18:16:10.034392 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:10.034266    6059 scope.go:122] "RemoveContainer" containerID="077d2e73bc11bc84485e47957030f5943498e7764d13d792f91131d577593e4f"
Nov 03 18:16:10.044201 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:10.044179    6059 scope.go:122] "RemoveContainer" containerID="077d2e73bc11bc84485e47957030f5943498e7764d13d792f91131d577593e4f"
Nov 03 18:16:10.044570 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:16:10.044548    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"077d2e73bc11bc84485e47957030f5943498e7764d13d792f91131d577593e4f\": container with ID starting with 077d2e73bc11bc84485e47957030f5943498e7764d13d792f91131d577593e4f not found: ID does not exist" containerID="077d2e73bc11bc84485e47957030f5943498e7764d13d792f91131d577593e4f"
Nov 03 18:16:10.044683 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:10.044577    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"077d2e73bc11bc84485e47957030f5943498e7764d13d792f91131d577593e4f"} err="failed to get container status \"077d2e73bc11bc84485e47957030f5943498e7764d13d792f91131d577593e4f\": rpc error: code = NotFound desc = could not find container \"077d2e73bc11bc84485e47957030f5943498e7764d13d792f91131d577593e4f\": container with ID starting with 077d2e73bc11bc84485e47957030f5943498e7764d13d792f91131d577593e4f not found: ID does not exist"
Nov 03 18:16:10.081345 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:10.081314    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25657ef4-b7fb-4fe6-848c-24dd44c22d62-provider-credentials\" (UniqueName: \"kubernetes.io/projected/25657ef4-b7fb-4fe6-848c-24dd44c22d62-provider-credentials\") pod \"25657ef4-b7fb-4fe6-848c-24dd44c22d62\" (UID: \"25657ef4-b7fb-4fe6-848c-24dd44c22d62\") "
Nov 03 18:16:10.081530 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:10.081376    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25657ef4-b7fb-4fe6-848c-24dd44c22d62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25657ef4-b7fb-4fe6-848c-24dd44c22d62-plugins-config-volume\") pod \"25657ef4-b7fb-4fe6-848c-24dd44c22d62\" (UID: \"25657ef4-b7fb-4fe6-848c-24dd44c22d62\") "
Nov 03 18:16:10.081530 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:10.081458    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25657ef4-b7fb-4fe6-848c-24dd44c22d62-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25657ef4-b7fb-4fe6-848c-24dd44c22d62-routing-overlay-volume\") pod \"25657ef4-b7fb-4fe6-848c-24dd44c22d62\" (UID: \"25657ef4-b7fb-4fe6-848c-24dd44c22d62\") "
Nov 03 18:16:10.081681 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:10.081661    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25657ef4-b7fb-4fe6-848c-24dd44c22d62-kube-api-access-t8ljs\" (UniqueName: \"kubernetes.io/projected/25657ef4-b7fb-4fe6-848c-24dd44c22d62-kube-api-access-t8ljs\") pod \"25657ef4-b7fb-4fe6-848c-24dd44c22d62\" (UID: \"25657ef4-b7fb-4fe6-848c-24dd44c22d62\") "
Nov 03 18:16:10.081823 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:10.081796    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25657ef4-b7fb-4fe6-848c-24dd44c22d62-routing-overlay-volume" pod "25657ef4-b7fb-4fe6-848c-24dd44c22d62" (UID: "25657ef4-b7fb-4fe6-848c-24dd44c22d62"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:10.081885 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:10.081807    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25657ef4-b7fb-4fe6-848c-24dd44c22d62-plugins-config-volume" pod "25657ef4-b7fb-4fe6-848c-24dd44c22d62" (UID: "25657ef4-b7fb-4fe6-848c-24dd44c22d62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:10.082013 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:10.081995    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25657ef4-b7fb-4fe6-848c-24dd44c22d62-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:10.082096 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:10.082015    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25657ef4-b7fb-4fe6-848c-24dd44c22d62-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:10.083658 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:10.083620    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25657ef4-b7fb-4fe6-848c-24dd44c22d62-provider-credentials" pod "25657ef4-b7fb-4fe6-848c-24dd44c22d62" (UID: "25657ef4-b7fb-4fe6-848c-24dd44c22d62"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:10.083786 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:10.083699    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25657ef4-b7fb-4fe6-848c-24dd44c22d62-kube-api-access-t8ljs" pod "25657ef4-b7fb-4fe6-848c-24dd44c22d62" (UID: "25657ef4-b7fb-4fe6-848c-24dd44c22d62"). InnerVolumeSpecName "kube-api-access-t8ljs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:10.183280 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:10.183244    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8ljs\" (UniqueName: \"kubernetes.io/projected/25657ef4-b7fb-4fe6-848c-24dd44c22d62-kube-api-access-t8ljs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:10.183280 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:10.183274    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25657ef4-b7fb-4fe6-848c-24dd44c22d62-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:10.458805 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:10.458772    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"]
Nov 03 18:16:10.463419 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:10.463390    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-85c8fg5s8"]
Nov 03 18:16:11.164668 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:11.164632    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:11.335972 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:11.335940    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25657ef4-b7fb-4fe6-848c-24dd44c22d62" path="/var/lib/kubelet/pods/25657ef4-b7fb-4fe6-848c-24dd44c22d62/volumes"
Nov 03 18:16:15.944958 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:15.944932    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:17.891516 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:17.891443    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"]
Nov 03 18:16:17.901227 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:17.901195    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:18.077889 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:18.077860    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:18.624455 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:18.624419    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:20.866702 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:20.866674    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:22.892498 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:22.892451    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm" podUID="43e8a2ca-2687-475e-8b25-0d04734a5314" containerName="payload-processing" containerID="cri-o://d1eaf49764981d51e7aad4d9f2cce86f864af212155a7d060143644e04e436f2" gracePeriod=25
Nov 03 18:16:23.114917 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:23.114882    6059 generic.go:356] "Generic (PLEG): container finished" podID="43e8a2ca-2687-475e-8b25-0d04734a5314" containerID="d1eaf49764981d51e7aad4d9f2cce86f864af212155a7d060143644e04e436f2" exitCode=0
Nov 03 18:16:23.115071 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:23.114955    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm" event={"ID":"43e8a2ca-2687-475e-8b25-0d04734a5314","Type":"ContainerDied","Data":"d1eaf49764981d51e7aad4d9f2cce86f864af212155a7d060143644e04e436f2"}
Nov 03 18:16:23.155209 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:23.155145    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"
Nov 03 18:16:23.226822 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:23.226793    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43e8a2ca-2687-475e-8b25-0d04734a5314-kube-api-access-zvmd5\" (UniqueName: \"kubernetes.io/projected/43e8a2ca-2687-475e-8b25-0d04734a5314-kube-api-access-zvmd5\") pod \"43e8a2ca-2687-475e-8b25-0d04734a5314\" (UID: \"43e8a2ca-2687-475e-8b25-0d04734a5314\") "
Nov 03 18:16:23.227003 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:23.226877    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43e8a2ca-2687-475e-8b25-0d04734a5314-provider-credentials\" (UniqueName: \"kubernetes.io/projected/43e8a2ca-2687-475e-8b25-0d04734a5314-provider-credentials\") pod \"43e8a2ca-2687-475e-8b25-0d04734a5314\" (UID: \"43e8a2ca-2687-475e-8b25-0d04734a5314\") "
Nov 03 18:16:23.227003 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:23.226983    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43e8a2ca-2687-475e-8b25-0d04734a5314-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43e8a2ca-2687-475e-8b25-0d04734a5314-routing-overlay-volume\") pod \"43e8a2ca-2687-475e-8b25-0d04734a5314\" (UID: \"43e8a2ca-2687-475e-8b25-0d04734a5314\") "
Nov 03 18:16:23.227190 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:23.227171    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43e8a2ca-2687-475e-8b25-0d04734a5314-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e8a2ca-2687-475e-8b25-0d04734a5314-plugins-config-volume\") pod \"43e8a2ca-2687-475e-8b25-0d04734a5314\" (UID: \"43e8a2ca-2687-475e-8b25-0d04734a5314\") "
Nov 03 18:16:23.227482 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:23.227345    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e8a2ca-2687-475e-8b25-0d04734a5314-routing-overlay-volume" pod "43e8a2ca-2687-475e-8b25-0d04734a5314" (UID: "43e8a2ca-2687-475e-8b25-0d04734a5314"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:23.227634 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:23.227574    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e8a2ca-2687-475e-8b25-0d04734a5314-plugins-config-volume" pod "43e8a2ca-2687-475e-8b25-0d04734a5314" (UID: "43e8a2ca-2687-475e-8b25-0d04734a5314"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:23.227712 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:23.227649    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43e8a2ca-2687-475e-8b25-0d04734a5314-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:23.229018 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:23.228996    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e8a2ca-2687-475e-8b25-0d04734a5314-provider-credentials" pod "43e8a2ca-2687-475e-8b25-0d04734a5314" (UID: "43e8a2ca-2687-475e-8b25-0d04734a5314"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:23.229285 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:23.229271    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e8a2ca-2687-475e-8b25-0d04734a5314-kube-api-access-zvmd5" pod "43e8a2ca-2687-475e-8b25-0d04734a5314" (UID: "43e8a2ca-2687-475e-8b25-0d04734a5314"). InnerVolumeSpecName "kube-api-access-zvmd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:23.328808 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:23.328771    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvmd5\" (UniqueName: \"kubernetes.io/projected/43e8a2ca-2687-475e-8b25-0d04734a5314-kube-api-access-zvmd5\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:23.328808 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:23.328801    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43e8a2ca-2687-475e-8b25-0d04734a5314-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:23.328808 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:23.328813    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e8a2ca-2687-475e-8b25-0d04734a5314-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:24.121591 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:24.121556    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm" event={"ID":"43e8a2ca-2687-475e-8b25-0d04734a5314","Type":"ContainerDied","Data":"4328fb3ae3a98dd2dcb6efc738e3536d684eed462aed83d7dc72d015c66cadb6"}
Nov 03 18:16:24.121591 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:24.121576    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"
Nov 03 18:16:24.122093 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:24.121624    6059 scope.go:122] "RemoveContainer" containerID="d1eaf49764981d51e7aad4d9f2cce86f864af212155a7d060143644e04e436f2"
Nov 03 18:16:24.291182 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:24.291143    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"]
Nov 03 18:16:24.302398 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:24.302347    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6bdf842xm"]
Nov 03 18:16:24.569468 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:24.569431    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:24.750666 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:24.750638    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:25.043211 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:25.043151    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:25.329897 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:25.329780    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:25.337611 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:25.337579    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43e8a2ca-2687-475e-8b25-0d04734a5314" path="/var/lib/kubelet/pods/43e8a2ca-2687-475e-8b25-0d04734a5314/volumes"
Nov 03 18:16:25.694628 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:25.694501    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:26.136453 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:26.136422    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:26.316017 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:26.315966    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:29.947143 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:29.947105    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:30.443040 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.443008    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"]
Nov 03 18:16:30.443513 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.443499    6059 cpu_manager.go:414] "containerMap: removing container" podUID="38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7" containerName="cleanup"
Nov 03 18:16:30.443563 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.443515    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e9e6f1-bf5b-49c6-a28b-5e223c62e9a7" containerName="cleanup"
Nov 03 18:16:30.443563 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.443539    6059 cpu_manager.go:414] "containerMap: removing container" podUID="25657ef4-b7fb-4fe6-848c-24dd44c22d62" containerName="payload-processing"
Nov 03 18:16:30.443563 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.443546    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25657ef4-b7fb-4fe6-848c-24dd44c22d62" containerName="payload-processing"
Nov 03 18:16:30.443668 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.443581    6059 cpu_manager.go:414] "containerMap: removing container" podUID="43e8a2ca-2687-475e-8b25-0d04734a5314" containerName="payload-processing"
Nov 03 18:16:30.443668 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.443589    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e8a2ca-2687-475e-8b25-0d04734a5314" containerName="payload-processing"
Nov 03 18:16:30.443668 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.443666    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e8a2ca-2687-475e-8b25-0d04734a5314" containerName="payload-processing"
Nov 03 18:16:30.443773 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.443681    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="25657ef4-b7fb-4fe6-848c-24dd44c22d62" containerName="payload-processing"
Nov 03 18:16:30.448178 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.448156    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"
Nov 03 18:16:30.486480 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.486451    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:30.504273 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.504240    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"routing-overlay\""
Nov 03 18:16:30.507717 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.507690    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"]
Nov 03 18:16:30.556018 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.555989    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-f2f8f7423350\""
Nov 03 18:16:30.556166 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.556091    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"payload-processing-external-model-e2e--b603cb9e-dockercfg-df5s6\""
Nov 03 18:16:30.605301 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.605268    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-8c9b0f\""
Nov 03 18:16:30.605301 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.605267    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"openshift-service-ca.crt\""
Nov 03 18:16:30.605301 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.605297    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"kube-root-ca.crt\""
Nov 03 18:16:30.628566 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.628536    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm\" (UID: \"3f3ca86d-8a62-4cdf-976d-efcec688ed8a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"
Nov 03 18:16:30.628765 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.628740    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm\" (UID: \"3f3ca86d-8a62-4cdf-976d-efcec688ed8a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"
Nov 03 18:16:30.628823 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.628779    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm\" (UID: \"3f3ca86d-8a62-4cdf-976d-efcec688ed8a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"
Nov 03 18:16:30.628870 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.628849    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9h5p\" (UniqueName: \"kubernetes.io/projected/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-kube-api-access-q9h5p\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm\" (UID: \"3f3ca86d-8a62-4cdf-976d-efcec688ed8a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"
Nov 03 18:16:30.730195 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.730159    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm\" (UID: \"3f3ca86d-8a62-4cdf-976d-efcec688ed8a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"
Nov 03 18:16:30.730195 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.730198    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm\" (UID: \"3f3ca86d-8a62-4cdf-976d-efcec688ed8a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"
Nov 03 18:16:30.730486 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.730225    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9h5p\" (UniqueName: \"kubernetes.io/projected/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-kube-api-access-q9h5p\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm\" (UID: \"3f3ca86d-8a62-4cdf-976d-efcec688ed8a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"
Nov 03 18:16:30.730486 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.730250    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm\" (UID: \"3f3ca86d-8a62-4cdf-976d-efcec688ed8a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"
Nov 03 18:16:30.730915 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.730888    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm\" (UID: \"3f3ca86d-8a62-4cdf-976d-efcec688ed8a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"
Nov 03 18:16:30.731039 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.731015    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm\" (UID: \"3f3ca86d-8a62-4cdf-976d-efcec688ed8a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"
Nov 03 18:16:30.732748 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.732730    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm\" (UID: \"3f3ca86d-8a62-4cdf-976d-efcec688ed8a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"
Nov 03 18:16:30.807976 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:30.807945    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9h5p\" (UniqueName: \"kubernetes.io/projected/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-kube-api-access-q9h5p\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm\" (UID: \"3f3ca86d-8a62-4cdf-976d-efcec688ed8a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"
Nov 03 18:16:31.061209 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:31.061127    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"
Nov 03 18:16:31.435862 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:16:31.435819    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3ca86d_8a62_4cdf_976d_efcec688ed8a.slice/crio-88ebd6922911e8aaecdb743ef84829e22d54ca82fa1fd6a8faa4c4db61b59231 WatchSource:0}: Error finding container 88ebd6922911e8aaecdb743ef84829e22d54ca82fa1fd6a8faa4c4db61b59231: Status 404 returned error can't find the container with id 88ebd6922911e8aaecdb743ef84829e22d54ca82fa1fd6a8faa4c4db61b59231
Nov 03 18:16:31.437929 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:31.437902    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"]
Nov 03 18:16:32.174722 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:32.174681    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm" event={"ID":"3f3ca86d-8a62-4cdf-976d-efcec688ed8a","Type":"ContainerStarted","Data":"695d0acfe660a1466e55115aa333e929d2d61a5d85efed765a4cfa5b42201a1e"}
Nov 03 18:16:32.175174 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:32.174777    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm" event={"ID":"3f3ca86d-8a62-4cdf-976d-efcec688ed8a","Type":"ContainerStarted","Data":"88ebd6922911e8aaecdb743ef84829e22d54ca82fa1fd6a8faa4c4db61b59231"}
Nov 03 18:16:32.175174 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:32.174813    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"
Nov 03 18:16:32.295792 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:32.295741    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm" podStartSLOduration=2.295727999 podStartE2EDuration="2.295727999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:16:32.295111293 +0000 UTC m=+3533.092824077" watchObservedRunningTime="2026-11-03 18:16:32.295727999 +0000 UTC m=+3533.093440800"
Nov 03 18:16:32.342888 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:32.342858    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:32.822590 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:32.822556    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"]
Nov 03 18:16:32.826512 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:32.826493    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"
Nov 03 18:16:32.936419 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:32.936384    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"routing-overlay\""
Nov 03 18:16:32.936661 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:32.936393    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"agc-cred-22f2e5b6fa88\""
Nov 03 18:16:32.936839 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:32.936393    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"kube-root-ca.crt\""
Nov 03 18:16:32.936839 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:32.936397    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-029ca0\""
Nov 03 18:16:32.936839 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:32.936416    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"openshift-service-ca.crt\""
Nov 03 18:16:32.936839 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:32.936424    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"payload-processing-external-model-e2e--58d254fa-dockercfg-99vcs\""
Nov 03 18:16:32.938924 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:32.938905    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"]
Nov 03 18:16:32.952902 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:32.952878    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fac721e-7ac1-48f0-8374-37ca3f687999-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz\" (UID: \"4fac721e-7ac1-48f0-8374-37ca3f687999\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"
Nov 03 18:16:32.953013 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:32.953001    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jfpr\" (UniqueName: \"kubernetes.io/projected/4fac721e-7ac1-48f0-8374-37ca3f687999-kube-api-access-7jfpr\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz\" (UID: \"4fac721e-7ac1-48f0-8374-37ca3f687999\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"
Nov 03 18:16:32.953075 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:32.953024    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4fac721e-7ac1-48f0-8374-37ca3f687999-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz\" (UID: \"4fac721e-7ac1-48f0-8374-37ca3f687999\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"
Nov 03 18:16:32.953075 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:32.953052    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4fac721e-7ac1-48f0-8374-37ca3f687999-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz\" (UID: \"4fac721e-7ac1-48f0-8374-37ca3f687999\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"
Nov 03 18:16:33.053767 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:33.053728    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jfpr\" (UniqueName: \"kubernetes.io/projected/4fac721e-7ac1-48f0-8374-37ca3f687999-kube-api-access-7jfpr\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz\" (UID: \"4fac721e-7ac1-48f0-8374-37ca3f687999\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"
Nov 03 18:16:33.053946 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:33.053775    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4fac721e-7ac1-48f0-8374-37ca3f687999-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz\" (UID: \"4fac721e-7ac1-48f0-8374-37ca3f687999\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"
Nov 03 18:16:33.053946 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:33.053827    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4fac721e-7ac1-48f0-8374-37ca3f687999-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz\" (UID: \"4fac721e-7ac1-48f0-8374-37ca3f687999\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"
Nov 03 18:16:33.053946 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:33.053875    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fac721e-7ac1-48f0-8374-37ca3f687999-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz\" (UID: \"4fac721e-7ac1-48f0-8374-37ca3f687999\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"
Nov 03 18:16:33.054512 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:33.054486    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4fac721e-7ac1-48f0-8374-37ca3f687999-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz\" (UID: \"4fac721e-7ac1-48f0-8374-37ca3f687999\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"
Nov 03 18:16:33.054638 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:33.054576    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fac721e-7ac1-48f0-8374-37ca3f687999-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz\" (UID: \"4fac721e-7ac1-48f0-8374-37ca3f687999\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"
Nov 03 18:16:33.056203 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:33.056180    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4fac721e-7ac1-48f0-8374-37ca3f687999-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz\" (UID: \"4fac721e-7ac1-48f0-8374-37ca3f687999\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"
Nov 03 18:16:33.117863 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:33.117784    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jfpr\" (UniqueName: \"kubernetes.io/projected/4fac721e-7ac1-48f0-8374-37ca3f687999-kube-api-access-7jfpr\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz\" (UID: \"4fac721e-7ac1-48f0-8374-37ca3f687999\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"
Nov 03 18:16:33.140875 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:33.140840    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"
Nov 03 18:16:33.545101 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:16:33.545069    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fac721e_7ac1_48f0_8374_37ca3f687999.slice/crio-d1e27931ac9a3547bbd030133b3ab45c1ac53379d4f5e81ac416d7061de3813c WatchSource:0}: Error finding container d1e27931ac9a3547bbd030133b3ab45c1ac53379d4f5e81ac416d7061de3813c: Status 404 returned error can't find the container with id d1e27931ac9a3547bbd030133b3ab45c1ac53379d4f5e81ac416d7061de3813c
Nov 03 18:16:33.547508 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:33.547485    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"]
Nov 03 18:16:34.194410 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:34.194367    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz" event={"ID":"4fac721e-7ac1-48f0-8374-37ca3f687999","Type":"ContainerStarted","Data":"7740491eca404089d554f201bb0fba40193a7d9860642a9ef73acdff91408d91"}
Nov 03 18:16:34.194598 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:34.194434    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz" event={"ID":"4fac721e-7ac1-48f0-8374-37ca3f687999","Type":"ContainerStarted","Data":"d1e27931ac9a3547bbd030133b3ab45c1ac53379d4f5e81ac416d7061de3813c"}
Nov 03 18:16:34.194598 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:34.194468    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"
Nov 03 18:16:34.316264 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:34.316203    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz" podStartSLOduration=2.316188309 podStartE2EDuration="2.316188309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:16:34.314789979 +0000 UTC m=+3535.112502760" watchObservedRunningTime="2026-11-03 18:16:34.316188309 +0000 UTC m=+3535.113901089"
Nov 03 18:16:35.348402 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:35.348351    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:36.860951 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:16:36.860917    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:16:40.163914 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:40.163864    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:43.190154 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:43.190074    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"
Nov 03 18:16:44.975017 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:44.974977    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:45.204219 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:45.204192    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"
Nov 03 18:16:49.855766 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:49.855729    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:50.614778 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:50.614743    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:54.661936 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:54.661897    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:16:59.470847 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:16:59.470790    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:04.346457 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:04.346420    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:06.424327 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:06.424270    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:06.860809 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:17:06.860770    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:17:09.166386 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:09.166317    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:13.963638 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:13.963599    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:15.051316 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:15.051281    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58"]
Nov 03 18:17:15.102887 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:15.102856    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34bf0720-thh58"]
Nov 03 18:17:15.335141 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:15.335063    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aa61d3c-fc87-4b7b-a873-348eb918ffd7" path="/var/lib/kubelet/pods/1aa61d3c-fc87-4b7b-a873-348eb918ffd7/volumes"
Nov 03 18:17:18.845098 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:18.845060    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:23.796479 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:23.796445    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:28.051281 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:28.051246    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db637277-44w92"]
Nov 03 18:17:28.102767 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:28.102734    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db637277-44w92"]
Nov 03 18:17:28.571437 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:28.571402    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:29.335538 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:29.335503    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c306988-214c-4fdc-8548-7b19ca819d66" path="/var/lib/kubelet/pods/9c306988-214c-4fdc-8548-7b19ca819d66/volumes"
Nov 03 18:17:30.484742 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:30.484703    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:30.668203 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:30.668170    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:33.460537 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:33.460484    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:34.369067 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:34.369032    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"]
Nov 03 18:17:34.373911 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:34.373892    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"
Nov 03 18:17:34.430958 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:34.430927    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-d13ad376f3be\""
Nov 03 18:17:34.431346 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:34.431201    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/21f21323-a8dd-4723-971e-cfca12f31f74-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84\" (UID: \"21f21323-a8dd-4723-971e-cfca12f31f74\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"
Nov 03 18:17:34.431651 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:34.431616    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21f21323-a8dd-4723-971e-cfca12f31f74-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84\" (UID: \"21f21323-a8dd-4723-971e-cfca12f31f74\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"
Nov 03 18:17:34.431785 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:34.431764    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21f21323-a8dd-4723-971e-cfca12f31f74-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84\" (UID: \"21f21323-a8dd-4723-971e-cfca12f31f74\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"
Nov 03 18:17:34.431856 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:34.431822    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdprz\" (UniqueName: \"kubernetes.io/projected/21f21323-a8dd-4723-971e-cfca12f31f74-kube-api-access-cdprz\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84\" (UID: \"21f21323-a8dd-4723-971e-cfca12f31f74\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"
Nov 03 18:17:34.484956 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:34.484925    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"]
Nov 03 18:17:34.534140 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:34.534101    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdprz\" (UniqueName: \"kubernetes.io/projected/21f21323-a8dd-4723-971e-cfca12f31f74-kube-api-access-cdprz\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84\" (UID: \"21f21323-a8dd-4723-971e-cfca12f31f74\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"
Nov 03 18:17:34.534306 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:34.534244    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/21f21323-a8dd-4723-971e-cfca12f31f74-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84\" (UID: \"21f21323-a8dd-4723-971e-cfca12f31f74\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"
Nov 03 18:17:34.534350 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:34.534334    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21f21323-a8dd-4723-971e-cfca12f31f74-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84\" (UID: \"21f21323-a8dd-4723-971e-cfca12f31f74\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"
Nov 03 18:17:34.534436 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:34.534420    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21f21323-a8dd-4723-971e-cfca12f31f74-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84\" (UID: \"21f21323-a8dd-4723-971e-cfca12f31f74\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"
Nov 03 18:17:34.535067 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:34.535038    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21f21323-a8dd-4723-971e-cfca12f31f74-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84\" (UID: \"21f21323-a8dd-4723-971e-cfca12f31f74\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"
Nov 03 18:17:34.535207 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:34.535108    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21f21323-a8dd-4723-971e-cfca12f31f74-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84\" (UID: \"21f21323-a8dd-4723-971e-cfca12f31f74\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"
Nov 03 18:17:34.536952 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:34.536932    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/21f21323-a8dd-4723-971e-cfca12f31f74-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84\" (UID: \"21f21323-a8dd-4723-971e-cfca12f31f74\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"
Nov 03 18:17:34.597498 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:34.597459    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdprz\" (UniqueName: \"kubernetes.io/projected/21f21323-a8dd-4723-971e-cfca12f31f74-kube-api-access-cdprz\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84\" (UID: \"21f21323-a8dd-4723-971e-cfca12f31f74\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"
Nov 03 18:17:34.686613 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:34.686527    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"
Nov 03 18:17:35.058091 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:17:35.058059    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f21323_a8dd_4723_971e_cfca12f31f74.slice/crio-4c3f0b513b905858f1d6acde3a463f7b731fe3c73f98e1b276c44cd95464ee6c WatchSource:0}: Error finding container 4c3f0b513b905858f1d6acde3a463f7b731fe3c73f98e1b276c44cd95464ee6c: Status 404 returned error can't find the container with id 4c3f0b513b905858f1d6acde3a463f7b731fe3c73f98e1b276c44cd95464ee6c
Nov 03 18:17:35.060408 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:35.060384    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"]
Nov 03 18:17:35.548195 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:35.548162    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84" event={"ID":"21f21323-a8dd-4723-971e-cfca12f31f74","Type":"ContainerStarted","Data":"c3e466a9a53712dc03ab08a979d48f3fd7bc7f2e9b8c1a97adcd2d5a38022fb0"}
Nov 03 18:17:35.548608 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:35.548231    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"
Nov 03 18:17:35.548608 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:35.548255    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84" event={"ID":"21f21323-a8dd-4723-971e-cfca12f31f74","Type":"ContainerStarted","Data":"4c3f0b513b905858f1d6acde3a463f7b731fe3c73f98e1b276c44cd95464ee6c"}
Nov 03 18:17:35.706117 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:35.706059    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84" podStartSLOduration=1.706042948 podStartE2EDuration="1.706042948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:17:35.703688173 +0000 UTC m=+3596.501400954" watchObservedRunningTime="2026-11-03 18:17:35.706042948 +0000 UTC m=+3596.503755726"
Nov 03 18:17:36.750517 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:36.750483    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"]
Nov 03 18:17:36.754443 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:36.754421    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"
Nov 03 18:17:36.808738 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:36.808704    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"agc-cred-dff87693fc31\""
Nov 03 18:17:36.817797 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:36.817767    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"]
Nov 03 18:17:36.860817 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:36.860788    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skdff\" (UniqueName: \"kubernetes.io/projected/d58b44cf-6725-4898-a0a0-94707f9f397e-kube-api-access-skdff\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd\" (UID: \"d58b44cf-6725-4898-a0a0-94707f9f397e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"
Nov 03 18:17:36.860980 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:36.860911    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d58b44cf-6725-4898-a0a0-94707f9f397e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd\" (UID: \"d58b44cf-6725-4898-a0a0-94707f9f397e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"
Nov 03 18:17:36.860980 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:36.860948    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d58b44cf-6725-4898-a0a0-94707f9f397e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd\" (UID: \"d58b44cf-6725-4898-a0a0-94707f9f397e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"
Nov 03 18:17:36.860980 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:36.860965    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d58b44cf-6725-4898-a0a0-94707f9f397e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd\" (UID: \"d58b44cf-6725-4898-a0a0-94707f9f397e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"
Nov 03 18:17:36.861509 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:17:36.861484    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:17:36.962338 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:36.962300    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skdff\" (UniqueName: \"kubernetes.io/projected/d58b44cf-6725-4898-a0a0-94707f9f397e-kube-api-access-skdff\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd\" (UID: \"d58b44cf-6725-4898-a0a0-94707f9f397e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"
Nov 03 18:17:36.962606 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:36.962400    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d58b44cf-6725-4898-a0a0-94707f9f397e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd\" (UID: \"d58b44cf-6725-4898-a0a0-94707f9f397e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"
Nov 03 18:17:36.962606 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:36.962432    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d58b44cf-6725-4898-a0a0-94707f9f397e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd\" (UID: \"d58b44cf-6725-4898-a0a0-94707f9f397e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"
Nov 03 18:17:36.962606 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:36.962447    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d58b44cf-6725-4898-a0a0-94707f9f397e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd\" (UID: \"d58b44cf-6725-4898-a0a0-94707f9f397e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"
Nov 03 18:17:36.963162 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:36.963137    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d58b44cf-6725-4898-a0a0-94707f9f397e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd\" (UID: \"d58b44cf-6725-4898-a0a0-94707f9f397e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"
Nov 03 18:17:36.963266 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:36.963236    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d58b44cf-6725-4898-a0a0-94707f9f397e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd\" (UID: \"d58b44cf-6725-4898-a0a0-94707f9f397e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"
Nov 03 18:17:36.965067 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:36.965047    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d58b44cf-6725-4898-a0a0-94707f9f397e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd\" (UID: \"d58b44cf-6725-4898-a0a0-94707f9f397e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"
Nov 03 18:17:37.025622 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:37.025545    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skdff\" (UniqueName: \"kubernetes.io/projected/d58b44cf-6725-4898-a0a0-94707f9f397e-kube-api-access-skdff\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd\" (UID: \"d58b44cf-6725-4898-a0a0-94707f9f397e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"
Nov 03 18:17:37.068323 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:37.068275    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"
Nov 03 18:17:37.465377 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:17:37.465320    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd58b44cf_6725_4898_a0a0_94707f9f397e.slice/crio-d69098dde965c716a7eea84fbabc3d13d71f468c0a2853d91e19fe711df0c66c WatchSource:0}: Error finding container d69098dde965c716a7eea84fbabc3d13d71f468c0a2853d91e19fe711df0c66c: Status 404 returned error can't find the container with id d69098dde965c716a7eea84fbabc3d13d71f468c0a2853d91e19fe711df0c66c
Nov 03 18:17:37.467002 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:37.466965    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"]
Nov 03 18:17:37.564078 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:37.564033    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd" event={"ID":"d58b44cf-6725-4898-a0a0-94707f9f397e","Type":"ContainerStarted","Data":"6da5f37ff3ba4474e46bd732c0b9281119e30a9fa7da9658be3fbc69bfb3d726"}
Nov 03 18:17:37.564289 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:37.564142    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"
Nov 03 18:17:37.564289 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:37.564181    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd" event={"ID":"d58b44cf-6725-4898-a0a0-94707f9f397e","Type":"ContainerStarted","Data":"d69098dde965c716a7eea84fbabc3d13d71f468c0a2853d91e19fe711df0c66c"}
Nov 03 18:17:37.685894 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:37.685834    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd" podStartSLOduration=1.685820901 podStartE2EDuration="1.685820901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:17:37.684419507 +0000 UTC m=+3598.482132290" watchObservedRunningTime="2026-11-03 18:17:37.685820901 +0000 UTC m=+3598.483533682"
Nov 03 18:17:38.351396 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:38.351344    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:39.051001 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:39.050967    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aece27a8-jg7b7"]
Nov 03 18:17:39.102877 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:39.102842    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aece27a8-jg7b7"]
Nov 03 18:17:39.335610 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:39.335531    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7ff32de-2054-4ef4-b11a-63d59ebe56bf" path="/var/lib/kubelet/pods/b7ff32de-2054-4ef4-b11a-63d59ebe56bf/volumes"
Nov 03 18:17:39.897853 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:39.897816    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:17:39.917041 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:39.917014    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:17:39.921820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:39.921798    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:17:39.934854 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:39.934832    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:17:43.155710 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:43.155666    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:46.557335 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:46.557306    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"
Nov 03 18:17:46.732933 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:46.732896    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"]
Nov 03 18:17:48.010335 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:48.010301    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:48.573203 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:48.573170    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"
Nov 03 18:17:48.748701 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:48.748669    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"]
Nov 03 18:17:50.944997 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:50.944964    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:51.734037 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:51.733996    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm" podUID="3f3ca86d-8a62-4cdf-976d-efcec688ed8a" containerName="payload-processing" containerID="cri-o://695d0acfe660a1466e55115aa333e929d2d61a5d85efed765a4cfa5b42201a1e" gracePeriod=25
Nov 03 18:17:51.983400 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:51.983350    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"
Nov 03 18:17:52.122601 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:52.122516    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-provider-credentials\") pod \"3f3ca86d-8a62-4cdf-976d-efcec688ed8a\" (UID: \"3f3ca86d-8a62-4cdf-976d-efcec688ed8a\") "
Nov 03 18:17:52.122601 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:52.122601    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-kube-api-access-q9h5p\" (UniqueName: \"kubernetes.io/projected/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-kube-api-access-q9h5p\") pod \"3f3ca86d-8a62-4cdf-976d-efcec688ed8a\" (UID: \"3f3ca86d-8a62-4cdf-976d-efcec688ed8a\") "
Nov 03 18:17:52.122844 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:52.122705    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-routing-overlay-volume\") pod \"3f3ca86d-8a62-4cdf-976d-efcec688ed8a\" (UID: \"3f3ca86d-8a62-4cdf-976d-efcec688ed8a\") "
Nov 03 18:17:52.122844 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:52.122744    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-plugins-config-volume\") pod \"3f3ca86d-8a62-4cdf-976d-efcec688ed8a\" (UID: \"3f3ca86d-8a62-4cdf-976d-efcec688ed8a\") "
Nov 03 18:17:52.123142 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:52.123116    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-routing-overlay-volume" pod "3f3ca86d-8a62-4cdf-976d-efcec688ed8a" (UID: "3f3ca86d-8a62-4cdf-976d-efcec688ed8a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:52.123307 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:52.123173    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-plugins-config-volume" pod "3f3ca86d-8a62-4cdf-976d-efcec688ed8a" (UID: "3f3ca86d-8a62-4cdf-976d-efcec688ed8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:52.123535 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:52.123515    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:52.123616 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:52.123540    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:52.124776 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:52.124750    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-provider-credentials" pod "3f3ca86d-8a62-4cdf-976d-efcec688ed8a" (UID: "3f3ca86d-8a62-4cdf-976d-efcec688ed8a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:52.124860 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:52.124811    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-kube-api-access-q9h5p" pod "3f3ca86d-8a62-4cdf-976d-efcec688ed8a" (UID: "3f3ca86d-8a62-4cdf-976d-efcec688ed8a"). InnerVolumeSpecName "kube-api-access-q9h5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:52.224923 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:52.224892    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:52.224923 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:52.224920    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9h5p\" (UniqueName: \"kubernetes.io/projected/3f3ca86d-8a62-4cdf-976d-efcec688ed8a-kube-api-access-q9h5p\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:52.649349 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:52.649311    6059 generic.go:356] "Generic (PLEG): container finished" podID="3f3ca86d-8a62-4cdf-976d-efcec688ed8a" containerID="695d0acfe660a1466e55115aa333e929d2d61a5d85efed765a4cfa5b42201a1e" exitCode=0
Nov 03 18:17:52.649522 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:52.649384    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"
Nov 03 18:17:52.649522 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:52.649403    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm" event={"ID":"3f3ca86d-8a62-4cdf-976d-efcec688ed8a","Type":"ContainerDied","Data":"695d0acfe660a1466e55115aa333e929d2d61a5d85efed765a4cfa5b42201a1e"}
Nov 03 18:17:52.649522 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:52.649455    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm" event={"ID":"3f3ca86d-8a62-4cdf-976d-efcec688ed8a","Type":"ContainerDied","Data":"88ebd6922911e8aaecdb743ef84829e22d54ca82fa1fd6a8faa4c4db61b59231"}
Nov 03 18:17:52.649522 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:52.649480    6059 scope.go:122] "RemoveContainer" containerID="695d0acfe660a1466e55115aa333e929d2d61a5d85efed765a4cfa5b42201a1e"
Nov 03 18:17:52.659292 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:52.659273    6059 scope.go:122] "RemoveContainer" containerID="695d0acfe660a1466e55115aa333e929d2d61a5d85efed765a4cfa5b42201a1e"
Nov 03 18:17:52.659641 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:17:52.659615    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"695d0acfe660a1466e55115aa333e929d2d61a5d85efed765a4cfa5b42201a1e\": container with ID starting with 695d0acfe660a1466e55115aa333e929d2d61a5d85efed765a4cfa5b42201a1e not found: ID does not exist" containerID="695d0acfe660a1466e55115aa333e929d2d61a5d85efed765a4cfa5b42201a1e"
Nov 03 18:17:52.659709 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:52.659650    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"695d0acfe660a1466e55115aa333e929d2d61a5d85efed765a4cfa5b42201a1e"} err="failed to get container status \"695d0acfe660a1466e55115aa333e929d2d61a5d85efed765a4cfa5b42201a1e\": rpc error: code = NotFound desc = could not find container \"695d0acfe660a1466e55115aa333e929d2d61a5d85efed765a4cfa5b42201a1e\": container with ID starting with 695d0acfe660a1466e55115aa333e929d2d61a5d85efed765a4cfa5b42201a1e not found: ID does not exist"
Nov 03 18:17:52.820653 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:52.820619    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"]
Nov 03 18:17:52.824663 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:52.824639    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4cz8zm"]
Nov 03 18:17:52.852229 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:52.852196    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:17:53.336176 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:53.336137    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f3ca86d-8a62-4cdf-976d-efcec688ed8a" path="/var/lib/kubelet/pods/3f3ca86d-8a62-4cdf-976d-efcec688ed8a/volumes"
Nov 03 18:17:53.749814 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:53.749765    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz" podUID="4fac721e-7ac1-48f0-8374-37ca3f687999" containerName="payload-processing" containerID="cri-o://7740491eca404089d554f201bb0fba40193a7d9860642a9ef73acdff91408d91" gracePeriod=25
Nov 03 18:17:54.005860 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:54.005792    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"
Nov 03 18:17:54.145654 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:54.145623    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fac721e-7ac1-48f0-8374-37ca3f687999-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4fac721e-7ac1-48f0-8374-37ca3f687999-routing-overlay-volume\") pod \"4fac721e-7ac1-48f0-8374-37ca3f687999\" (UID: \"4fac721e-7ac1-48f0-8374-37ca3f687999\") "
Nov 03 18:17:54.145826 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:54.145722    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fac721e-7ac1-48f0-8374-37ca3f687999-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fac721e-7ac1-48f0-8374-37ca3f687999-plugins-config-volume\") pod \"4fac721e-7ac1-48f0-8374-37ca3f687999\" (UID: \"4fac721e-7ac1-48f0-8374-37ca3f687999\") "
Nov 03 18:17:54.145826 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:54.145759    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fac721e-7ac1-48f0-8374-37ca3f687999-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4fac721e-7ac1-48f0-8374-37ca3f687999-provider-credentials\") pod \"4fac721e-7ac1-48f0-8374-37ca3f687999\" (UID: \"4fac721e-7ac1-48f0-8374-37ca3f687999\") "
Nov 03 18:17:54.145944 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:54.145830    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fac721e-7ac1-48f0-8374-37ca3f687999-kube-api-access-7jfpr\" (UniqueName: \"kubernetes.io/projected/4fac721e-7ac1-48f0-8374-37ca3f687999-kube-api-access-7jfpr\") pod \"4fac721e-7ac1-48f0-8374-37ca3f687999\" (UID: \"4fac721e-7ac1-48f0-8374-37ca3f687999\") "
Nov 03 18:17:54.146103 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:54.146073    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fac721e-7ac1-48f0-8374-37ca3f687999-routing-overlay-volume" pod "4fac721e-7ac1-48f0-8374-37ca3f687999" (UID: "4fac721e-7ac1-48f0-8374-37ca3f687999"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:54.146173 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:54.146139    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fac721e-7ac1-48f0-8374-37ca3f687999-plugins-config-volume" pod "4fac721e-7ac1-48f0-8374-37ca3f687999" (UID: "4fac721e-7ac1-48f0-8374-37ca3f687999"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:54.147943 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:54.147916    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fac721e-7ac1-48f0-8374-37ca3f687999-kube-api-access-7jfpr" pod "4fac721e-7ac1-48f0-8374-37ca3f687999" (UID: "4fac721e-7ac1-48f0-8374-37ca3f687999"). InnerVolumeSpecName "kube-api-access-7jfpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:54.148087 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:54.147950    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fac721e-7ac1-48f0-8374-37ca3f687999-provider-credentials" pod "4fac721e-7ac1-48f0-8374-37ca3f687999" (UID: "4fac721e-7ac1-48f0-8374-37ca3f687999"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:54.247478 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:54.247444    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jfpr\" (UniqueName: \"kubernetes.io/projected/4fac721e-7ac1-48f0-8374-37ca3f687999-kube-api-access-7jfpr\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:54.247478 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:54.247473    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4fac721e-7ac1-48f0-8374-37ca3f687999-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:54.247478 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:54.247484    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fac721e-7ac1-48f0-8374-37ca3f687999-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:54.247734 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:54.247493    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4fac721e-7ac1-48f0-8374-37ca3f687999-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:54.662940 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:54.662901    6059 generic.go:356] "Generic (PLEG): container finished" podID="4fac721e-7ac1-48f0-8374-37ca3f687999" containerID="7740491eca404089d554f201bb0fba40193a7d9860642a9ef73acdff91408d91" exitCode=0
Nov 03 18:17:54.663468 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:54.662944    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"
Nov 03 18:17:54.663468 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:54.662987    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz" event={"ID":"4fac721e-7ac1-48f0-8374-37ca3f687999","Type":"ContainerDied","Data":"7740491eca404089d554f201bb0fba40193a7d9860642a9ef73acdff91408d91"}
Nov 03 18:17:54.663468 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:54.663044    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz" event={"ID":"4fac721e-7ac1-48f0-8374-37ca3f687999","Type":"ContainerDied","Data":"d1e27931ac9a3547bbd030133b3ab45c1ac53379d4f5e81ac416d7061de3813c"}
Nov 03 18:17:54.663468 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:54.663072    6059 scope.go:122] "RemoveContainer" containerID="7740491eca404089d554f201bb0fba40193a7d9860642a9ef73acdff91408d91"
Nov 03 18:17:54.673272 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:54.673255    6059 scope.go:122] "RemoveContainer" containerID="7740491eca404089d554f201bb0fba40193a7d9860642a9ef73acdff91408d91"
Nov 03 18:17:54.673589 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:17:54.673552    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7740491eca404089d554f201bb0fba40193a7d9860642a9ef73acdff91408d91\": container with ID starting with 7740491eca404089d554f201bb0fba40193a7d9860642a9ef73acdff91408d91 not found: ID does not exist" containerID="7740491eca404089d554f201bb0fba40193a7d9860642a9ef73acdff91408d91"
Nov 03 18:17:54.673695 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:54.673594    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7740491eca404089d554f201bb0fba40193a7d9860642a9ef73acdff91408d91"} err="failed to get container status \"7740491eca404089d554f201bb0fba40193a7d9860642a9ef73acdff91408d91\": rpc error: code = NotFound desc = could not find container \"7740491eca404089d554f201bb0fba40193a7d9860642a9ef73acdff91408d91\": container with ID starting with 7740491eca404089d554f201bb0fba40193a7d9860642a9ef73acdff91408d91 not found: ID does not exist"
Nov 03 18:17:54.833248 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:54.833204    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"]
Nov 03 18:17:54.837508 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:54.837482    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd8wcwfz"]
Nov 03 18:17:55.051185 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:55.051147    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q"]
Nov 03 18:17:55.053627 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:55.053604    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26398498-7bj7q"]
Nov 03 18:17:55.335248 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:55.335167    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4781b5b6-14d9-4261-8cf3-043e2b912a28" path="/var/lib/kubelet/pods/4781b5b6-14d9-4261-8cf3-043e2b912a28/volumes"
Nov 03 18:17:55.335601 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:55.335586    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fac721e-7ac1-48f0-8374-37ca3f687999" path="/var/lib/kubelet/pods/4fac721e-7ac1-48f0-8374-37ca3f687999/volumes"
Nov 03 18:17:57.630665 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:17:57.630588    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:02.464505 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:18:02.464464    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:06.860468 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:18:06.860431    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:18:07.262967 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:18:07.262937    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:09.050275 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:18:09.050240    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x"]
Nov 03 18:18:09.101781 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:18:09.101743    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec2761d7-fhb8x"]
Nov 03 18:18:09.335220 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:18:09.335138    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f62e4b5c-d858-4d3a-9149-c4c76d6c21c6" path="/var/lib/kubelet/pods/f62e4b5c-d858-4d3a-9149-c4c76d6c21c6/volumes"
Nov 03 18:18:12.137686 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:18:12.137649    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:17.081217 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:18:17.081180    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:21.920387 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:18:21.920343    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:26.677269 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:18:26.677236    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:30.154982 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:18:30.154944    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:31.557756 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:18:31.557722    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:36.346319 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:18:36.346290    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:36.861568 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:18:36.861531    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:18:41.356305 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:18:41.356250    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:42.945906 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:18:42.945871    6059 scope.go:122] "RemoveContainer" containerID="ca1790540895224ba1e600ca4d5afdb75b3caacd69c0f1040191f97ac24dd92e"
Nov 03 18:18:42.958848 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:18:42.958826    6059 scope.go:122] "RemoveContainer" containerID="7cf12d1684f8feee52bd767edcab34433a40159931fa8358e55d2ddccca8699c"
Nov 03 18:18:42.967421 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:18:42.967399    6059 scope.go:122] "RemoveContainer" containerID="71b8205363f734940986b9f4fc713a96911c449ffa753a49b1e8ca20071d30ac"
Nov 03 18:18:46.164686 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:18:46.164655    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:51.073118 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:18:51.073000    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:51.319968 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:18:51.319900    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:18:55.869964 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:18:55.869931    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:00.769033 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:19:00.768939    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:05.553861 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:19:05.553828    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:06.861152 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:19:06.861114    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:19:10.358918 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:19:10.358884    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:15.261172 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:19:15.261134    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:16.051757 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:19:16.051720    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw"]
Nov 03 18:19:16.103249 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:19:16.103212    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e029470-vmgvw"]
Nov 03 18:19:17.335055 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:19:17.334975    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69377556-aba2-4899-b10d-f6eb868a6ca4" path="/var/lib/kubelet/pods/69377556-aba2-4899-b10d-f6eb868a6ca4/volumes"
Nov 03 18:19:20.069211 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:19:20.069160    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:24.966904 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:19:24.966850    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:29.869728 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:19:29.869656    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:34.665809 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:19:34.665769    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:36.861618 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:19:36.861576    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:19:39.494618 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:19:39.494576    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:39.684122 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:19:39.684080    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:41.895602 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:19:41.895574    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:42.074342 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:19:42.074306    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:43.007451 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:19:43.007419    6059 scope.go:122] "RemoveContainer" containerID="51a5a0eeb0d390e8731b6369e72af375be33d5edb82c5116a1c1dc8e2a02a532"
Nov 03 18:19:43.018993 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:19:43.018972    6059 scope.go:122] "RemoveContainer" containerID="0dfee0ded9d1455afc6cf088ee8ce70ccc7ef251d14de8e9379a8a40b0db24d9"
Nov 03 18:19:44.285641 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:19:44.285603    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:49.141436 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:19:49.141401    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:51.759911 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:19:51.759875    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:53.980329 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:19:53.980293    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:19:58.965610 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:19:58.965320    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:03.768155 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:03.768093    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:06.860875 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:20:06.860835    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:20:08.554749 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:08.554712    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:13.461137 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:13.461095    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:18.255970 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:18.255933    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:23.162633 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:23.162599    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:28.090287 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:28.090254    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:31.572962 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:31.572926    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"]
Nov 03 18:20:31.573427 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:31.573164    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j" podUID="565e1871-9784-4483-ad19-17413b47017e" containerName="istio-proxy" containerID="cri-o://07beb1a50ed37b27bf44b14902c2205eeedce86545d9e24ae8ac98bbf768bb49" gracePeriod=30
Nov 03 18:20:31.625100 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:31.625052    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"]
Nov 03 18:20:31.625328 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:31.625259    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr" podUID="6af1e128-ecfb-4449-a2ab-0d6a3732ca71" containerName="istio-proxy" containerID="cri-o://8b96819365fa52902fa2c94775775b9586c3d3d17815337171acb8006bbccb06" gracePeriod=30
Nov 03 18:20:31.627142 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:31.627112    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"]
Nov 03 18:20:31.627386 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:31.627327    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk" podUID="8c8cc50b-9b04-4398-a692-a75d4125fba8" containerName="istio-proxy" containerID="cri-o://904ba53e5789cc5d699261e4ba685e181143a96e15ec956dcd3082f9bdbca422" gracePeriod=30
Nov 03 18:20:31.628996 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:31.628966    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"]
Nov 03 18:20:31.629196 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:31.629172    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw" podUID="bbcbf9e3-2774-4872-95d5-4f27b7ad17cc" containerName="istio-proxy" containerID="cri-o://aacbd52827741c953232fcb544c82cb856e25476e3b584675e16631e8b7382b7" gracePeriod=30
Nov 03 18:20:33.060168 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:33.059956    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:36.877879 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:20:36.877699    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:20:36.915897 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:36.915870    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:20:36.934924 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:36.934830    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:20:36.938797 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:36.938764    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:20:36.977727 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:36.977699    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:20:37.099300 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.099268    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-workload-certs\") pod \"565e1871-9784-4483-ad19-17413b47017e\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") "
Nov 03 18:20:37.099300 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.099315    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c8cc50b-9b04-4398-a692-a75d4125fba8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8cc50b-9b04-4398-a692-a75d4125fba8-istiod-ca-cert\") pod \"8c8cc50b-9b04-4398-a692-a75d4125fba8\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") "
Nov 03 18:20:37.099586 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.099343    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-token\" (UniqueName: \"kubernetes.io/projected/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-token\") pod \"8c8cc50b-9b04-4398-a692-a75d4125fba8\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") "
Nov 03 18:20:37.099586 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.099431    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/565e1871-9784-4483-ad19-17413b47017e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565e1871-9784-4483-ad19-17413b47017e-istiod-ca-cert\") pod \"565e1871-9784-4483-ad19-17413b47017e\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") "
Nov 03 18:20:37.099586 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.099463    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-credential-socket\") pod \"565e1871-9784-4483-ad19-17413b47017e\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") "
Nov 03 18:20:37.099586 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.099499    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-credential-socket\") pod \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") "
Nov 03 18:20:37.099586 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.099534    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-workload-certs\") pod \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") "
Nov 03 18:20:37.099586 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.099569    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istiod-ca-cert\") pod \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") "
Nov 03 18:20:37.100645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.099594    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istiod-ca-cert\") pod \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") "
Nov 03 18:20:37.100645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.099631    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-envoy\") pod \"8c8cc50b-9b04-4398-a692-a75d4125fba8\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") "
Nov 03 18:20:37.100645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.099669    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-workload-socket\") pod \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") "
Nov 03 18:20:37.100645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.099687    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-workload-certs" pod "565e1871-9784-4483-ad19-17413b47017e" (UID: "565e1871-9784-4483-ad19-17413b47017e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:37.100645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.099701    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-istio-envoy\") pod \"565e1871-9784-4483-ad19-17413b47017e\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") "
Nov 03 18:20:37.100645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.099910    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-credential-socket" pod "6af1e128-ecfb-4449-a2ab-0d6a3732ca71" (UID: "6af1e128-ecfb-4449-a2ab-0d6a3732ca71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:37.100645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.099939    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8cc50b-9b04-4398-a692-a75d4125fba8-istiod-ca-cert" pod "8c8cc50b-9b04-4398-a692-a75d4125fba8" (UID: "8c8cc50b-9b04-4398-a692-a75d4125fba8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:20:37.100645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.099965    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-workload-certs" pod "6af1e128-ecfb-4449-a2ab-0d6a3732ca71" (UID: "6af1e128-ecfb-4449-a2ab-0d6a3732ca71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:37.100645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.100018    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/565e1871-9784-4483-ad19-17413b47017e-istiod-ca-cert" pod "565e1871-9784-4483-ad19-17413b47017e" (UID: "565e1871-9784-4483-ad19-17413b47017e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:20:37.100645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.100203    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-workload-socket" pod "bbcbf9e3-2774-4872-95d5-4f27b7ad17cc" (UID: "bbcbf9e3-2774-4872-95d5-4f27b7ad17cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:37.100645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.100255    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istiod-ca-cert" pod "bbcbf9e3-2774-4872-95d5-4f27b7ad17cc" (UID: "bbcbf9e3-2774-4872-95d5-4f27b7ad17cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:20:37.100645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.100265    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-credential-socket" pod "565e1871-9784-4483-ad19-17413b47017e" (UID: "565e1871-9784-4483-ad19-17413b47017e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:37.100645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.100266    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-podinfo\") pod \"8c8cc50b-9b04-4398-a692-a75d4125fba8\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") "
Nov 03 18:20:37.100645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.100322    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-podinfo\") pod \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") "
Nov 03 18:20:37.100645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.100382    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-workload-socket\") pod \"8c8cc50b-9b04-4398-a692-a75d4125fba8\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") "
Nov 03 18:20:37.100645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.100424    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-token\" (UniqueName: \"kubernetes.io/projected/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-token\") pod \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") "
Nov 03 18:20:37.100645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.100458    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/565e1871-9784-4483-ad19-17413b47017e-istio-token\" (UniqueName: \"kubernetes.io/projected/565e1871-9784-4483-ad19-17413b47017e-istio-token\") pod \"565e1871-9784-4483-ad19-17413b47017e\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") "
Nov 03 18:20:37.100645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.100494    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-credential-socket\") pod \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") "
Nov 03 18:20:37.100645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.100520    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-token\" (UniqueName: \"kubernetes.io/projected/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-token\") pod \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") "
Nov 03 18:20:37.100645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.100571    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-workload-socket\") pod \"565e1871-9784-4483-ad19-17413b47017e\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") "
Nov 03 18:20:37.100645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.100606    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-workload-certs\") pod \"8c8cc50b-9b04-4398-a692-a75d4125fba8\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") "
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.100706    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istiod-ca-cert" pod "6af1e128-ecfb-4449-a2ab-0d6a3732ca71" (UID: "6af1e128-ecfb-4449-a2ab-0d6a3732ca71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.100812    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-workload-socket" pod "8c8cc50b-9b04-4398-a692-a75d4125fba8" (UID: "8c8cc50b-9b04-4398-a692-a75d4125fba8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.100931    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-credential-socket" pod "bbcbf9e3-2774-4872-95d5-4f27b7ad17cc" (UID: "bbcbf9e3-2774-4872-95d5-4f27b7ad17cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.101183    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-istio-data\") pod \"565e1871-9784-4483-ad19-17413b47017e\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") "
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.101239    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8cc50b-9b04-4398-a692-a75d4125fba8-kube-api-access-9tqsd\" (UniqueName: \"kubernetes.io/projected/8c8cc50b-9b04-4398-a692-a75d4125fba8-kube-api-access-9tqsd\") pod \"8c8cc50b-9b04-4398-a692-a75d4125fba8\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") "
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.101308    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-credential-socket\") pod \"8c8cc50b-9b04-4398-a692-a75d4125fba8\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") "
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.101338    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-workload-certs\") pod \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") "
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.101384    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/565e1871-9784-4483-ad19-17413b47017e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565e1871-9784-4483-ad19-17413b47017e-istio-podinfo\") pod \"565e1871-9784-4483-ad19-17413b47017e\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") "
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.101425    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-data\") pod \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") "
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.101452    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/565e1871-9784-4483-ad19-17413b47017e-kube-api-access-q8tpr\" (UniqueName: \"kubernetes.io/projected/565e1871-9784-4483-ad19-17413b47017e-kube-api-access-q8tpr\") pod \"565e1871-9784-4483-ad19-17413b47017e\" (UID: \"565e1871-9784-4483-ad19-17413b47017e\") "
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.101484    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-podinfo\") pod \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") "
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.101519    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-data\") pod \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") "
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.101551    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-data\") pod \"8c8cc50b-9b04-4398-a692-a75d4125fba8\" (UID: \"8c8cc50b-9b04-4398-a692-a75d4125fba8\") "
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.101605    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-workload-socket\") pod \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") "
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.101639    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-kube-api-access-2tzkm\" (UniqueName: \"kubernetes.io/projected/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-kube-api-access-2tzkm\") pod \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") "
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.101669    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-envoy\") pod \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\" (UID: \"6af1e128-ecfb-4449-a2ab-0d6a3732ca71\") "
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.101697    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-kube-api-access-gc2qp\" (UniqueName: \"kubernetes.io/projected/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-kube-api-access-gc2qp\") pod \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") "
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.101734    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-envoy\") pod \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\" (UID: \"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc\") "
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.102252    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.102273    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8cc50b-9b04-4398-a692-a75d4125fba8-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.102290    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565e1871-9784-4483-ad19-17413b47017e-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.102306    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.102321    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.102339    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.102421    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.102438    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.102456    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.102471    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.102486    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.102671    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-token" pod "8c8cc50b-9b04-4398-a692-a75d4125fba8" (UID: "8c8cc50b-9b04-4398-a692-a75d4125fba8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:37.102789 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.102685    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-istio-envoy" pod "565e1871-9784-4483-ad19-17413b47017e" (UID: "565e1871-9784-4483-ad19-17413b47017e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:37.105061 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.102888    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-data" pod "8c8cc50b-9b04-4398-a692-a75d4125fba8" (UID: "8c8cc50b-9b04-4398-a692-a75d4125fba8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:37.105061 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.103068    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-data" pod "6af1e128-ecfb-4449-a2ab-0d6a3732ca71" (UID: "6af1e128-ecfb-4449-a2ab-0d6a3732ca71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:37.105061 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.103213    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-workload-socket" pod "6af1e128-ecfb-4449-a2ab-0d6a3732ca71" (UID: "6af1e128-ecfb-4449-a2ab-0d6a3732ca71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:37.105061 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.103262    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-workload-certs" pod "8c8cc50b-9b04-4398-a692-a75d4125fba8" (UID: "8c8cc50b-9b04-4398-a692-a75d4125fba8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:37.105061 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.103599    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-workload-certs" pod "bbcbf9e3-2774-4872-95d5-4f27b7ad17cc" (UID: "bbcbf9e3-2774-4872-95d5-4f27b7ad17cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:37.105061 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.103564    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-credential-socket" pod "8c8cc50b-9b04-4398-a692-a75d4125fba8" (UID: "8c8cc50b-9b04-4398-a692-a75d4125fba8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:37.105061 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.103668    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-data" pod "bbcbf9e3-2774-4872-95d5-4f27b7ad17cc" (UID: "bbcbf9e3-2774-4872-95d5-4f27b7ad17cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:37.105061 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.103667    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-workload-socket" pod "565e1871-9784-4483-ad19-17413b47017e" (UID: "565e1871-9784-4483-ad19-17413b47017e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:37.105061 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.103944    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-istio-data" pod "565e1871-9784-4483-ad19-17413b47017e" (UID: "565e1871-9784-4483-ad19-17413b47017e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:37.105575 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.105269    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-podinfo" pod "bbcbf9e3-2774-4872-95d5-4f27b7ad17cc" (UID: "bbcbf9e3-2774-4872-95d5-4f27b7ad17cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:20:37.105811 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.105783    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-podinfo" pod "6af1e128-ecfb-4449-a2ab-0d6a3732ca71" (UID: "6af1e128-ecfb-4449-a2ab-0d6a3732ca71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:20:37.106482 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.106447    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-kube-api-access-2tzkm" pod "6af1e128-ecfb-4449-a2ab-0d6a3732ca71" (UID: "6af1e128-ecfb-4449-a2ab-0d6a3732ca71"). InnerVolumeSpecName "kube-api-access-2tzkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:37.106482 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.106464    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-envoy" pod "6af1e128-ecfb-4449-a2ab-0d6a3732ca71" (UID: "6af1e128-ecfb-4449-a2ab-0d6a3732ca71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:37.106624 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.106492    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-podinfo" pod "8c8cc50b-9b04-4398-a692-a75d4125fba8" (UID: "8c8cc50b-9b04-4398-a692-a75d4125fba8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:20:37.106624 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.106595    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-token" pod "bbcbf9e3-2774-4872-95d5-4f27b7ad17cc" (UID: "bbcbf9e3-2774-4872-95d5-4f27b7ad17cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:37.106736 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.106695    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-envoy" pod "8c8cc50b-9b04-4398-a692-a75d4125fba8" (UID: "8c8cc50b-9b04-4398-a692-a75d4125fba8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:37.107820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.107792    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565e1871-9784-4483-ad19-17413b47017e-kube-api-access-q8tpr" pod "565e1871-9784-4483-ad19-17413b47017e" (UID: "565e1871-9784-4483-ad19-17413b47017e"). InnerVolumeSpecName "kube-api-access-q8tpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:37.107939 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.107829    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565e1871-9784-4483-ad19-17413b47017e-istio-token" pod "565e1871-9784-4483-ad19-17413b47017e" (UID: "565e1871-9784-4483-ad19-17413b47017e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:37.107939 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.107842    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-envoy" pod "bbcbf9e3-2774-4872-95d5-4f27b7ad17cc" (UID: "bbcbf9e3-2774-4872-95d5-4f27b7ad17cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:37.108058 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.108014    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-token" pod "6af1e128-ecfb-4449-a2ab-0d6a3732ca71" (UID: "6af1e128-ecfb-4449-a2ab-0d6a3732ca71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:37.108165 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.108149    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-kube-api-access-gc2qp" pod "bbcbf9e3-2774-4872-95d5-4f27b7ad17cc" (UID: "bbcbf9e3-2774-4872-95d5-4f27b7ad17cc"). InnerVolumeSpecName "kube-api-access-gc2qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:37.108322 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.108298    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/565e1871-9784-4483-ad19-17413b47017e-istio-podinfo" pod "565e1871-9784-4483-ad19-17413b47017e" (UID: "565e1871-9784-4483-ad19-17413b47017e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:20:37.108636 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.108619    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8cc50b-9b04-4398-a692-a75d4125fba8-kube-api-access-9tqsd" pod "8c8cc50b-9b04-4398-a692-a75d4125fba8" (UID: "8c8cc50b-9b04-4398-a692-a75d4125fba8"). InnerVolumeSpecName "kube-api-access-9tqsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:37.203793 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.203685    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.203793 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.203725    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.203793 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.203740    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tzkm\" (UniqueName: \"kubernetes.io/projected/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-kube-api-access-2tzkm\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.203793 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.203753    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.203793 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.203766    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc2qp\" (UniqueName: \"kubernetes.io/projected/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-kube-api-access-gc2qp\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.203793 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.203780    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.203793 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.203793    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.204236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.203807    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.204236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.203824    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.204236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.203836    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8cc50b-9b04-4398-a692-a75d4125fba8-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.204236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.203848    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.204236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.203860    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.204236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.203872    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565e1871-9784-4483-ad19-17413b47017e-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.204236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.203885    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.204236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.203898    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.204236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.203910    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.204236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.203922    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565e1871-9784-4483-ad19-17413b47017e-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.204236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.203935    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tqsd\" (UniqueName: \"kubernetes.io/projected/8c8cc50b-9b04-4398-a692-a75d4125fba8-kube-api-access-9tqsd\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.204236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.203951    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cc50b-9b04-4398-a692-a75d4125fba8-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.204236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.203966    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.204236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.203978    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565e1871-9784-4483-ad19-17413b47017e-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.204236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.203992    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.204236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.204007    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8tpr\" (UniqueName: \"kubernetes.io/projected/565e1871-9784-4483-ad19-17413b47017e-kube-api-access-q8tpr\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.204236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.204022    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.204236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.204039    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6af1e128-ecfb-4449-a2ab-0d6a3732ca71-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:37.600158 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.600069    6059 generic.go:356] "Generic (PLEG): container finished" podID="565e1871-9784-4483-ad19-17413b47017e" containerID="07beb1a50ed37b27bf44b14902c2205eeedce86545d9e24ae8ac98bbf768bb49" exitCode=0
Nov 03 18:20:37.600158 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.600138    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"
Nov 03 18:20:37.600369 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.600123    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j" event={"ID":"565e1871-9784-4483-ad19-17413b47017e","Type":"ContainerDied","Data":"07beb1a50ed37b27bf44b14902c2205eeedce86545d9e24ae8ac98bbf768bb49"}
Nov 03 18:20:37.600369 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.600335    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j" event={"ID":"565e1871-9784-4483-ad19-17413b47017e","Type":"ContainerDied","Data":"ef6df904df8df53fba1b1c4d85f2d99b4c6fdda6309de3a918ef6180749c09a0"}
Nov 03 18:20:37.600566 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.600383    6059 scope.go:122] "RemoveContainer" containerID="07beb1a50ed37b27bf44b14902c2205eeedce86545d9e24ae8ac98bbf768bb49"
Nov 03 18:20:37.602026 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.602002    6059 generic.go:356] "Generic (PLEG): container finished" podID="6af1e128-ecfb-4449-a2ab-0d6a3732ca71" containerID="8b96819365fa52902fa2c94775775b9586c3d3d17815337171acb8006bbccb06" exitCode=0
Nov 03 18:20:37.602180 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.602120    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"
Nov 03 18:20:37.602263 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.602124    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr" event={"ID":"6af1e128-ecfb-4449-a2ab-0d6a3732ca71","Type":"ContainerDied","Data":"8b96819365fa52902fa2c94775775b9586c3d3d17815337171acb8006bbccb06"}
Nov 03 18:20:37.602318 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.602286    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr" event={"ID":"6af1e128-ecfb-4449-a2ab-0d6a3732ca71","Type":"ContainerDied","Data":"bff2c8909dee02818829e9729e9a2ff184eccc294eb9f78505c178b1487169d3"}
Nov 03 18:20:37.604456 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.604181    6059 generic.go:356] "Generic (PLEG): container finished" podID="8c8cc50b-9b04-4398-a692-a75d4125fba8" containerID="904ba53e5789cc5d699261e4ba685e181143a96e15ec956dcd3082f9bdbca422" exitCode=0
Nov 03 18:20:37.604456 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.604262    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"
Nov 03 18:20:37.604456 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.604319    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk" event={"ID":"8c8cc50b-9b04-4398-a692-a75d4125fba8","Type":"ContainerDied","Data":"904ba53e5789cc5d699261e4ba685e181143a96e15ec956dcd3082f9bdbca422"}
Nov 03 18:20:37.605016 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.604992    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk" event={"ID":"8c8cc50b-9b04-4398-a692-a75d4125fba8","Type":"ContainerDied","Data":"8bf4d16abd2092c105bc18d56456a165cd8366639fdbd4a4926dffb94922e2db"}
Nov 03 18:20:37.605956 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.605931    6059 generic.go:356] "Generic (PLEG): container finished" podID="bbcbf9e3-2774-4872-95d5-4f27b7ad17cc" containerID="aacbd52827741c953232fcb544c82cb856e25476e3b584675e16631e8b7382b7" exitCode=0
Nov 03 18:20:37.606093 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.605994    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw" event={"ID":"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc","Type":"ContainerDied","Data":"aacbd52827741c953232fcb544c82cb856e25476e3b584675e16631e8b7382b7"}
Nov 03 18:20:37.606093 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.606028    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw" event={"ID":"bbcbf9e3-2774-4872-95d5-4f27b7ad17cc","Type":"ContainerDied","Data":"eae33fa97d3d1c4d15316ea9ac2900433f74b81af91f08701fde5390cb047918"}
Nov 03 18:20:37.606234 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.606111    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"
Nov 03 18:20:37.628034 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.627961    6059 scope.go:122] "RemoveContainer" containerID="07beb1a50ed37b27bf44b14902c2205eeedce86545d9e24ae8ac98bbf768bb49"
Nov 03 18:20:37.628394 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:20:37.628366    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07beb1a50ed37b27bf44b14902c2205eeedce86545d9e24ae8ac98bbf768bb49\": container with ID starting with 07beb1a50ed37b27bf44b14902c2205eeedce86545d9e24ae8ac98bbf768bb49 not found: ID does not exist" containerID="07beb1a50ed37b27bf44b14902c2205eeedce86545d9e24ae8ac98bbf768bb49"
Nov 03 18:20:37.628480 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.628409    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07beb1a50ed37b27bf44b14902c2205eeedce86545d9e24ae8ac98bbf768bb49"} err="failed to get container status \"07beb1a50ed37b27bf44b14902c2205eeedce86545d9e24ae8ac98bbf768bb49\": rpc error: code = NotFound desc = could not find container \"07beb1a50ed37b27bf44b14902c2205eeedce86545d9e24ae8ac98bbf768bb49\": container with ID starting with 07beb1a50ed37b27bf44b14902c2205eeedce86545d9e24ae8ac98bbf768bb49 not found: ID does not exist"
Nov 03 18:20:37.628480 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.628434    6059 scope.go:122] "RemoveContainer" containerID="8b96819365fa52902fa2c94775775b9586c3d3d17815337171acb8006bbccb06"
Nov 03 18:20:37.637412 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.637394    6059 scope.go:122] "RemoveContainer" containerID="8b96819365fa52902fa2c94775775b9586c3d3d17815337171acb8006bbccb06"
Nov 03 18:20:37.637734 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:20:37.637715    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b96819365fa52902fa2c94775775b9586c3d3d17815337171acb8006bbccb06\": container with ID starting with 8b96819365fa52902fa2c94775775b9586c3d3d17815337171acb8006bbccb06 not found: ID does not exist" containerID="8b96819365fa52902fa2c94775775b9586c3d3d17815337171acb8006bbccb06"
Nov 03 18:20:37.637791 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.637742    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b96819365fa52902fa2c94775775b9586c3d3d17815337171acb8006bbccb06"} err="failed to get container status \"8b96819365fa52902fa2c94775775b9586c3d3d17815337171acb8006bbccb06\": rpc error: code = NotFound desc = could not find container \"8b96819365fa52902fa2c94775775b9586c3d3d17815337171acb8006bbccb06\": container with ID starting with 8b96819365fa52902fa2c94775775b9586c3d3d17815337171acb8006bbccb06 not found: ID does not exist"
Nov 03 18:20:37.637791 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.637760    6059 scope.go:122] "RemoveContainer" containerID="904ba53e5789cc5d699261e4ba685e181143a96e15ec956dcd3082f9bdbca422"
Nov 03 18:20:37.646526 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.646506    6059 scope.go:122] "RemoveContainer" containerID="904ba53e5789cc5d699261e4ba685e181143a96e15ec956dcd3082f9bdbca422"
Nov 03 18:20:37.646833 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:20:37.646810    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"904ba53e5789cc5d699261e4ba685e181143a96e15ec956dcd3082f9bdbca422\": container with ID starting with 904ba53e5789cc5d699261e4ba685e181143a96e15ec956dcd3082f9bdbca422 not found: ID does not exist" containerID="904ba53e5789cc5d699261e4ba685e181143a96e15ec956dcd3082f9bdbca422"
Nov 03 18:20:37.646902 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.646842    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"904ba53e5789cc5d699261e4ba685e181143a96e15ec956dcd3082f9bdbca422"} err="failed to get container status \"904ba53e5789cc5d699261e4ba685e181143a96e15ec956dcd3082f9bdbca422\": rpc error: code = NotFound desc = could not find container \"904ba53e5789cc5d699261e4ba685e181143a96e15ec956dcd3082f9bdbca422\": container with ID starting with 904ba53e5789cc5d699261e4ba685e181143a96e15ec956dcd3082f9bdbca422 not found: ID does not exist"
Nov 03 18:20:37.646902 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.646861    6059 scope.go:122] "RemoveContainer" containerID="aacbd52827741c953232fcb544c82cb856e25476e3b584675e16631e8b7382b7"
Nov 03 18:20:37.656286 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.656264    6059 scope.go:122] "RemoveContainer" containerID="aacbd52827741c953232fcb544c82cb856e25476e3b584675e16631e8b7382b7"
Nov 03 18:20:37.656650 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:20:37.656627    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aacbd52827741c953232fcb544c82cb856e25476e3b584675e16631e8b7382b7\": container with ID starting with aacbd52827741c953232fcb544c82cb856e25476e3b584675e16631e8b7382b7 not found: ID does not exist" containerID="aacbd52827741c953232fcb544c82cb856e25476e3b584675e16631e8b7382b7"
Nov 03 18:20:37.656699 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.656658    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aacbd52827741c953232fcb544c82cb856e25476e3b584675e16631e8b7382b7"} err="failed to get container status \"aacbd52827741c953232fcb544c82cb856e25476e3b584675e16631e8b7382b7\": rpc error: code = NotFound desc = could not find container \"aacbd52827741c953232fcb544c82cb856e25476e3b584675e16631e8b7382b7\": container with ID starting with aacbd52827741c953232fcb544c82cb856e25476e3b584675e16631e8b7382b7 not found: ID does not exist"
Nov 03 18:20:37.776289 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.776253    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"]
Nov 03 18:20:37.781069 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.781041    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-2qd5j"]
Nov 03 18:20:37.898292 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.898209    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"]
Nov 03 18:20:37.910805 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:37.910775    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kjgtk"]
Nov 03 18:20:38.002780 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:38.002735    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:38.025454 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:38.025405    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"]
Nov 03 18:20:38.030315 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:38.030288    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6trbr"]
Nov 03 18:20:38.145055 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:38.145017    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"]
Nov 03 18:20:38.150744 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:38.150676    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8kglw"]
Nov 03 18:20:39.336295 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:39.336261    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="565e1871-9784-4483-ad19-17413b47017e" path="/var/lib/kubelet/pods/565e1871-9784-4483-ad19-17413b47017e/volumes"
Nov 03 18:20:39.336739 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:39.336725    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6af1e128-ecfb-4449-a2ab-0d6a3732ca71" path="/var/lib/kubelet/pods/6af1e128-ecfb-4449-a2ab-0d6a3732ca71/volumes"
Nov 03 18:20:39.337114 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:39.337103    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c8cc50b-9b04-4398-a692-a75d4125fba8" path="/var/lib/kubelet/pods/8c8cc50b-9b04-4398-a692-a75d4125fba8/volumes"
Nov 03 18:20:39.337810 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:39.337795    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbcbf9e3-2774-4872-95d5-4f27b7ad17cc" path="/var/lib/kubelet/pods/bbcbf9e3-2774-4872-95d5-4f27b7ad17cc/volumes"
Nov 03 18:20:42.857628 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:42.857590    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:43.045542 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:43.045511    6059 scope.go:122] "RemoveContainer" containerID="0d0130c4187bc93736064b94afd7acc376a4bf6c37c52496906583f67a0d32e1"
Nov 03 18:20:47.672574 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:47.672459    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:52.035874 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:52.035843    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:52.532795 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:52.532761    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:57.364427 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:57.364382    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:20:59.607834 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:20:59.607802    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:02.263578 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:02.263543    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:06.861391 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:21:06.861333    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:21:07.146607 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:07.146517    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:12.057942 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:12.057904    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:16.885482 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:16.885448    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:20.030585 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:20.030553    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:20.232191 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:20.232154    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:21.182163 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:21.182121    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:21.879765 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:21.879723    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:23.932265 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.932229    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"]
Nov 03 18:21:23.932739 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.932686    6059 cpu_manager.go:414] "containerMap: removing container" podUID="565e1871-9784-4483-ad19-17413b47017e" containerName="istio-proxy"
Nov 03 18:21:23.932739 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.932699    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="565e1871-9784-4483-ad19-17413b47017e" containerName="istio-proxy"
Nov 03 18:21:23.932739 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.932712    6059 cpu_manager.go:414] "containerMap: removing container" podUID="6af1e128-ecfb-4449-a2ab-0d6a3732ca71" containerName="istio-proxy"
Nov 03 18:21:23.932739 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.932717    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af1e128-ecfb-4449-a2ab-0d6a3732ca71" containerName="istio-proxy"
Nov 03 18:21:23.932739 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.932739    6059 cpu_manager.go:414] "containerMap: removing container" podUID="4fac721e-7ac1-48f0-8374-37ca3f687999" containerName="payload-processing"
Nov 03 18:21:23.932919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.932744    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fac721e-7ac1-48f0-8374-37ca3f687999" containerName="payload-processing"
Nov 03 18:21:23.932919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.932758    6059 cpu_manager.go:414] "containerMap: removing container" podUID="3f3ca86d-8a62-4cdf-976d-efcec688ed8a" containerName="payload-processing"
Nov 03 18:21:23.932919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.932763    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f3ca86d-8a62-4cdf-976d-efcec688ed8a" containerName="payload-processing"
Nov 03 18:21:23.932919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.932772    6059 cpu_manager.go:414] "containerMap: removing container" podUID="8c8cc50b-9b04-4398-a692-a75d4125fba8" containerName="istio-proxy"
Nov 03 18:21:23.932919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.932777    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c8cc50b-9b04-4398-a692-a75d4125fba8" containerName="istio-proxy"
Nov 03 18:21:23.932919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.932790    6059 cpu_manager.go:414] "containerMap: removing container" podUID="bbcbf9e3-2774-4872-95d5-4f27b7ad17cc" containerName="istio-proxy"
Nov 03 18:21:23.932919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.932794    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbcbf9e3-2774-4872-95d5-4f27b7ad17cc" containerName="istio-proxy"
Nov 03 18:21:23.932919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.932858    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="565e1871-9784-4483-ad19-17413b47017e" containerName="istio-proxy"
Nov 03 18:21:23.932919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.932865    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af1e128-ecfb-4449-a2ab-0d6a3732ca71" containerName="istio-proxy"
Nov 03 18:21:23.932919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.932875    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fac721e-7ac1-48f0-8374-37ca3f687999" containerName="payload-processing"
Nov 03 18:21:23.932919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.932883    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f3ca86d-8a62-4cdf-976d-efcec688ed8a" containerName="payload-processing"
Nov 03 18:21:23.932919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.932888    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c8cc50b-9b04-4398-a692-a75d4125fba8" containerName="istio-proxy"
Nov 03 18:21:23.932919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.932895    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbcbf9e3-2774-4872-95d5-4f27b7ad17cc" containerName="istio-proxy"
Nov 03 18:21:23.936216 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.936198    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"
Nov 03 18:21:23.955632 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.955591    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/450728f8-87df-4e5a-85fd-902d0ae81879-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb\" (UID: \"450728f8-87df-4e5a-85fd-902d0ae81879\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"
Nov 03 18:21:23.955792 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.955659    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp9qc\" (UniqueName: \"kubernetes.io/projected/450728f8-87df-4e5a-85fd-902d0ae81879-kube-api-access-fp9qc\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb\" (UID: \"450728f8-87df-4e5a-85fd-902d0ae81879\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"
Nov 03 18:21:23.955792 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.955766    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/450728f8-87df-4e5a-85fd-902d0ae81879-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb\" (UID: \"450728f8-87df-4e5a-85fd-902d0ae81879\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"
Nov 03 18:21:23.955873 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.955798    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450728f8-87df-4e5a-85fd-902d0ae81879-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb\" (UID: \"450728f8-87df-4e5a-85fd-902d0ae81879\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"
Nov 03 18:21:23.991943 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.991914    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-7afe86be4901\""
Nov 03 18:21:23.997958 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:23.997928    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"]
Nov 03 18:21:24.056478 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:24.056438    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/450728f8-87df-4e5a-85fd-902d0ae81879-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb\" (UID: \"450728f8-87df-4e5a-85fd-902d0ae81879\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"
Nov 03 18:21:24.056663 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:24.056526    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp9qc\" (UniqueName: \"kubernetes.io/projected/450728f8-87df-4e5a-85fd-902d0ae81879-kube-api-access-fp9qc\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb\" (UID: \"450728f8-87df-4e5a-85fd-902d0ae81879\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"
Nov 03 18:21:24.056663 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:24.056571    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/450728f8-87df-4e5a-85fd-902d0ae81879-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb\" (UID: \"450728f8-87df-4e5a-85fd-902d0ae81879\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"
Nov 03 18:21:24.056663 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:24.056592    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450728f8-87df-4e5a-85fd-902d0ae81879-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb\" (UID: \"450728f8-87df-4e5a-85fd-902d0ae81879\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"
Nov 03 18:21:24.057191 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:24.057165    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/450728f8-87df-4e5a-85fd-902d0ae81879-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb\" (UID: \"450728f8-87df-4e5a-85fd-902d0ae81879\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"
Nov 03 18:21:24.057318 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:24.057300    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450728f8-87df-4e5a-85fd-902d0ae81879-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb\" (UID: \"450728f8-87df-4e5a-85fd-902d0ae81879\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"
Nov 03 18:21:24.059868 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:24.059839    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/450728f8-87df-4e5a-85fd-902d0ae81879-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb\" (UID: \"450728f8-87df-4e5a-85fd-902d0ae81879\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"
Nov 03 18:21:24.119525 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:24.119495    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp9qc\" (UniqueName: \"kubernetes.io/projected/450728f8-87df-4e5a-85fd-902d0ae81879-kube-api-access-fp9qc\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb\" (UID: \"450728f8-87df-4e5a-85fd-902d0ae81879\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"
Nov 03 18:21:24.248951 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:24.248912    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"
Nov 03 18:21:24.626070 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:21:24.626016    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450728f8_87df_4e5a_85fd_902d0ae81879.slice/crio-04a4d47f860ad1e480508c2ed181646b387299535e7dd7fb717b3325d0bd434c WatchSource:0}: Error finding container 04a4d47f860ad1e480508c2ed181646b387299535e7dd7fb717b3325d0bd434c: Status 404 returned error can't find the container with id 04a4d47f860ad1e480508c2ed181646b387299535e7dd7fb717b3325d0bd434c
Nov 03 18:21:24.628050 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:24.628027    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"]
Nov 03 18:21:24.868484 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:24.868444    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb" event={"ID":"450728f8-87df-4e5a-85fd-902d0ae81879","Type":"ContainerStarted","Data":"fc31d63b157fb67a58b325f00fdcf5f66c868a34a444180a40040dc1c990e32e"}
Nov 03 18:21:24.868672 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:24.868495    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb" event={"ID":"450728f8-87df-4e5a-85fd-902d0ae81879","Type":"ContainerStarted","Data":"04a4d47f860ad1e480508c2ed181646b387299535e7dd7fb717b3325d0bd434c"}
Nov 03 18:21:24.868672 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:24.868591    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"
Nov 03 18:21:25.022433 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:25.022343    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb" podStartSLOduration=2.022324103 podStartE2EDuration="2.022324103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:21:25.008140028 +0000 UTC m=+3825.805853116" watchObservedRunningTime="2026-11-03 18:21:25.022324103 +0000 UTC m=+3825.820036884"
Nov 03 18:21:26.312746 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:26.312711    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"]
Nov 03 18:21:26.316722 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:26.316704    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"
Nov 03 18:21:26.371059 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:26.371024    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"agc-cred-4ecbbc60c14c\""
Nov 03 18:21:26.379418 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:26.379384    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af9cc25c-490d-4b26-ba22-54f5e945c495-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv\" (UID: \"af9cc25c-490d-4b26-ba22-54f5e945c495\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"
Nov 03 18:21:26.379608 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:26.379536    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af9cc25c-490d-4b26-ba22-54f5e945c495-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv\" (UID: \"af9cc25c-490d-4b26-ba22-54f5e945c495\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"
Nov 03 18:21:26.379608 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:26.379575    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2njb\" (UniqueName: \"kubernetes.io/projected/af9cc25c-490d-4b26-ba22-54f5e945c495-kube-api-access-t2njb\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv\" (UID: \"af9cc25c-490d-4b26-ba22-54f5e945c495\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"
Nov 03 18:21:26.379726 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:26.379690    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af9cc25c-490d-4b26-ba22-54f5e945c495-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv\" (UID: \"af9cc25c-490d-4b26-ba22-54f5e945c495\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"
Nov 03 18:21:26.424618 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:26.424583    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"]
Nov 03 18:21:26.480970 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:26.480929    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af9cc25c-490d-4b26-ba22-54f5e945c495-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv\" (UID: \"af9cc25c-490d-4b26-ba22-54f5e945c495\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"
Nov 03 18:21:26.481184 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:26.481010    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af9cc25c-490d-4b26-ba22-54f5e945c495-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv\" (UID: \"af9cc25c-490d-4b26-ba22-54f5e945c495\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"
Nov 03 18:21:26.481184 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:26.481031    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2njb\" (UniqueName: \"kubernetes.io/projected/af9cc25c-490d-4b26-ba22-54f5e945c495-kube-api-access-t2njb\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv\" (UID: \"af9cc25c-490d-4b26-ba22-54f5e945c495\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"
Nov 03 18:21:26.481184 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:26.481057    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af9cc25c-490d-4b26-ba22-54f5e945c495-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv\" (UID: \"af9cc25c-490d-4b26-ba22-54f5e945c495\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"
Nov 03 18:21:26.481771 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:26.481746    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af9cc25c-490d-4b26-ba22-54f5e945c495-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv\" (UID: \"af9cc25c-490d-4b26-ba22-54f5e945c495\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"
Nov 03 18:21:26.481865 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:26.481784    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af9cc25c-490d-4b26-ba22-54f5e945c495-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv\" (UID: \"af9cc25c-490d-4b26-ba22-54f5e945c495\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"
Nov 03 18:21:26.483609 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:26.483592    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af9cc25c-490d-4b26-ba22-54f5e945c495-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv\" (UID: \"af9cc25c-490d-4b26-ba22-54f5e945c495\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"
Nov 03 18:21:26.544450 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:26.544417    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2njb\" (UniqueName: \"kubernetes.io/projected/af9cc25c-490d-4b26-ba22-54f5e945c495-kube-api-access-t2njb\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv\" (UID: \"af9cc25c-490d-4b26-ba22-54f5e945c495\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"
Nov 03 18:21:26.630712 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:26.630622    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"
Nov 03 18:21:26.811534 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:26.811492    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:27.000986 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:21:27.000951    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9cc25c_490d_4b26_ba22_54f5e945c495.slice/crio-3004159c57e15998f42eb2cf5531fdf19003183e67376e8e64e3fb87c22195e9 WatchSource:0}: Error finding container 3004159c57e15998f42eb2cf5531fdf19003183e67376e8e64e3fb87c22195e9: Status 404 returned error can't find the container with id 3004159c57e15998f42eb2cf5531fdf19003183e67376e8e64e3fb87c22195e9
Nov 03 18:21:27.003526 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:27.003488    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"]
Nov 03 18:21:27.885868 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:27.885831    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv" event={"ID":"af9cc25c-490d-4b26-ba22-54f5e945c495","Type":"ContainerStarted","Data":"08fd0eca4f07b9682466f5d8db2b7388669fe2f6c5b87645e747a8a76cab5c32"}
Nov 03 18:21:27.886276 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:27.885889    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv" event={"ID":"af9cc25c-490d-4b26-ba22-54f5e945c495","Type":"ContainerStarted","Data":"3004159c57e15998f42eb2cf5531fdf19003183e67376e8e64e3fb87c22195e9"}
Nov 03 18:21:27.886276 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:27.885950    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"
Nov 03 18:21:28.039166 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:28.039104    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv" podStartSLOduration=2.039087793 podStartE2EDuration="2.039087793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:21:28.038403695 +0000 UTC m=+3828.836116476" watchObservedRunningTime="2026-11-03 18:21:28.039087793 +0000 UTC m=+3828.836800576"
Nov 03 18:21:31.664577 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:31.664521    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:31.987380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:31.987309    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:32.181861 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:32.181818    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml"]
Nov 03 18:21:32.186026 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:32.186001    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml"
Nov 03 18:21:32.187612 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:32.187585    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:32.238581 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:32.238492    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7mdj\" (UniqueName: \"kubernetes.io/projected/22ce7784-efe4-4ad1-b567-ecd69149f2f3-kube-api-access-m7mdj\") pod \"e2e-usage-sim-70698c-5f646997c4-dx8ml\" (UID: \"22ce7784-efe4-4ad1-b567-ecd69149f2f3\") " pod="e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml"
Nov 03 18:21:32.238764 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:32.238739    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/22ce7784-efe4-4ad1-b567-ecd69149f2f3-tls\") pod \"e2e-usage-sim-70698c-5f646997c4-dx8ml\" (UID: \"22ce7784-efe4-4ad1-b567-ecd69149f2f3\") " pod="e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml"
Nov 03 18:21:32.242568 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:32.242401    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-82ab32\"/\"kube-root-ca.crt\""
Nov 03 18:21:32.242744 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:32.242719    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-82ab32\"/\"openshift-service-ca.crt\""
Nov 03 18:21:32.242886 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:32.242740    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-82ab32\"/\"default-dockercfg-w2nm9\""
Nov 03 18:21:32.244471 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:32.244451    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml"]
Nov 03 18:21:32.294053 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:32.294019    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-82ab32\"/\"e2e-usage-sim-70698c-cert\""
Nov 03 18:21:32.340057 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:32.340025    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7mdj\" (UniqueName: \"kubernetes.io/projected/22ce7784-efe4-4ad1-b567-ecd69149f2f3-kube-api-access-m7mdj\") pod \"e2e-usage-sim-70698c-5f646997c4-dx8ml\" (UID: \"22ce7784-efe4-4ad1-b567-ecd69149f2f3\") " pod="e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml"
Nov 03 18:21:32.340243 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:32.340172    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/22ce7784-efe4-4ad1-b567-ecd69149f2f3-tls\") pod \"e2e-usage-sim-70698c-5f646997c4-dx8ml\" (UID: \"22ce7784-efe4-4ad1-b567-ecd69149f2f3\") " pod="e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml"
Nov 03 18:21:32.342822 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:32.342790    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/22ce7784-efe4-4ad1-b567-ecd69149f2f3-tls\") pod \"e2e-usage-sim-70698c-5f646997c4-dx8ml\" (UID: \"22ce7784-efe4-4ad1-b567-ecd69149f2f3\") " pod="e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml"
Nov 03 18:21:32.403075 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:32.403038    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7mdj\" (UniqueName: \"kubernetes.io/projected/22ce7784-efe4-4ad1-b567-ecd69149f2f3-kube-api-access-m7mdj\") pod \"e2e-usage-sim-70698c-5f646997c4-dx8ml\" (UID: \"22ce7784-efe4-4ad1-b567-ecd69149f2f3\") " pod="e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml"
Nov 03 18:21:32.498929 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:32.498835    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml"
Nov 03 18:21:32.875372 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:32.875337    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml"]
Nov 03 18:21:32.913687 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:32.913651    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml" event={"ID":"22ce7784-efe4-4ad1-b567-ecd69149f2f3","Type":"ContainerStarted","Data":"685a414a931335b1b3c27914efcb81450d944421ba1a39776a0508c72e359696"}
Nov 03 18:21:33.921663 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:33.921626    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml" event={"ID":"22ce7784-efe4-4ad1-b567-ecd69149f2f3","Type":"ContainerStarted","Data":"a10ebdc7479cc7340aed0bce0534217af2804e50372c51286b4c1c419538ede3"}
Nov 03 18:21:33.922168 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:33.921821    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml"
Nov 03 18:21:33.926736 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:33.926710    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml"
Nov 03 18:21:34.042325 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:34.042272    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml" podStartSLOduration=2.042258426 podStartE2EDuration="2.042258426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:21:34.041411982 +0000 UTC m=+3834.839124767" watchObservedRunningTime="2026-11-03 18:21:34.042258426 +0000 UTC m=+3834.839971209"
Nov 03 18:21:35.877407 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:35.877374    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"
Nov 03 18:21:36.069343 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:36.069309    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"]
Nov 03 18:21:36.562026 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:36.561987    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:36.862695 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:21:36.862604    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:21:38.894073 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:38.894042    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"
Nov 03 18:21:39.068815 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:39.068780    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"]
Nov 03 18:21:41.070506 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:41.070427    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84" podUID="21f21323-a8dd-4723-971e-cfca12f31f74" containerName="payload-processing" containerID="cri-o://c3e466a9a53712dc03ab08a979d48f3fd7bc7f2e9b8c1a97adcd2d5a38022fb0" gracePeriod=25
Nov 03 18:21:41.336341 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:41.332296    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"
Nov 03 18:21:41.441747 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:41.441718    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21f21323-a8dd-4723-971e-cfca12f31f74-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21f21323-a8dd-4723-971e-cfca12f31f74-routing-overlay-volume\") pod \"21f21323-a8dd-4723-971e-cfca12f31f74\" (UID: \"21f21323-a8dd-4723-971e-cfca12f31f74\") "
Nov 03 18:21:41.441953 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:41.441828    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21f21323-a8dd-4723-971e-cfca12f31f74-kube-api-access-cdprz\" (UniqueName: \"kubernetes.io/projected/21f21323-a8dd-4723-971e-cfca12f31f74-kube-api-access-cdprz\") pod \"21f21323-a8dd-4723-971e-cfca12f31f74\" (UID: \"21f21323-a8dd-4723-971e-cfca12f31f74\") "
Nov 03 18:21:41.441953 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:41.441864    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21f21323-a8dd-4723-971e-cfca12f31f74-provider-credentials\" (UniqueName: \"kubernetes.io/projected/21f21323-a8dd-4723-971e-cfca12f31f74-provider-credentials\") pod \"21f21323-a8dd-4723-971e-cfca12f31f74\" (UID: \"21f21323-a8dd-4723-971e-cfca12f31f74\") "
Nov 03 18:21:41.441953 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:41.441884    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21f21323-a8dd-4723-971e-cfca12f31f74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21f21323-a8dd-4723-971e-cfca12f31f74-plugins-config-volume\") pod \"21f21323-a8dd-4723-971e-cfca12f31f74\" (UID: \"21f21323-a8dd-4723-971e-cfca12f31f74\") "
Nov 03 18:21:41.442140 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:41.442075    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f21323-a8dd-4723-971e-cfca12f31f74-routing-overlay-volume" pod "21f21323-a8dd-4723-971e-cfca12f31f74" (UID: "21f21323-a8dd-4723-971e-cfca12f31f74"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:21:41.442348 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:41.442327    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f21323-a8dd-4723-971e-cfca12f31f74-plugins-config-volume" pod "21f21323-a8dd-4723-971e-cfca12f31f74" (UID: "21f21323-a8dd-4723-971e-cfca12f31f74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:21:41.443888 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:41.443864    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f21323-a8dd-4723-971e-cfca12f31f74-kube-api-access-cdprz" pod "21f21323-a8dd-4723-971e-cfca12f31f74" (UID: "21f21323-a8dd-4723-971e-cfca12f31f74"). InnerVolumeSpecName "kube-api-access-cdprz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:21:41.443997 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:41.443945    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f21323-a8dd-4723-971e-cfca12f31f74-provider-credentials" pod "21f21323-a8dd-4723-971e-cfca12f31f74" (UID: "21f21323-a8dd-4723-971e-cfca12f31f74"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:21:41.543423 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:41.543391    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21f21323-a8dd-4723-971e-cfca12f31f74-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:21:41.543423 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:41.543419    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdprz\" (UniqueName: \"kubernetes.io/projected/21f21323-a8dd-4723-971e-cfca12f31f74-kube-api-access-cdprz\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:21:41.543423 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:41.543429    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/21f21323-a8dd-4723-971e-cfca12f31f74-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:21:41.543677 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:41.543437    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21f21323-a8dd-4723-971e-cfca12f31f74-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:21:41.580931 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:41.580901    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:41.967880 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:41.967836    6059 generic.go:356] "Generic (PLEG): container finished" podID="21f21323-a8dd-4723-971e-cfca12f31f74" containerID="c3e466a9a53712dc03ab08a979d48f3fd7bc7f2e9b8c1a97adcd2d5a38022fb0" exitCode=0
Nov 03 18:21:41.968108 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:41.967904    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"
Nov 03 18:21:41.968108 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:41.967916    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84" event={"ID":"21f21323-a8dd-4723-971e-cfca12f31f74","Type":"ContainerDied","Data":"c3e466a9a53712dc03ab08a979d48f3fd7bc7f2e9b8c1a97adcd2d5a38022fb0"}
Nov 03 18:21:41.968108 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:41.967969    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84" event={"ID":"21f21323-a8dd-4723-971e-cfca12f31f74","Type":"ContainerDied","Data":"4c3f0b513b905858f1d6acde3a463f7b731fe3c73f98e1b276c44cd95464ee6c"}
Nov 03 18:21:41.968108 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:41.967993    6059 scope.go:122] "RemoveContainer" containerID="c3e466a9a53712dc03ab08a979d48f3fd7bc7f2e9b8c1a97adcd2d5a38022fb0"
Nov 03 18:21:41.980070 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:41.980009    6059 scope.go:122] "RemoveContainer" containerID="c3e466a9a53712dc03ab08a979d48f3fd7bc7f2e9b8c1a97adcd2d5a38022fb0"
Nov 03 18:21:41.980350 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:21:41.980330    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e466a9a53712dc03ab08a979d48f3fd7bc7f2e9b8c1a97adcd2d5a38022fb0\": container with ID starting with c3e466a9a53712dc03ab08a979d48f3fd7bc7f2e9b8c1a97adcd2d5a38022fb0 not found: ID does not exist" containerID="c3e466a9a53712dc03ab08a979d48f3fd7bc7f2e9b8c1a97adcd2d5a38022fb0"
Nov 03 18:21:41.980442 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:41.980373    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e466a9a53712dc03ab08a979d48f3fd7bc7f2e9b8c1a97adcd2d5a38022fb0"} err="failed to get container status \"c3e466a9a53712dc03ab08a979d48f3fd7bc7f2e9b8c1a97adcd2d5a38022fb0\": rpc error: code = NotFound desc = could not find container \"c3e466a9a53712dc03ab08a979d48f3fd7bc7f2e9b8c1a97adcd2d5a38022fb0\": container with ID starting with c3e466a9a53712dc03ab08a979d48f3fd7bc7f2e9b8c1a97adcd2d5a38022fb0 not found: ID does not exist"
Nov 03 18:21:42.160865 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:42.160829    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"]
Nov 03 18:21:42.212833 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:42.212801    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-65879xx84"]
Nov 03 18:21:42.321957 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:42.321874    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:42.512908 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:42.512874    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:43.335159 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:43.335122    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21f21323-a8dd-4723-971e-cfca12f31f74" path="/var/lib/kubelet/pods/21f21323-a8dd-4723-971e-cfca12f31f74/volumes"
Nov 03 18:21:44.069122 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:44.069064    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd" podUID="d58b44cf-6725-4898-a0a0-94707f9f397e" containerName="payload-processing" containerID="cri-o://6da5f37ff3ba4474e46bd732c0b9281119e30a9fa7da9658be3fbc69bfb3d726" gracePeriod=25
Nov 03 18:21:44.321107 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:44.321049    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"
Nov 03 18:21:44.473946 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:44.473912    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d58b44cf-6725-4898-a0a0-94707f9f397e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d58b44cf-6725-4898-a0a0-94707f9f397e-plugins-config-volume\") pod \"d58b44cf-6725-4898-a0a0-94707f9f397e\" (UID: \"d58b44cf-6725-4898-a0a0-94707f9f397e\") "
Nov 03 18:21:44.474448 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:44.473978    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d58b44cf-6725-4898-a0a0-94707f9f397e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d58b44cf-6725-4898-a0a0-94707f9f397e-provider-credentials\") pod \"d58b44cf-6725-4898-a0a0-94707f9f397e\" (UID: \"d58b44cf-6725-4898-a0a0-94707f9f397e\") "
Nov 03 18:21:44.474448 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:44.474006    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d58b44cf-6725-4898-a0a0-94707f9f397e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d58b44cf-6725-4898-a0a0-94707f9f397e-routing-overlay-volume\") pod \"d58b44cf-6725-4898-a0a0-94707f9f397e\" (UID: \"d58b44cf-6725-4898-a0a0-94707f9f397e\") "
Nov 03 18:21:44.474448 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:44.474034    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d58b44cf-6725-4898-a0a0-94707f9f397e-kube-api-access-skdff\" (UniqueName: \"kubernetes.io/projected/d58b44cf-6725-4898-a0a0-94707f9f397e-kube-api-access-skdff\") pod \"d58b44cf-6725-4898-a0a0-94707f9f397e\" (UID: \"d58b44cf-6725-4898-a0a0-94707f9f397e\") "
Nov 03 18:21:44.474448 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:44.474397    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d58b44cf-6725-4898-a0a0-94707f9f397e-routing-overlay-volume" pod "d58b44cf-6725-4898-a0a0-94707f9f397e" (UID: "d58b44cf-6725-4898-a0a0-94707f9f397e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:21:44.474635 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:44.474455    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d58b44cf-6725-4898-a0a0-94707f9f397e-plugins-config-volume" pod "d58b44cf-6725-4898-a0a0-94707f9f397e" (UID: "d58b44cf-6725-4898-a0a0-94707f9f397e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:21:44.476193 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:44.476166    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d58b44cf-6725-4898-a0a0-94707f9f397e-kube-api-access-skdff" pod "d58b44cf-6725-4898-a0a0-94707f9f397e" (UID: "d58b44cf-6725-4898-a0a0-94707f9f397e"). InnerVolumeSpecName "kube-api-access-skdff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:21:44.476294 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:44.476217    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d58b44cf-6725-4898-a0a0-94707f9f397e-provider-credentials" pod "d58b44cf-6725-4898-a0a0-94707f9f397e" (UID: "d58b44cf-6725-4898-a0a0-94707f9f397e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:21:44.575982 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:44.575900    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d58b44cf-6725-4898-a0a0-94707f9f397e-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:21:44.575982 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:44.575930    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d58b44cf-6725-4898-a0a0-94707f9f397e-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:21:44.575982 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:44.575941    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d58b44cf-6725-4898-a0a0-94707f9f397e-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:21:44.575982 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:44.575950    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skdff\" (UniqueName: \"kubernetes.io/projected/d58b44cf-6725-4898-a0a0-94707f9f397e-kube-api-access-skdff\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:21:44.986941 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:44.986908    6059 generic.go:356] "Generic (PLEG): container finished" podID="d58b44cf-6725-4898-a0a0-94707f9f397e" containerID="6da5f37ff3ba4474e46bd732c0b9281119e30a9fa7da9658be3fbc69bfb3d726" exitCode=0
Nov 03 18:21:44.987129 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:44.986958    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"
Nov 03 18:21:44.987129 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:44.986987    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd" event={"ID":"d58b44cf-6725-4898-a0a0-94707f9f397e","Type":"ContainerDied","Data":"6da5f37ff3ba4474e46bd732c0b9281119e30a9fa7da9658be3fbc69bfb3d726"}
Nov 03 18:21:44.987129 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:44.987040    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd" event={"ID":"d58b44cf-6725-4898-a0a0-94707f9f397e","Type":"ContainerDied","Data":"d69098dde965c716a7eea84fbabc3d13d71f468c0a2853d91e19fe711df0c66c"}
Nov 03 18:21:44.987129 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:44.987061    6059 scope.go:122] "RemoveContainer" containerID="6da5f37ff3ba4474e46bd732c0b9281119e30a9fa7da9658be3fbc69bfb3d726"
Nov 03 18:21:44.996773 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:44.996748    6059 scope.go:122] "RemoveContainer" containerID="6da5f37ff3ba4474e46bd732c0b9281119e30a9fa7da9658be3fbc69bfb3d726"
Nov 03 18:21:44.997087 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:21:44.997068    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da5f37ff3ba4474e46bd732c0b9281119e30a9fa7da9658be3fbc69bfb3d726\": container with ID starting with 6da5f37ff3ba4474e46bd732c0b9281119e30a9fa7da9658be3fbc69bfb3d726 not found: ID does not exist" containerID="6da5f37ff3ba4474e46bd732c0b9281119e30a9fa7da9658be3fbc69bfb3d726"
Nov 03 18:21:44.997165 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:44.997096    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da5f37ff3ba4474e46bd732c0b9281119e30a9fa7da9658be3fbc69bfb3d726"} err="failed to get container status \"6da5f37ff3ba4474e46bd732c0b9281119e30a9fa7da9658be3fbc69bfb3d726\": rpc error: code = NotFound desc = could not find container \"6da5f37ff3ba4474e46bd732c0b9281119e30a9fa7da9658be3fbc69bfb3d726\": container with ID starting with 6da5f37ff3ba4474e46bd732c0b9281119e30a9fa7da9658be3fbc69bfb3d726 not found: ID does not exist"
Nov 03 18:21:45.159465 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.159429    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"]
Nov 03 18:21:45.210933 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.210899    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-57597kfpd"]
Nov 03 18:21:45.335540 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.335464    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d58b44cf-6725-4898-a0a0-94707f9f397e" path="/var/lib/kubelet/pods/d58b44cf-6725-4898-a0a0-94707f9f397e/volumes"
Nov 03 18:21:45.537431 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.537398    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"]
Nov 03 18:21:45.537831 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.537818    6059 cpu_manager.go:414] "containerMap: removing container" podUID="21f21323-a8dd-4723-971e-cfca12f31f74" containerName="payload-processing"
Nov 03 18:21:45.537877 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.537832    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21f21323-a8dd-4723-971e-cfca12f31f74" containerName="payload-processing"
Nov 03 18:21:45.537877 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.537853    6059 cpu_manager.go:414] "containerMap: removing container" podUID="d58b44cf-6725-4898-a0a0-94707f9f397e" containerName="payload-processing"
Nov 03 18:21:45.537877 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.537859    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d58b44cf-6725-4898-a0a0-94707f9f397e" containerName="payload-processing"
Nov 03 18:21:45.537978 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.537941    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="21f21323-a8dd-4723-971e-cfca12f31f74" containerName="payload-processing"
Nov 03 18:21:45.537978 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.537954    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d58b44cf-6725-4898-a0a0-94707f9f397e" containerName="payload-processing"
Nov 03 18:21:45.542612 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.542593    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"
Nov 03 18:21:45.599061 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.598999    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"agc-cred-bc0ef1b15e38\""
Nov 03 18:21:45.602153 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.602129    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"]
Nov 03 18:21:45.686701 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.686661    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thnzn\" (UniqueName: \"kubernetes.io/projected/d3d58540-7d6f-459b-bcc5-e21804a1085b-kube-api-access-thnzn\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2\" (UID: \"d3d58540-7d6f-459b-bcc5-e21804a1085b\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"
Nov 03 18:21:45.686914 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.686716    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3d58540-7d6f-459b-bcc5-e21804a1085b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2\" (UID: \"d3d58540-7d6f-459b-bcc5-e21804a1085b\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"
Nov 03 18:21:45.686914 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.686788    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d58540-7d6f-459b-bcc5-e21804a1085b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2\" (UID: \"d3d58540-7d6f-459b-bcc5-e21804a1085b\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"
Nov 03 18:21:45.687047 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.687005    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3d58540-7d6f-459b-bcc5-e21804a1085b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2\" (UID: \"d3d58540-7d6f-459b-bcc5-e21804a1085b\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"
Nov 03 18:21:45.788398 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.788348    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3d58540-7d6f-459b-bcc5-e21804a1085b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2\" (UID: \"d3d58540-7d6f-459b-bcc5-e21804a1085b\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"
Nov 03 18:21:45.788578 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.788423    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thnzn\" (UniqueName: \"kubernetes.io/projected/d3d58540-7d6f-459b-bcc5-e21804a1085b-kube-api-access-thnzn\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2\" (UID: \"d3d58540-7d6f-459b-bcc5-e21804a1085b\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"
Nov 03 18:21:45.788578 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.788470    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3d58540-7d6f-459b-bcc5-e21804a1085b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2\" (UID: \"d3d58540-7d6f-459b-bcc5-e21804a1085b\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"
Nov 03 18:21:45.788578 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.788495    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d58540-7d6f-459b-bcc5-e21804a1085b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2\" (UID: \"d3d58540-7d6f-459b-bcc5-e21804a1085b\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"
Nov 03 18:21:45.789285 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.789256    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3d58540-7d6f-459b-bcc5-e21804a1085b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2\" (UID: \"d3d58540-7d6f-459b-bcc5-e21804a1085b\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"
Nov 03 18:21:45.789367 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.789339    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d58540-7d6f-459b-bcc5-e21804a1085b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2\" (UID: \"d3d58540-7d6f-459b-bcc5-e21804a1085b\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"
Nov 03 18:21:45.790885 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.790865    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3d58540-7d6f-459b-bcc5-e21804a1085b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2\" (UID: \"d3d58540-7d6f-459b-bcc5-e21804a1085b\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"
Nov 03 18:21:45.851846 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.851771    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thnzn\" (UniqueName: \"kubernetes.io/projected/d3d58540-7d6f-459b-bcc5-e21804a1085b-kube-api-access-thnzn\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2\" (UID: \"d3d58540-7d6f-459b-bcc5-e21804a1085b\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"
Nov 03 18:21:45.856580 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:45.856546    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"
Nov 03 18:21:46.227813 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:21:46.227786    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d58540_7d6f_459b_bcc5_e21804a1085b.slice/crio-5d900d1e9f7af8b45b1b80d7f9a62425cc40fc1a3185895b69c12d56f540c164 WatchSource:0}: Error finding container 5d900d1e9f7af8b45b1b80d7f9a62425cc40fc1a3185895b69c12d56f540c164: Status 404 returned error can't find the container with id 5d900d1e9f7af8b45b1b80d7f9a62425cc40fc1a3185895b69c12d56f540c164
Nov 03 18:21:46.229919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:46.229896    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"]
Nov 03 18:21:46.463018 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:46.462977    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:47.000449 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:47.000405    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2" event={"ID":"d3d58540-7d6f-459b-bcc5-e21804a1085b","Type":"ContainerStarted","Data":"d8dce6c0533e6ffe62ed5d68122bf313a776dac706a268dbddb1ed1b79e9859b"}
Nov 03 18:21:47.000920 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:47.000462    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2" event={"ID":"d3d58540-7d6f-459b-bcc5-e21804a1085b","Type":"ContainerStarted","Data":"5d900d1e9f7af8b45b1b80d7f9a62425cc40fc1a3185895b69c12d56f540c164"}
Nov 03 18:21:47.000920 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:47.000602    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"
Nov 03 18:21:47.128804 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:47.128746    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2" podStartSLOduration=2.128729542 podStartE2EDuration="2.128729542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:21:47.127981117 +0000 UTC m=+3847.925693903" watchObservedRunningTime="2026-11-03 18:21:47.128729542 +0000 UTC m=+3847.926442324"
Nov 03 18:21:47.920002 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:47.919959    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"]
Nov 03 18:21:47.923938 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:47.923914    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"
Nov 03 18:21:47.978001 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:47.977975    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-6244ce06c11a\""
Nov 03 18:21:48.010312 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:48.010276    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2\" (UID: \"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"
Nov 03 18:21:48.010312 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:48.010314    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2\" (UID: \"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"
Nov 03 18:21:48.010740 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:48.010442    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2\" (UID: \"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"
Nov 03 18:21:48.010740 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:48.010491    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7xjm\" (UniqueName: \"kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-kube-api-access-q7xjm\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2\" (UID: \"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"
Nov 03 18:21:48.031445 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:48.031415    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"]
Nov 03 18:21:48.111095 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:48.111045    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2\" (UID: \"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"
Nov 03 18:21:48.111095 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:48.111093    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2\" (UID: \"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"
Nov 03 18:21:48.111395 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:48.111163    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2\" (UID: \"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"
Nov 03 18:21:48.111395 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:48.111340    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7xjm\" (UniqueName: \"kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-kube-api-access-q7xjm\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2\" (UID: \"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"
Nov 03 18:21:48.111921 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:48.111885    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2\" (UID: \"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"
Nov 03 18:21:48.112046 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:48.111981    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2\" (UID: \"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"
Nov 03 18:21:48.113768 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:48.113749    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2\" (UID: \"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"
Nov 03 18:21:48.174575 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:48.174498    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7xjm\" (UniqueName: \"kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-kube-api-access-q7xjm\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2\" (UID: \"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"
Nov 03 18:21:48.237711 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:48.237680    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"
Nov 03 18:21:48.607676 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:21:48.607631    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce33491c_941e_44dd_a4ad_d8ca6a3dbbe6.slice/crio-0b980be967a90ec5b668b3e2bc31f9d75bbb2a3f6427bef91ef2d581a5bbddda WatchSource:0}: Error finding container 0b980be967a90ec5b668b3e2bc31f9d75bbb2a3f6427bef91ef2d581a5bbddda: Status 404 returned error can't find the container with id 0b980be967a90ec5b668b3e2bc31f9d75bbb2a3f6427bef91ef2d581a5bbddda
Nov 03 18:21:48.609852 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:48.609828    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"]
Nov 03 18:21:49.013421 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:49.013386    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2" event={"ID":"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6","Type":"ContainerStarted","Data":"ffa91730e1e7f2f8f83672dc60b3a8e9275b02e197c9c131191222308dd987cb"}
Nov 03 18:21:49.013802 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:49.013446    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2" event={"ID":"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6","Type":"ContainerStarted","Data":"0b980be967a90ec5b668b3e2bc31f9d75bbb2a3f6427bef91ef2d581a5bbddda"}
Nov 03 18:21:49.013802 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:49.013561    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"
Nov 03 18:21:49.139325 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:49.139273    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2" podStartSLOduration=2.139259166 podStartE2EDuration="2.139259166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:21:49.138268491 +0000 UTC m=+3849.935981272" watchObservedRunningTime="2026-11-03 18:21:49.139259166 +0000 UTC m=+3849.936971946"
Nov 03 18:21:51.263584 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:51.263552    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:52.385974 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:52.385937    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:56.300638 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:56.300599    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:21:58.010152 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:58.010122    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"
Nov 03 18:21:58.189432 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:21:58.189395    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"]
Nov 03 18:22:00.022783 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:00.022751    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"
Nov 03 18:22:00.198812 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:00.198772    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"]
Nov 03 18:22:01.161011 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:01.160976    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:03.190531 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:03.190488    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv" podUID="af9cc25c-490d-4b26-ba22-54f5e945c495" containerName="payload-processing" containerID="cri-o://08fd0eca4f07b9682466f5d8db2b7388669fe2f6c5b87645e747a8a76cab5c32" gracePeriod=25
Nov 03 18:22:03.432891 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:03.432859    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:03.447881 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:03.447822    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"
Nov 03 18:22:03.465157 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:03.465127    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af9cc25c-490d-4b26-ba22-54f5e945c495-kube-api-access-t2njb\" (UniqueName: \"kubernetes.io/projected/af9cc25c-490d-4b26-ba22-54f5e945c495-kube-api-access-t2njb\") pod \"af9cc25c-490d-4b26-ba22-54f5e945c495\" (UID: \"af9cc25c-490d-4b26-ba22-54f5e945c495\") "
Nov 03 18:22:03.465267 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:03.465171    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af9cc25c-490d-4b26-ba22-54f5e945c495-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af9cc25c-490d-4b26-ba22-54f5e945c495-plugins-config-volume\") pod \"af9cc25c-490d-4b26-ba22-54f5e945c495\" (UID: \"af9cc25c-490d-4b26-ba22-54f5e945c495\") "
Nov 03 18:22:03.465267 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:03.465213    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af9cc25c-490d-4b26-ba22-54f5e945c495-provider-credentials\" (UniqueName: \"kubernetes.io/projected/af9cc25c-490d-4b26-ba22-54f5e945c495-provider-credentials\") pod \"af9cc25c-490d-4b26-ba22-54f5e945c495\" (UID: \"af9cc25c-490d-4b26-ba22-54f5e945c495\") "
Nov 03 18:22:03.465422 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:03.465280    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af9cc25c-490d-4b26-ba22-54f5e945c495-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af9cc25c-490d-4b26-ba22-54f5e945c495-routing-overlay-volume\") pod \"af9cc25c-490d-4b26-ba22-54f5e945c495\" (UID: \"af9cc25c-490d-4b26-ba22-54f5e945c495\") "
Nov 03 18:22:03.465610 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:03.465585    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af9cc25c-490d-4b26-ba22-54f5e945c495-routing-overlay-volume" pod "af9cc25c-490d-4b26-ba22-54f5e945c495" (UID: "af9cc25c-490d-4b26-ba22-54f5e945c495"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:22:03.465701 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:03.465679    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af9cc25c-490d-4b26-ba22-54f5e945c495-plugins-config-volume" pod "af9cc25c-490d-4b26-ba22-54f5e945c495" (UID: "af9cc25c-490d-4b26-ba22-54f5e945c495"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:22:03.465752 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:03.465689    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af9cc25c-490d-4b26-ba22-54f5e945c495-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:22:03.467200 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:03.467178    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9cc25c-490d-4b26-ba22-54f5e945c495-kube-api-access-t2njb" pod "af9cc25c-490d-4b26-ba22-54f5e945c495" (UID: "af9cc25c-490d-4b26-ba22-54f5e945c495"). InnerVolumeSpecName "kube-api-access-t2njb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:03.467286 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:03.467269    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9cc25c-490d-4b26-ba22-54f5e945c495-provider-credentials" pod "af9cc25c-490d-4b26-ba22-54f5e945c495" (UID: "af9cc25c-490d-4b26-ba22-54f5e945c495"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:03.566253 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:03.566213    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af9cc25c-490d-4b26-ba22-54f5e945c495-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:22:03.566253 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:03.566243    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af9cc25c-490d-4b26-ba22-54f5e945c495-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:22:03.566253 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:03.566256    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2njb\" (UniqueName: \"kubernetes.io/projected/af9cc25c-490d-4b26-ba22-54f5e945c495-kube-api-access-t2njb\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:22:04.098375 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:04.098326    6059 generic.go:356] "Generic (PLEG): container finished" podID="af9cc25c-490d-4b26-ba22-54f5e945c495" containerID="08fd0eca4f07b9682466f5d8db2b7388669fe2f6c5b87645e747a8a76cab5c32" exitCode=0
Nov 03 18:22:04.098561 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:04.098391    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"
Nov 03 18:22:04.098561 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:04.098399    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv" event={"ID":"af9cc25c-490d-4b26-ba22-54f5e945c495","Type":"ContainerDied","Data":"08fd0eca4f07b9682466f5d8db2b7388669fe2f6c5b87645e747a8a76cab5c32"}
Nov 03 18:22:04.098561 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:04.098452    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv" event={"ID":"af9cc25c-490d-4b26-ba22-54f5e945c495","Type":"ContainerDied","Data":"3004159c57e15998f42eb2cf5531fdf19003183e67376e8e64e3fb87c22195e9"}
Nov 03 18:22:04.098561 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:04.098466    6059 scope.go:122] "RemoveContainer" containerID="08fd0eca4f07b9682466f5d8db2b7388669fe2f6c5b87645e747a8a76cab5c32"
Nov 03 18:22:04.108236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:04.108220    6059 scope.go:122] "RemoveContainer" containerID="08fd0eca4f07b9682466f5d8db2b7388669fe2f6c5b87645e747a8a76cab5c32"
Nov 03 18:22:04.108530 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:22:04.108511    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08fd0eca4f07b9682466f5d8db2b7388669fe2f6c5b87645e747a8a76cab5c32\": container with ID starting with 08fd0eca4f07b9682466f5d8db2b7388669fe2f6c5b87645e747a8a76cab5c32 not found: ID does not exist" containerID="08fd0eca4f07b9682466f5d8db2b7388669fe2f6c5b87645e747a8a76cab5c32"
Nov 03 18:22:04.108590 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:04.108538    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08fd0eca4f07b9682466f5d8db2b7388669fe2f6c5b87645e747a8a76cab5c32"} err="failed to get container status \"08fd0eca4f07b9682466f5d8db2b7388669fe2f6c5b87645e747a8a76cab5c32\": rpc error: code = NotFound desc = could not find container \"08fd0eca4f07b9682466f5d8db2b7388669fe2f6c5b87645e747a8a76cab5c32\": container with ID starting with 08fd0eca4f07b9682466f5d8db2b7388669fe2f6c5b87645e747a8a76cab5c32 not found: ID does not exist"
Nov 03 18:22:04.268765 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:04.268728    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"]
Nov 03 18:22:04.320504 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:04.320470    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f5kbqlv"]
Nov 03 18:22:05.199962 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:05.199921    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb" podUID="450728f8-87df-4e5a-85fd-902d0ae81879" containerName="payload-processing" containerID="cri-o://fc31d63b157fb67a58b325f00fdcf5f66c868a34a444180a40040dc1c990e32e" gracePeriod=25
Nov 03 18:22:05.334330 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:05.334299    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af9cc25c-490d-4b26-ba22-54f5e945c495" path="/var/lib/kubelet/pods/af9cc25c-490d-4b26-ba22-54f5e945c495/volumes"
Nov 03 18:22:05.457828 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:05.457806    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"
Nov 03 18:22:05.485658 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:05.485621    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/450728f8-87df-4e5a-85fd-902d0ae81879-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/450728f8-87df-4e5a-85fd-902d0ae81879-routing-overlay-volume\") pod \"450728f8-87df-4e5a-85fd-902d0ae81879\" (UID: \"450728f8-87df-4e5a-85fd-902d0ae81879\") "
Nov 03 18:22:05.485829 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:05.485688    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/450728f8-87df-4e5a-85fd-902d0ae81879-kube-api-access-fp9qc\" (UniqueName: \"kubernetes.io/projected/450728f8-87df-4e5a-85fd-902d0ae81879-kube-api-access-fp9qc\") pod \"450728f8-87df-4e5a-85fd-902d0ae81879\" (UID: \"450728f8-87df-4e5a-85fd-902d0ae81879\") "
Nov 03 18:22:05.485829 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:05.485738    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/450728f8-87df-4e5a-85fd-902d0ae81879-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450728f8-87df-4e5a-85fd-902d0ae81879-plugins-config-volume\") pod \"450728f8-87df-4e5a-85fd-902d0ae81879\" (UID: \"450728f8-87df-4e5a-85fd-902d0ae81879\") "
Nov 03 18:22:05.485829 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:05.485768    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/450728f8-87df-4e5a-85fd-902d0ae81879-provider-credentials\" (UniqueName: \"kubernetes.io/projected/450728f8-87df-4e5a-85fd-902d0ae81879-provider-credentials\") pod \"450728f8-87df-4e5a-85fd-902d0ae81879\" (UID: \"450728f8-87df-4e5a-85fd-902d0ae81879\") "
Nov 03 18:22:05.486045 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:05.486016    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/450728f8-87df-4e5a-85fd-902d0ae81879-routing-overlay-volume" pod "450728f8-87df-4e5a-85fd-902d0ae81879" (UID: "450728f8-87df-4e5a-85fd-902d0ae81879"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:22:05.486115 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:05.486080    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/450728f8-87df-4e5a-85fd-902d0ae81879-plugins-config-volume" pod "450728f8-87df-4e5a-85fd-902d0ae81879" (UID: "450728f8-87df-4e5a-85fd-902d0ae81879"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:22:05.487915 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:05.487894    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/450728f8-87df-4e5a-85fd-902d0ae81879-kube-api-access-fp9qc" pod "450728f8-87df-4e5a-85fd-902d0ae81879" (UID: "450728f8-87df-4e5a-85fd-902d0ae81879"). InnerVolumeSpecName "kube-api-access-fp9qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:05.487993 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:05.487918    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/450728f8-87df-4e5a-85fd-902d0ae81879-provider-credentials" pod "450728f8-87df-4e5a-85fd-902d0ae81879" (UID: "450728f8-87df-4e5a-85fd-902d0ae81879"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:05.587287 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:05.587251    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/450728f8-87df-4e5a-85fd-902d0ae81879-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:22:05.587287 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:05.587290    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp9qc\" (UniqueName: \"kubernetes.io/projected/450728f8-87df-4e5a-85fd-902d0ae81879-kube-api-access-fp9qc\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:22:05.587504 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:05.587308    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450728f8-87df-4e5a-85fd-902d0ae81879-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:22:05.587504 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:05.587325    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/450728f8-87df-4e5a-85fd-902d0ae81879-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:22:06.065005 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:06.064955    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:06.111218 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:06.111177    6059 generic.go:356] "Generic (PLEG): container finished" podID="450728f8-87df-4e5a-85fd-902d0ae81879" containerID="fc31d63b157fb67a58b325f00fdcf5f66c868a34a444180a40040dc1c990e32e" exitCode=0
Nov 03 18:22:06.111412 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:06.111232    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"
Nov 03 18:22:06.111412 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:06.111265    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb" event={"ID":"450728f8-87df-4e5a-85fd-902d0ae81879","Type":"ContainerDied","Data":"fc31d63b157fb67a58b325f00fdcf5f66c868a34a444180a40040dc1c990e32e"}
Nov 03 18:22:06.111412 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:06.111327    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb" event={"ID":"450728f8-87df-4e5a-85fd-902d0ae81879","Type":"ContainerDied","Data":"04a4d47f860ad1e480508c2ed181646b387299535e7dd7fb717b3325d0bd434c"}
Nov 03 18:22:06.111412 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:06.111383    6059 scope.go:122] "RemoveContainer" containerID="fc31d63b157fb67a58b325f00fdcf5f66c868a34a444180a40040dc1c990e32e"
Nov 03 18:22:06.121104 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:06.121080    6059 scope.go:122] "RemoveContainer" containerID="fc31d63b157fb67a58b325f00fdcf5f66c868a34a444180a40040dc1c990e32e"
Nov 03 18:22:06.121447 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:22:06.121428    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc31d63b157fb67a58b325f00fdcf5f66c868a34a444180a40040dc1c990e32e\": container with ID starting with fc31d63b157fb67a58b325f00fdcf5f66c868a34a444180a40040dc1c990e32e not found: ID does not exist" containerID="fc31d63b157fb67a58b325f00fdcf5f66c868a34a444180a40040dc1c990e32e"
Nov 03 18:22:06.121508 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:06.121457    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc31d63b157fb67a58b325f00fdcf5f66c868a34a444180a40040dc1c990e32e"} err="failed to get container status \"fc31d63b157fb67a58b325f00fdcf5f66c868a34a444180a40040dc1c990e32e\": rpc error: code = NotFound desc = could not find container \"fc31d63b157fb67a58b325f00fdcf5f66c868a34a444180a40040dc1c990e32e\": container with ID starting with fc31d63b157fb67a58b325f00fdcf5f66c868a34a444180a40040dc1c990e32e not found: ID does not exist"
Nov 03 18:22:06.283465 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:06.283429    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"]
Nov 03 18:22:06.289065 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:06.289034    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674vrbwb"]
Nov 03 18:22:06.861550 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:22:06.861504    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:22:07.334769 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:07.334734    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="450728f8-87df-4e5a-85fd-902d0ae81879" path="/var/lib/kubelet/pods/450728f8-87df-4e5a-85fd-902d0ae81879/volumes"
Nov 03 18:22:10.961776 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:10.961740    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:15.887391 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:15.887342    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:20.858504 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:20.858408    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:25.679201 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:25.679160    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:30.594517 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:30.594483    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:35.469713 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:35.469674    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:36.861039 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:22:36.861002    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:22:39.942961 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:39.942930    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:22:39.957895 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:39.957867    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:22:39.960979 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:39.960959    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:22:39.974073 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:39.974050    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:22:40.460894 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:40.460860    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:45.281057 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:45.281017    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:50.118546 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:50.118503    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:52.738761 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:52.738721    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:54.960063 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:54.960025    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:22:59.858069 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:22:59.858034    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:23:04.659155 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:23:04.659117    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:23:06.860093 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:23:06.860059    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:23:09.473382 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:23:09.473327    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:23:14.366223 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:23:14.366175    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:23:19.152118 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:23:19.152082    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:23:24.033809 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:23:24.033776    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:23:28.780647 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:23:28.780613    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:23:33.573792 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:23:33.573759    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:23:36.548614 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:23:36.548583    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:23:36.860999 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:23:36.860913    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:23:38.462990 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:23:38.462953    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:23:43.260755 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:23:43.260723    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:23:48.153094 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:23:48.153059    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:23:53.074318 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:23:53.074280    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:23:53.262858 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:23:53.262813    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:23:57.879782 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:23:57.879749    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:24:02.653535 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:24:02.653495    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:24:06.860437 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:24:06.860399    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:24:07.474191 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:24:07.474150    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:24:12.256790 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:24:12.256762    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:24:17.054727 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:24:17.054685    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:24:21.862878 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:24:21.862841    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:24:26.664245 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:24:26.664204    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:24:31.482344 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:24:31.482308    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:24:36.265069 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:24:36.265020    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:24:36.860107 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:24:36.860070    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:24:41.060069 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:24:41.060016    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:24:45.860034 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:24:45.860003    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:24:50.653212 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:24:50.653180    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:24:53.454041 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:24:53.454010    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:24:55.456620 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:24:55.456583    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:00.253537 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:00.253500    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:05.150309 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:05.150275    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:06.860283 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:25:06.860240    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:25:09.955297 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:09.955264    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:14.762676 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:14.762641    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:19.650562 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:19.650486    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:24.461659 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:24.461623    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:29.241126 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:29.241095    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:32.204077 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.204040    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-726791df-nqhtr"]
Nov 03 18:25:32.204595 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.204482    6059 cpu_manager.go:414] "containerMap: removing container" podUID="450728f8-87df-4e5a-85fd-902d0ae81879" containerName="payload-processing"
Nov 03 18:25:32.204595 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.204493    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="450728f8-87df-4e5a-85fd-902d0ae81879" containerName="payload-processing"
Nov 03 18:25:32.204595 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.204502    6059 cpu_manager.go:414] "containerMap: removing container" podUID="af9cc25c-490d-4b26-ba22-54f5e945c495" containerName="payload-processing"
Nov 03 18:25:32.204595 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.204507    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af9cc25c-490d-4b26-ba22-54f5e945c495" containerName="payload-processing"
Nov 03 18:25:32.204595 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.204594    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="450728f8-87df-4e5a-85fd-902d0ae81879" containerName="payload-processing"
Nov 03 18:25:32.204795 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.204603    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="af9cc25c-490d-4b26-ba22-54f5e945c495" containerName="payload-processing"
Nov 03 18:25:32.207795 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.207779    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726791df-nqhtr"
Nov 03 18:25:32.262280 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.262253    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:25:32.262443 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.262307    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:25:32.262443 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.262378    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:25:32.314019 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.313989    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 18:25:32.315906 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.315887    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-726791df-nqhtr"]
Nov 03 18:25:32.367971 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.367939    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9114e7f0-5812-47c3-a08b-e77e810a8188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-726791df-nqhtr\" (UID: \"9114e7f0-5812-47c3-a08b-e77e810a8188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726791df-nqhtr"
Nov 03 18:25:32.368118 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.367978    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7s2z\" (UniqueName: \"kubernetes.io/projected/9114e7f0-5812-47c3-a08b-e77e810a8188-kube-api-access-q7s2z\") pod \"maas-api-revoke-sub-726791df-nqhtr\" (UID: \"9114e7f0-5812-47c3-a08b-e77e810a8188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726791df-nqhtr"
Nov 03 18:25:32.368118 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.368077    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9114e7f0-5812-47c3-a08b-e77e810a8188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-726791df-nqhtr\" (UID: \"9114e7f0-5812-47c3-a08b-e77e810a8188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726791df-nqhtr"
Nov 03 18:25:32.468827 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.468797    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9114e7f0-5812-47c3-a08b-e77e810a8188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-726791df-nqhtr\" (UID: \"9114e7f0-5812-47c3-a08b-e77e810a8188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726791df-nqhtr"
Nov 03 18:25:32.468998 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.468902    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9114e7f0-5812-47c3-a08b-e77e810a8188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-726791df-nqhtr\" (UID: \"9114e7f0-5812-47c3-a08b-e77e810a8188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726791df-nqhtr"
Nov 03 18:25:32.468998 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.468927    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7s2z\" (UniqueName: \"kubernetes.io/projected/9114e7f0-5812-47c3-a08b-e77e810a8188-kube-api-access-q7s2z\") pod \"maas-api-revoke-sub-726791df-nqhtr\" (UID: \"9114e7f0-5812-47c3-a08b-e77e810a8188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726791df-nqhtr"
Nov 03 18:25:32.469613 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.469585    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9114e7f0-5812-47c3-a08b-e77e810a8188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-726791df-nqhtr\" (UID: \"9114e7f0-5812-47c3-a08b-e77e810a8188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726791df-nqhtr"
Nov 03 18:25:32.471272 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.471248    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9114e7f0-5812-47c3-a08b-e77e810a8188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-726791df-nqhtr\" (UID: \"9114e7f0-5812-47c3-a08b-e77e810a8188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726791df-nqhtr"
Nov 03 18:25:32.532954 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.532910    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7s2z\" (UniqueName: \"kubernetes.io/projected/9114e7f0-5812-47c3-a08b-e77e810a8188-kube-api-access-q7s2z\") pod \"maas-api-revoke-sub-726791df-nqhtr\" (UID: \"9114e7f0-5812-47c3-a08b-e77e810a8188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726791df-nqhtr"
Nov 03 18:25:32.820181 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:32.820104    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726791df-nqhtr"
Nov 03 18:25:33.189887 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:25:33.189852    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9114e7f0_5812_47c3_a08b_e77e810a8188.slice/crio-77bdff74931885b220f04249cb68fdb29204110c24024ecc35610d5c3a80e0df WatchSource:0}: Error finding container 77bdff74931885b220f04249cb68fdb29204110c24024ecc35610d5c3a80e0df: Status 404 returned error can't find the container with id 77bdff74931885b220f04249cb68fdb29204110c24024ecc35610d5c3a80e0df
Nov 03 18:25:33.191397 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:33.191373    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-726791df-nqhtr"]
Nov 03 18:25:33.229382 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:33.229338    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726791df-nqhtr" event={"ID":"9114e7f0-5812-47c3-a08b-e77e810a8188","Type":"ContainerStarted","Data":"77bdff74931885b220f04249cb68fdb29204110c24024ecc35610d5c3a80e0df"}
Nov 03 18:25:34.065126 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:34.065087    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:34.236057 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:34.236026    6059 generic.go:356] "Generic (PLEG): container finished" podID="9114e7f0-5812-47c3-a08b-e77e810a8188" containerID="6a9a33108b200a2706ab6784ba38c9a6495540457c7c0490983e86bccae267c7" exitCode=0
Nov 03 18:25:34.236461 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:34.236112    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726791df-nqhtr" event={"ID":"9114e7f0-5812-47c3-a08b-e77e810a8188","Type":"ContainerDied","Data":"6a9a33108b200a2706ab6784ba38c9a6495540457c7c0490983e86bccae267c7"}
Nov 03 18:25:35.369382 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:35.369342    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726791df-nqhtr"
Nov 03 18:25:35.502392 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:35.502286    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9114e7f0-5812-47c3-a08b-e77e810a8188-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9114e7f0-5812-47c3-a08b-e77e810a8188-maas-api-openshift-ca\") pod \"9114e7f0-5812-47c3-a08b-e77e810a8188\" (UID: \"9114e7f0-5812-47c3-a08b-e77e810a8188\") "
Nov 03 18:25:35.502392 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:35.502384    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9114e7f0-5812-47c3-a08b-e77e810a8188-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9114e7f0-5812-47c3-a08b-e77e810a8188-maas-api-xks-ca\") pod \"9114e7f0-5812-47c3-a08b-e77e810a8188\" (UID: \"9114e7f0-5812-47c3-a08b-e77e810a8188\") "
Nov 03 18:25:35.502636 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:35.502437    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9114e7f0-5812-47c3-a08b-e77e810a8188-kube-api-access-q7s2z\" (UniqueName: \"kubernetes.io/projected/9114e7f0-5812-47c3-a08b-e77e810a8188-kube-api-access-q7s2z\") pod \"9114e7f0-5812-47c3-a08b-e77e810a8188\" (UID: \"9114e7f0-5812-47c3-a08b-e77e810a8188\") "
Nov 03 18:25:35.502735 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:35.502707    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9114e7f0-5812-47c3-a08b-e77e810a8188-maas-api-openshift-ca" pod "9114e7f0-5812-47c3-a08b-e77e810a8188" (UID: "9114e7f0-5812-47c3-a08b-e77e810a8188"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:25:35.502983 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:35.502956    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9114e7f0-5812-47c3-a08b-e77e810a8188-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:25:35.504523 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:35.504495    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9114e7f0-5812-47c3-a08b-e77e810a8188-maas-api-xks-ca" pod "9114e7f0-5812-47c3-a08b-e77e810a8188" (UID: "9114e7f0-5812-47c3-a08b-e77e810a8188"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:25:35.504617 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:35.504496    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9114e7f0-5812-47c3-a08b-e77e810a8188-kube-api-access-q7s2z" pod "9114e7f0-5812-47c3-a08b-e77e810a8188" (UID: "9114e7f0-5812-47c3-a08b-e77e810a8188"). InnerVolumeSpecName "kube-api-access-q7s2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:25:35.603890 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:35.603855    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9114e7f0-5812-47c3-a08b-e77e810a8188-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:25:35.603890 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:35.603886    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7s2z\" (UniqueName: \"kubernetes.io/projected/9114e7f0-5812-47c3-a08b-e77e810a8188-kube-api-access-q7s2z\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:25:36.249045 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:36.249010    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726791df-nqhtr"
Nov 03 18:25:36.249227 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:36.249059    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726791df-nqhtr" event={"ID":"9114e7f0-5812-47c3-a08b-e77e810a8188","Type":"ContainerDied","Data":"77bdff74931885b220f04249cb68fdb29204110c24024ecc35610d5c3a80e0df"}
Nov 03 18:25:36.249227 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:36.249113    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77bdff74931885b220f04249cb68fdb29204110c24024ecc35610d5c3a80e0df"
Nov 03 18:25:36.860479 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:25:36.860444    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:25:38.852642 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:38.852608    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:43.671589 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:43.671531    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:43.870734 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:43.870698    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml"]
Nov 03 18:25:43.871257 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:43.871238    6059 cpu_manager.go:414] "containerMap: removing container" podUID="9114e7f0-5812-47c3-a08b-e77e810a8188" containerName="revoke-keys"
Nov 03 18:25:43.871314 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:43.871258    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9114e7f0-5812-47c3-a08b-e77e810a8188" containerName="revoke-keys"
Nov 03 18:25:43.871391 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:43.871340    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9114e7f0-5812-47c3-a08b-e77e810a8188" containerName="revoke-keys"
Nov 03 18:25:43.874325 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:43.874308    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml"
Nov 03 18:25:43.931755 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:43.931646    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:25:43.931755 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:43.931646    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:25:43.931755 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:43.931693    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:25:43.932091 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:43.931782    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 18:25:43.934548 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:43.934522    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml"]
Nov 03 18:25:43.986052 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:43.986017    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxrk7\" (UniqueName: \"kubernetes.io/projected/99f6c5fc-3a07-4154-9570-cff5b49feedd-kube-api-access-dxrk7\") pod \"maas-api-revoke-sub-ee19937a-56dml\" (UID: \"99f6c5fc-3a07-4154-9570-cff5b49feedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml"
Nov 03 18:25:43.986225 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:43.986064    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f6c5fc-3a07-4154-9570-cff5b49feedd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee19937a-56dml\" (UID: \"99f6c5fc-3a07-4154-9570-cff5b49feedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml"
Nov 03 18:25:43.986225 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:43.986206    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f6c5fc-3a07-4154-9570-cff5b49feedd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee19937a-56dml\" (UID: \"99f6c5fc-3a07-4154-9570-cff5b49feedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml"
Nov 03 18:25:44.087395 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:44.087332    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxrk7\" (UniqueName: \"kubernetes.io/projected/99f6c5fc-3a07-4154-9570-cff5b49feedd-kube-api-access-dxrk7\") pod \"maas-api-revoke-sub-ee19937a-56dml\" (UID: \"99f6c5fc-3a07-4154-9570-cff5b49feedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml"
Nov 03 18:25:44.087550 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:44.087420    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f6c5fc-3a07-4154-9570-cff5b49feedd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee19937a-56dml\" (UID: \"99f6c5fc-3a07-4154-9570-cff5b49feedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml"
Nov 03 18:25:44.087550 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:44.087514    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f6c5fc-3a07-4154-9570-cff5b49feedd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee19937a-56dml\" (UID: \"99f6c5fc-3a07-4154-9570-cff5b49feedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml"
Nov 03 18:25:44.088330 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:44.088296    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f6c5fc-3a07-4154-9570-cff5b49feedd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee19937a-56dml\" (UID: \"99f6c5fc-3a07-4154-9570-cff5b49feedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml"
Nov 03 18:25:44.089881 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:44.089851    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f6c5fc-3a07-4154-9570-cff5b49feedd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee19937a-56dml\" (UID: \"99f6c5fc-3a07-4154-9570-cff5b49feedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml"
Nov 03 18:25:44.151607 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:44.151580    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxrk7\" (UniqueName: \"kubernetes.io/projected/99f6c5fc-3a07-4154-9570-cff5b49feedd-kube-api-access-dxrk7\") pod \"maas-api-revoke-sub-ee19937a-56dml\" (UID: \"99f6c5fc-3a07-4154-9570-cff5b49feedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml"
Nov 03 18:25:44.186662 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:44.186595    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml"
Nov 03 18:25:44.561660 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:44.561623    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml"]
Nov 03 18:25:44.567798 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:25:44.567769    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f6c5fc_3a07_4154_9570_cff5b49feedd.slice/crio-5d0868a6d4aaf7c4d35829b85f420b18cd7bff6049c80ba2d4f00bc226f30911 WatchSource:0}: Error finding container 5d0868a6d4aaf7c4d35829b85f420b18cd7bff6049c80ba2d4f00bc226f30911: Status 404 returned error can't find the container with id 5d0868a6d4aaf7c4d35829b85f420b18cd7bff6049c80ba2d4f00bc226f30911
Nov 03 18:25:45.301779 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:45.301739    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml" event={"ID":"99f6c5fc-3a07-4154-9570-cff5b49feedd","Type":"ContainerStarted","Data":"fbe1d07f4538df91be61d3ddb83baca0c51e6abb28823ed8838ecc6a2c1848cf"}
Nov 03 18:25:45.302168 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:45.301805    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml" event={"ID":"99f6c5fc-3a07-4154-9570-cff5b49feedd","Type":"ContainerStarted","Data":"5d0868a6d4aaf7c4d35829b85f420b18cd7bff6049c80ba2d4f00bc226f30911"}
Nov 03 18:25:45.422083 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:45.422013    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml" podStartSLOduration=2.421995878 podStartE2EDuration="2.421995878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:25:45.420812062 +0000 UTC m=+4086.218524856" watchObservedRunningTime="2026-11-03 18:25:45.421995878 +0000 UTC m=+4086.219708663"
Nov 03 18:25:46.308650 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:46.308610    6059 generic.go:356] "Generic (PLEG): container finished" podID="99f6c5fc-3a07-4154-9570-cff5b49feedd" containerID="fbe1d07f4538df91be61d3ddb83baca0c51e6abb28823ed8838ecc6a2c1848cf" exitCode=0
Nov 03 18:25:46.309081 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:46.308693    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml" event={"ID":"99f6c5fc-3a07-4154-9570-cff5b49feedd","Type":"ContainerDied","Data":"fbe1d07f4538df91be61d3ddb83baca0c51e6abb28823ed8838ecc6a2c1848cf"}
Nov 03 18:25:47.447517 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:47.447494    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml"
Nov 03 18:25:47.519311 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:47.519280    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99f6c5fc-3a07-4154-9570-cff5b49feedd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f6c5fc-3a07-4154-9570-cff5b49feedd-maas-api-xks-ca\") pod \"99f6c5fc-3a07-4154-9570-cff5b49feedd\" (UID: \"99f6c5fc-3a07-4154-9570-cff5b49feedd\") "
Nov 03 18:25:47.519494 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:47.519370    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99f6c5fc-3a07-4154-9570-cff5b49feedd-kube-api-access-dxrk7\" (UniqueName: \"kubernetes.io/projected/99f6c5fc-3a07-4154-9570-cff5b49feedd-kube-api-access-dxrk7\") pod \"99f6c5fc-3a07-4154-9570-cff5b49feedd\" (UID: \"99f6c5fc-3a07-4154-9570-cff5b49feedd\") "
Nov 03 18:25:47.519567 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:47.519522    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99f6c5fc-3a07-4154-9570-cff5b49feedd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f6c5fc-3a07-4154-9570-cff5b49feedd-maas-api-openshift-ca\") pod \"99f6c5fc-3a07-4154-9570-cff5b49feedd\" (UID: \"99f6c5fc-3a07-4154-9570-cff5b49feedd\") "
Nov 03 18:25:47.519875 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:47.519848    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f6c5fc-3a07-4154-9570-cff5b49feedd-maas-api-openshift-ca" pod "99f6c5fc-3a07-4154-9570-cff5b49feedd" (UID: "99f6c5fc-3a07-4154-9570-cff5b49feedd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:25:47.521431 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:47.521407    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99f6c5fc-3a07-4154-9570-cff5b49feedd-maas-api-xks-ca" pod "99f6c5fc-3a07-4154-9570-cff5b49feedd" (UID: "99f6c5fc-3a07-4154-9570-cff5b49feedd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:25:47.521526 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:47.521429    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f6c5fc-3a07-4154-9570-cff5b49feedd-kube-api-access-dxrk7" pod "99f6c5fc-3a07-4154-9570-cff5b49feedd" (UID: "99f6c5fc-3a07-4154-9570-cff5b49feedd"). InnerVolumeSpecName "kube-api-access-dxrk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:25:47.620693 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:47.620608    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f6c5fc-3a07-4154-9570-cff5b49feedd-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:25:47.620693 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:47.620635    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxrk7\" (UniqueName: \"kubernetes.io/projected/99f6c5fc-3a07-4154-9570-cff5b49feedd-kube-api-access-dxrk7\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:25:47.620693 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:47.620646    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f6c5fc-3a07-4154-9570-cff5b49feedd-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:25:48.320758 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:48.320724    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml"
Nov 03 18:25:48.320939 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:48.320754    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml" event={"ID":"99f6c5fc-3a07-4154-9570-cff5b49feedd","Type":"ContainerDied","Data":"5d0868a6d4aaf7c4d35829b85f420b18cd7bff6049c80ba2d4f00bc226f30911"}
Nov 03 18:25:48.320939 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:48.320800    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d0868a6d4aaf7c4d35829b85f420b18cd7bff6049c80ba2d4f00bc226f30911"
Nov 03 18:25:48.648273 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:48.648188    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:53.410597 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:53.410561    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:53.803126 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:53.803092    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:57.012192 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:57.012159    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:57.194566 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:57.194533    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:57.656008 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:57.655969    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml"]
Nov 03 18:25:57.656237 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:57.656212    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml" podUID="22ce7784-efe4-4ad1-b567-ecd69149f2f3" containerName="sim" containerID="cri-o://a10ebdc7479cc7340aed0bce0534217af2804e50372c51286b4c1c419538ede3" gracePeriod=30
Nov 03 18:25:57.834910 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:57.834880    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"]
Nov 03 18:25:57.835367 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:57.835339    6059 cpu_manager.go:414] "containerMap: removing container" podUID="99f6c5fc-3a07-4154-9570-cff5b49feedd" containerName="revoke-keys"
Nov 03 18:25:57.835471 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:57.835367    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99f6c5fc-3a07-4154-9570-cff5b49feedd" containerName="revoke-keys"
Nov 03 18:25:57.835471 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:57.835437    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="99f6c5fc-3a07-4154-9570-cff5b49feedd" containerName="revoke-keys"
Nov 03 18:25:57.838902 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:57.838877    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"
Nov 03 18:25:57.901262 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:57.901232    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"]
Nov 03 18:25:57.905673 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:57.905645    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:57.917671 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:57.917608    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml"
Nov 03 18:25:57.929098 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:57.929066    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/598f1c49-a188-4e50-9c6f-75b4468a126e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r\" (UID: \"598f1c49-a188-4e50-9c6f-75b4468a126e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"
Nov 03 18:25:57.929256 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:57.929119    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7s9b\" (UniqueName: \"kubernetes.io/projected/598f1c49-a188-4e50-9c6f-75b4468a126e-kube-api-access-c7s9b\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r\" (UID: \"598f1c49-a188-4e50-9c6f-75b4468a126e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"
Nov 03 18:25:57.929256 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:57.929185    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/598f1c49-a188-4e50-9c6f-75b4468a126e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r\" (UID: \"598f1c49-a188-4e50-9c6f-75b4468a126e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"
Nov 03 18:25:57.929347 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:57.929297    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/598f1c49-a188-4e50-9c6f-75b4468a126e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r\" (UID: \"598f1c49-a188-4e50-9c6f-75b4468a126e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"
Nov 03 18:25:58.030226 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.030194    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22ce7784-efe4-4ad1-b567-ecd69149f2f3-tls\" (UniqueName: \"kubernetes.io/secret/22ce7784-efe4-4ad1-b567-ecd69149f2f3-tls\") pod \"22ce7784-efe4-4ad1-b567-ecd69149f2f3\" (UID: \"22ce7784-efe4-4ad1-b567-ecd69149f2f3\") "
Nov 03 18:25:58.030608 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.030309    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ce7784-efe4-4ad1-b567-ecd69149f2f3-kube-api-access-m7mdj\" (UniqueName: \"kubernetes.io/projected/22ce7784-efe4-4ad1-b567-ecd69149f2f3-kube-api-access-m7mdj\") pod \"22ce7784-efe4-4ad1-b567-ecd69149f2f3\" (UID: \"22ce7784-efe4-4ad1-b567-ecd69149f2f3\") "
Nov 03 18:25:58.030676 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.030597    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/598f1c49-a188-4e50-9c6f-75b4468a126e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r\" (UID: \"598f1c49-a188-4e50-9c6f-75b4468a126e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"
Nov 03 18:25:58.030676 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.030652    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/598f1c49-a188-4e50-9c6f-75b4468a126e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r\" (UID: \"598f1c49-a188-4e50-9c6f-75b4468a126e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"
Nov 03 18:25:58.030779 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.030710    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7s9b\" (UniqueName: \"kubernetes.io/projected/598f1c49-a188-4e50-9c6f-75b4468a126e-kube-api-access-c7s9b\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r\" (UID: \"598f1c49-a188-4e50-9c6f-75b4468a126e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"
Nov 03 18:25:58.030779 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.030727    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/598f1c49-a188-4e50-9c6f-75b4468a126e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r\" (UID: \"598f1c49-a188-4e50-9c6f-75b4468a126e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"
Nov 03 18:25:58.031328 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.031301    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/598f1c49-a188-4e50-9c6f-75b4468a126e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r\" (UID: \"598f1c49-a188-4e50-9c6f-75b4468a126e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"
Nov 03 18:25:58.031476 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.031444    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/598f1c49-a188-4e50-9c6f-75b4468a126e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r\" (UID: \"598f1c49-a188-4e50-9c6f-75b4468a126e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"
Nov 03 18:25:58.033004 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.032976    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22ce7784-efe4-4ad1-b567-ecd69149f2f3-tls" pod "22ce7784-efe4-4ad1-b567-ecd69149f2f3" (UID: "22ce7784-efe4-4ad1-b567-ecd69149f2f3"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:25:58.033106 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.033004    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ce7784-efe4-4ad1-b567-ecd69149f2f3-kube-api-access-m7mdj" pod "22ce7784-efe4-4ad1-b567-ecd69149f2f3" (UID: "22ce7784-efe4-4ad1-b567-ecd69149f2f3"). InnerVolumeSpecName "kube-api-access-m7mdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:25:58.033383 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.033340    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/598f1c49-a188-4e50-9c6f-75b4468a126e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r\" (UID: \"598f1c49-a188-4e50-9c6f-75b4468a126e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"
Nov 03 18:25:58.096756 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.096714    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7s9b\" (UniqueName: \"kubernetes.io/projected/598f1c49-a188-4e50-9c6f-75b4468a126e-kube-api-access-c7s9b\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r\" (UID: \"598f1c49-a188-4e50-9c6f-75b4468a126e\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"
Nov 03 18:25:58.131964 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.131932    6059 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/22ce7784-efe4-4ad1-b567-ecd69149f2f3-tls\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:25:58.131964 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.131962    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7mdj\" (UniqueName: \"kubernetes.io/projected/22ce7784-efe4-4ad1-b567-ecd69149f2f3-kube-api-access-m7mdj\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:25:58.155881 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.155845    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"
Nov 03 18:25:58.362515 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.362477    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:25:58.373824 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.373791    6059 generic.go:356] "Generic (PLEG): container finished" podID="22ce7784-efe4-4ad1-b567-ecd69149f2f3" containerID="a10ebdc7479cc7340aed0bce0534217af2804e50372c51286b4c1c419538ede3" exitCode=0
Nov 03 18:25:58.373983 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.373838    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml"
Nov 03 18:25:58.373983 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.373877    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml" event={"ID":"22ce7784-efe4-4ad1-b567-ecd69149f2f3","Type":"ContainerDied","Data":"a10ebdc7479cc7340aed0bce0534217af2804e50372c51286b4c1c419538ede3"}
Nov 03 18:25:58.373983 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.373921    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml" event={"ID":"22ce7784-efe4-4ad1-b567-ecd69149f2f3","Type":"ContainerDied","Data":"685a414a931335b1b3c27914efcb81450d944421ba1a39776a0508c72e359696"}
Nov 03 18:25:58.373983 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.373944    6059 scope.go:122] "RemoveContainer" containerID="a10ebdc7479cc7340aed0bce0534217af2804e50372c51286b4c1c419538ede3"
Nov 03 18:25:58.384223 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.384203    6059 scope.go:122] "RemoveContainer" containerID="a10ebdc7479cc7340aed0bce0534217af2804e50372c51286b4c1c419538ede3"
Nov 03 18:25:58.384515 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:25:58.384495    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a10ebdc7479cc7340aed0bce0534217af2804e50372c51286b4c1c419538ede3\": container with ID starting with a10ebdc7479cc7340aed0bce0534217af2804e50372c51286b4c1c419538ede3 not found: ID does not exist" containerID="a10ebdc7479cc7340aed0bce0534217af2804e50372c51286b4c1c419538ede3"
Nov 03 18:25:58.384598 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.384523    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a10ebdc7479cc7340aed0bce0534217af2804e50372c51286b4c1c419538ede3"} err="failed to get container status \"a10ebdc7479cc7340aed0bce0534217af2804e50372c51286b4c1c419538ede3\": rpc error: code = NotFound desc = could not find container \"a10ebdc7479cc7340aed0bce0534217af2804e50372c51286b4c1c419538ede3\": container with ID starting with a10ebdc7479cc7340aed0bce0534217af2804e50372c51286b4c1c419538ede3 not found: ID does not exist"
Nov 03 18:25:58.531677 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:25:58.531648    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598f1c49_a188_4e50_9c6f_75b4468a126e.slice/crio-faca15a4f1ec2c20214c37819119dc9606fd966da951ab3a9cf293b80a88b209 WatchSource:0}: Error finding container faca15a4f1ec2c20214c37819119dc9606fd966da951ab3a9cf293b80a88b209: Status 404 returned error can't find the container with id faca15a4f1ec2c20214c37819119dc9606fd966da951ab3a9cf293b80a88b209
Nov 03 18:25:58.533695 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.533669    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"]
Nov 03 18:25:58.544196 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.544169    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml"]
Nov 03 18:25:58.551307 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:58.551285    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-82ab32/e2e-usage-sim-70698c-5f646997c4-dx8ml"]
Nov 03 18:25:59.334750 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:59.334716    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22ce7784-efe4-4ad1-b567-ecd69149f2f3" path="/var/lib/kubelet/pods/22ce7784-efe4-4ad1-b567-ecd69149f2f3/volumes"
Nov 03 18:25:59.381847 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:59.381811    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r" event={"ID":"598f1c49-a188-4e50-9c6f-75b4468a126e","Type":"ContainerStarted","Data":"ab0b546f8cc1cd6852e66d1267803de873d018a42acf1309ca904490924c8c99"}
Nov 03 18:25:59.382030 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:59.381869    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r" event={"ID":"598f1c49-a188-4e50-9c6f-75b4468a126e","Type":"ContainerStarted","Data":"faca15a4f1ec2c20214c37819119dc9606fd966da951ab3a9cf293b80a88b209"}
Nov 03 18:25:59.382030 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:59.381898    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"
Nov 03 18:25:59.503670 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:25:59.503608    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r" podStartSLOduration=2.503592325 podStartE2EDuration="2.503592325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:25:59.503019349 +0000 UTC m=+4100.300732134" watchObservedRunningTime="2026-11-03 18:25:59.503592325 +0000 UTC m=+4100.301305106"
Nov 03 18:26:00.234165 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:00.234125    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"]
Nov 03 18:26:00.235272 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:00.235244    6059 cpu_manager.go:414] "containerMap: removing container" podUID="22ce7784-efe4-4ad1-b567-ecd69149f2f3" containerName="sim"
Nov 03 18:26:00.235272 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:00.235271    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ce7784-efe4-4ad1-b567-ecd69149f2f3" containerName="sim"
Nov 03 18:26:00.235508 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:00.235493    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ce7784-efe4-4ad1-b567-ecd69149f2f3" containerName="sim"
Nov 03 18:26:00.240059 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:00.240034    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"
Nov 03 18:26:00.296891 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:00.296862    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"]
Nov 03 18:26:00.353603 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:00.353576    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz\" (UID: \"124f0414-5bef-4ff1-87e5-ce3b1739e468\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"
Nov 03 18:26:00.353960 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:00.353608    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/124f0414-5bef-4ff1-87e5-ce3b1739e468-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz\" (UID: \"124f0414-5bef-4ff1-87e5-ce3b1739e468\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"
Nov 03 18:26:00.353960 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:00.353639    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/124f0414-5bef-4ff1-87e5-ce3b1739e468-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz\" (UID: \"124f0414-5bef-4ff1-87e5-ce3b1739e468\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"
Nov 03 18:26:00.353960 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:00.353751    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln2tl\" (UniqueName: \"kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-kube-api-access-ln2tl\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz\" (UID: \"124f0414-5bef-4ff1-87e5-ce3b1739e468\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"
Nov 03 18:26:00.454928 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:00.454895    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz\" (UID: \"124f0414-5bef-4ff1-87e5-ce3b1739e468\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"
Nov 03 18:26:00.454928 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:00.454931    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/124f0414-5bef-4ff1-87e5-ce3b1739e468-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz\" (UID: \"124f0414-5bef-4ff1-87e5-ce3b1739e468\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"
Nov 03 18:26:00.455159 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:00.454968    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/124f0414-5bef-4ff1-87e5-ce3b1739e468-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz\" (UID: \"124f0414-5bef-4ff1-87e5-ce3b1739e468\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"
Nov 03 18:26:00.455159 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:00.454994    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln2tl\" (UniqueName: \"kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-kube-api-access-ln2tl\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz\" (UID: \"124f0414-5bef-4ff1-87e5-ce3b1739e468\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"
Nov 03 18:26:00.455651 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:00.455618    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/124f0414-5bef-4ff1-87e5-ce3b1739e468-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz\" (UID: \"124f0414-5bef-4ff1-87e5-ce3b1739e468\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"
Nov 03 18:26:00.455794 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:00.455773    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/124f0414-5bef-4ff1-87e5-ce3b1739e468-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz\" (UID: \"124f0414-5bef-4ff1-87e5-ce3b1739e468\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"
Nov 03 18:26:00.457526 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:00.457503    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz\" (UID: \"124f0414-5bef-4ff1-87e5-ce3b1739e468\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"
Nov 03 18:26:00.520346 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:00.520271    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln2tl\" (UniqueName: \"kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-kube-api-access-ln2tl\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz\" (UID: \"124f0414-5bef-4ff1-87e5-ce3b1739e468\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"
Nov 03 18:26:00.553958 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:00.553926    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"
Nov 03 18:26:00.927549 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:00.927521    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"]
Nov 03 18:26:00.927794 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:26:00.927768    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124f0414_5bef_4ff1_87e5_ce3b1739e468.slice/crio-0a5ef8f3c509adba2ef7a544e5768b72ebccc4d18c105d4bce35bd4e8f0f1c17 WatchSource:0}: Error finding container 0a5ef8f3c509adba2ef7a544e5768b72ebccc4d18c105d4bce35bd4e8f0f1c17: Status 404 returned error can't find the container with id 0a5ef8f3c509adba2ef7a544e5768b72ebccc4d18c105d4bce35bd4e8f0f1c17
Nov 03 18:26:01.393797 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:01.393760    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz" event={"ID":"124f0414-5bef-4ff1-87e5-ce3b1739e468","Type":"ContainerStarted","Data":"d741ea0b361b354f238c3cccd5b9ac8b239314531ffcf5d6f92b9a1d0ba4f71b"}
Nov 03 18:26:01.394234 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:01.393817    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz" event={"ID":"124f0414-5bef-4ff1-87e5-ce3b1739e468","Type":"ContainerStarted","Data":"0a5ef8f3c509adba2ef7a544e5768b72ebccc4d18c105d4bce35bd4e8f0f1c17"}
Nov 03 18:26:01.394234 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:01.393892    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"
Nov 03 18:26:01.515167 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:01.515115    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz" podStartSLOduration=1.515095302 podStartE2EDuration="1.515095302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:26:01.514341281 +0000 UTC m=+4102.312054062" watchObservedRunningTime="2026-11-03 18:26:01.515095302 +0000 UTC m=+4102.312808083"
Nov 03 18:26:02.386892 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:02.386860    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:02.477843 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:02.477811    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-7afe86be4901: secret "agc-cred-7afe86be4901" not found
Nov 03 18:26:02.477843 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:02.477847    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz: secret "agc-cred-7afe86be4901" not found
Nov 03 18:26:02.478259 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:02.477894    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-provider-credentials podName:124f0414-5bef-4ff1-87e5-ce3b1739e468 nodeName:}" failed. No retries permitted until 2026-11-03 18:26:02.97787716 +0000 UTC m=+4103.775589932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz" (UID: "124f0414-5bef-4ff1-87e5-ce3b1739e468") : secret "agc-cred-7afe86be4901" not found
Nov 03 18:26:02.568861 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:02.568820    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:02.619766 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:02.619734    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"]
Nov 03 18:26:02.651491 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:02.651405    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"]
Nov 03 18:26:02.655490 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:02.655466    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"
Nov 03 18:26:02.734235 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:02.734203    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"]
Nov 03 18:26:02.780915 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:02.780885    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d396930-b2b9-45df-aa1d-042dd368fef3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg\" (UID: \"2d396930-b2b9-45df-aa1d-042dd368fef3\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"
Nov 03 18:26:02.781075 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:02.780929    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d396930-b2b9-45df-aa1d-042dd368fef3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg\" (UID: \"2d396930-b2b9-45df-aa1d-042dd368fef3\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"
Nov 03 18:26:02.781075 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:02.780947    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d396930-b2b9-45df-aa1d-042dd368fef3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg\" (UID: \"2d396930-b2b9-45df-aa1d-042dd368fef3\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"
Nov 03 18:26:02.781151 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:02.781095    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbrnf\" (UniqueName: \"kubernetes.io/projected/2d396930-b2b9-45df-aa1d-042dd368fef3-kube-api-access-bbrnf\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg\" (UID: \"2d396930-b2b9-45df-aa1d-042dd368fef3\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"
Nov 03 18:26:02.882588 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:02.882544    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d396930-b2b9-45df-aa1d-042dd368fef3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg\" (UID: \"2d396930-b2b9-45df-aa1d-042dd368fef3\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"
Nov 03 18:26:02.882882 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:02.882595    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d396930-b2b9-45df-aa1d-042dd368fef3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg\" (UID: \"2d396930-b2b9-45df-aa1d-042dd368fef3\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"
Nov 03 18:26:02.882882 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:02.882704    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbrnf\" (UniqueName: \"kubernetes.io/projected/2d396930-b2b9-45df-aa1d-042dd368fef3-kube-api-access-bbrnf\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg\" (UID: \"2d396930-b2b9-45df-aa1d-042dd368fef3\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"
Nov 03 18:26:02.882882 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:02.882825    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d396930-b2b9-45df-aa1d-042dd368fef3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg\" (UID: \"2d396930-b2b9-45df-aa1d-042dd368fef3\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"
Nov 03 18:26:02.883392 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:02.883345    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d396930-b2b9-45df-aa1d-042dd368fef3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg\" (UID: \"2d396930-b2b9-45df-aa1d-042dd368fef3\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"
Nov 03 18:26:02.883574 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:02.883557    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d396930-b2b9-45df-aa1d-042dd368fef3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg\" (UID: \"2d396930-b2b9-45df-aa1d-042dd368fef3\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"
Nov 03 18:26:02.885163 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:02.885146    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d396930-b2b9-45df-aa1d-042dd368fef3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg\" (UID: \"2d396930-b2b9-45df-aa1d-042dd368fef3\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"
Nov 03 18:26:02.945702 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:02.945644    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbrnf\" (UniqueName: \"kubernetes.io/projected/2d396930-b2b9-45df-aa1d-042dd368fef3-kube-api-access-bbrnf\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg\" (UID: \"2d396930-b2b9-45df-aa1d-042dd368fef3\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"
Nov 03 18:26:02.968537 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:02.968507    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"
Nov 03 18:26:02.984286 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:02.984264    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-7afe86be4901: secret "agc-cred-7afe86be4901" not found
Nov 03 18:26:02.984395 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:02.984292    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz: secret "agc-cred-7afe86be4901" not found
Nov 03 18:26:02.984395 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:02.984342    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-provider-credentials podName:124f0414-5bef-4ff1-87e5-ce3b1739e468 nodeName:}" failed. No retries permitted until 2026-11-03 18:26:03.98432636 +0000 UTC m=+4104.782039120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz" (UID: "124f0414-5bef-4ff1-87e5-ce3b1739e468") : secret "agc-cred-7afe86be4901" not found
Nov 03 18:26:03.179674 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:03.179641    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5"]
Nov 03 18:26:03.184388 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:03.184348    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5"
Nov 03 18:26:03.240577 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:03.240549    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:26:03.240739 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:03.240624    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:26:03.240837 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:03.240803    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 18:26:03.240904 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:03.240820    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:26:03.243837 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:03.243812    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5"]
Nov 03 18:26:03.260175 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:03.260093    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:03.287480 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:03.287454    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d8v5\" (UniqueName: \"kubernetes.io/projected/46ff3966-c839-4d0b-880f-0dd889cc54c3-kube-api-access-8d8v5\") pod \"maas-api-revoke-sub-4a8933b7-5h7g5\" (UID: \"46ff3966-c839-4d0b-880f-0dd889cc54c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5"
Nov 03 18:26:03.287598 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:03.287492    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ff3966-c839-4d0b-880f-0dd889cc54c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a8933b7-5h7g5\" (UID: \"46ff3966-c839-4d0b-880f-0dd889cc54c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5"
Nov 03 18:26:03.287649 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:03.287595    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ff3966-c839-4d0b-880f-0dd889cc54c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a8933b7-5h7g5\" (UID: \"46ff3966-c839-4d0b-880f-0dd889cc54c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5"
Nov 03 18:26:03.346645 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:26:03.346618    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d396930_b2b9_45df_aa1d_042dd368fef3.slice/crio-446b6c9d1e1bd492f1116c8f4d24a553b4746227ba770995f01a8f7bd84a1a18 WatchSource:0}: Error finding container 446b6c9d1e1bd492f1116c8f4d24a553b4746227ba770995f01a8f7bd84a1a18: Status 404 returned error can't find the container with id 446b6c9d1e1bd492f1116c8f4d24a553b4746227ba770995f01a8f7bd84a1a18
Nov 03 18:26:03.348246 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:03.348219    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"]
Nov 03 18:26:03.388418 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:03.388391    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ff3966-c839-4d0b-880f-0dd889cc54c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a8933b7-5h7g5\" (UID: \"46ff3966-c839-4d0b-880f-0dd889cc54c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5"
Nov 03 18:26:03.388547 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:03.388529    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d8v5\" (UniqueName: \"kubernetes.io/projected/46ff3966-c839-4d0b-880f-0dd889cc54c3-kube-api-access-8d8v5\") pod \"maas-api-revoke-sub-4a8933b7-5h7g5\" (UID: \"46ff3966-c839-4d0b-880f-0dd889cc54c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5"
Nov 03 18:26:03.388605 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:03.388574    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ff3966-c839-4d0b-880f-0dd889cc54c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a8933b7-5h7g5\" (UID: \"46ff3966-c839-4d0b-880f-0dd889cc54c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5"
Nov 03 18:26:03.389249 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:03.389123    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ff3966-c839-4d0b-880f-0dd889cc54c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a8933b7-5h7g5\" (UID: \"46ff3966-c839-4d0b-880f-0dd889cc54c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5"
Nov 03 18:26:03.390842 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:03.390823    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ff3966-c839-4d0b-880f-0dd889cc54c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a8933b7-5h7g5\" (UID: \"46ff3966-c839-4d0b-880f-0dd889cc54c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5"
Nov 03 18:26:03.406318 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:03.406291    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg" event={"ID":"2d396930-b2b9-45df-aa1d-042dd368fef3","Type":"ContainerStarted","Data":"446b6c9d1e1bd492f1116c8f4d24a553b4746227ba770995f01a8f7bd84a1a18"}
Nov 03 18:26:03.451504 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:03.451479    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d8v5\" (UniqueName: \"kubernetes.io/projected/46ff3966-c839-4d0b-880f-0dd889cc54c3-kube-api-access-8d8v5\") pod \"maas-api-revoke-sub-4a8933b7-5h7g5\" (UID: \"46ff3966-c839-4d0b-880f-0dd889cc54c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5"
Nov 03 18:26:03.497634 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:03.497577    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5"
Nov 03 18:26:03.865818 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:26:03.865771    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ff3966_c839_4d0b_880f_0dd889cc54c3.slice/crio-1255e4df8d89f59c6969d232d2aa71bdb0bf4b380d498f6798e6d0b51d299112 WatchSource:0}: Error finding container 1255e4df8d89f59c6969d232d2aa71bdb0bf4b380d498f6798e6d0b51d299112: Status 404 returned error can't find the container with id 1255e4df8d89f59c6969d232d2aa71bdb0bf4b380d498f6798e6d0b51d299112
Nov 03 18:26:03.867833 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:03.867811    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5"]
Nov 03 18:26:03.996733 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:03.996698    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-7afe86be4901: secret "agc-cred-7afe86be4901" not found
Nov 03 18:26:03.996733 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:03.996737    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz: secret "agc-cred-7afe86be4901" not found
Nov 03 18:26:03.996977 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:03.996818    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-provider-credentials podName:124f0414-5bef-4ff1-87e5-ce3b1739e468 nodeName:}" failed. No retries permitted until 2026-11-03 18:26:05.996797333 +0000 UTC m=+4106.794510110 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz" (UID: "124f0414-5bef-4ff1-87e5-ce3b1739e468") : secret "agc-cred-7afe86be4901" not found
Nov 03 18:26:04.413211 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:04.413112    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg" event={"ID":"2d396930-b2b9-45df-aa1d-042dd368fef3","Type":"ContainerStarted","Data":"e41c73121a42d43acb1d9c40c6bc183cc2e401fc4b8ad526f51539f1a12b787d"}
Nov 03 18:26:04.413417 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:04.413267    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"
Nov 03 18:26:04.414700 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:04.414675    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5" event={"ID":"46ff3966-c839-4d0b-880f-0dd889cc54c3","Type":"ContainerStarted","Data":"d302e64b466b74d1c421ef23f671e9d4f0b29ca8c5a3d9e0fad74ad64bbab936"}
Nov 03 18:26:04.414800 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:04.414716    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5" event={"ID":"46ff3966-c839-4d0b-880f-0dd889cc54c3","Type":"ContainerStarted","Data":"1255e4df8d89f59c6969d232d2aa71bdb0bf4b380d498f6798e6d0b51d299112"}
Nov 03 18:26:04.534252 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:04.534200    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg" podStartSLOduration=2.534187371 podStartE2EDuration="2.534187371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:26:04.53321378 +0000 UTC m=+4105.330926561" watchObservedRunningTime="2026-11-03 18:26:04.534187371 +0000 UTC m=+4105.331900177"
Nov 03 18:26:04.653519 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:04.653465    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5" podStartSLOduration=1.653448348 podStartE2EDuration="1.653448348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:26:04.652465 +0000 UTC m=+4105.450177780" watchObservedRunningTime="2026-11-03 18:26:04.653448348 +0000 UTC m=+4105.451161130"
Nov 03 18:26:04.996775 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:04.996686    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"]
Nov 03 18:26:05.032222 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:05.032178    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"]
Nov 03 18:26:05.036024 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:05.036003    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"
Nov 03 18:26:05.112167 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:05.112126    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"]
Nov 03 18:26:05.209870 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:05.209835    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70bb010d-98bf-46b4-83b7-a99004ee7a50-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6\" (UID: \"70bb010d-98bf-46b4-83b7-a99004ee7a50\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"
Nov 03 18:26:05.210050 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:05.209988    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xglxn\" (UniqueName: \"kubernetes.io/projected/70bb010d-98bf-46b4-83b7-a99004ee7a50-kube-api-access-xglxn\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6\" (UID: \"70bb010d-98bf-46b4-83b7-a99004ee7a50\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"
Nov 03 18:26:05.210050 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:05.210037    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70bb010d-98bf-46b4-83b7-a99004ee7a50-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6\" (UID: \"70bb010d-98bf-46b4-83b7-a99004ee7a50\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"
Nov 03 18:26:05.210129 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:05.210115    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70bb010d-98bf-46b4-83b7-a99004ee7a50-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6\" (UID: \"70bb010d-98bf-46b4-83b7-a99004ee7a50\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"
Nov 03 18:26:05.311761 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:05.311676    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70bb010d-98bf-46b4-83b7-a99004ee7a50-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6\" (UID: \"70bb010d-98bf-46b4-83b7-a99004ee7a50\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"
Nov 03 18:26:05.311902 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:05.311776    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xglxn\" (UniqueName: \"kubernetes.io/projected/70bb010d-98bf-46b4-83b7-a99004ee7a50-kube-api-access-xglxn\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6\" (UID: \"70bb010d-98bf-46b4-83b7-a99004ee7a50\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"
Nov 03 18:26:05.311902 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:05.311811    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70bb010d-98bf-46b4-83b7-a99004ee7a50-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6\" (UID: \"70bb010d-98bf-46b4-83b7-a99004ee7a50\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"
Nov 03 18:26:05.311902 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:05.311852    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70bb010d-98bf-46b4-83b7-a99004ee7a50-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6\" (UID: \"70bb010d-98bf-46b4-83b7-a99004ee7a50\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"
Nov 03 18:26:05.312544 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:05.312515    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70bb010d-98bf-46b4-83b7-a99004ee7a50-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6\" (UID: \"70bb010d-98bf-46b4-83b7-a99004ee7a50\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"
Nov 03 18:26:05.312658 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:05.312602    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70bb010d-98bf-46b4-83b7-a99004ee7a50-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6\" (UID: \"70bb010d-98bf-46b4-83b7-a99004ee7a50\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"
Nov 03 18:26:05.314249 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:05.314222    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70bb010d-98bf-46b4-83b7-a99004ee7a50-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6\" (UID: \"70bb010d-98bf-46b4-83b7-a99004ee7a50\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"
Nov 03 18:26:05.374460 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:05.374433    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xglxn\" (UniqueName: \"kubernetes.io/projected/70bb010d-98bf-46b4-83b7-a99004ee7a50-kube-api-access-xglxn\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6\" (UID: \"70bb010d-98bf-46b4-83b7-a99004ee7a50\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"
Nov 03 18:26:05.421693 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:05.421660    6059 generic.go:356] "Generic (PLEG): container finished" podID="46ff3966-c839-4d0b-880f-0dd889cc54c3" containerID="d302e64b466b74d1c421ef23f671e9d4f0b29ca8c5a3d9e0fad74ad64bbab936" exitCode=0
Nov 03 18:26:05.421868 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:05.421730    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5" event={"ID":"46ff3966-c839-4d0b-880f-0dd889cc54c3","Type":"ContainerDied","Data":"d302e64b466b74d1c421ef23f671e9d4f0b29ca8c5a3d9e0fad74ad64bbab936"}
Nov 03 18:26:05.649871 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:05.649785    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"
Nov 03 18:26:06.019215 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:06.019179    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-7afe86be4901: secret "agc-cred-7afe86be4901" not found
Nov 03 18:26:06.019345 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:06.019225    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz: secret "agc-cred-7afe86be4901" not found
Nov 03 18:26:06.019345 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:06.019303    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-provider-credentials podName:124f0414-5bef-4ff1-87e5-ce3b1739e468 nodeName:}" failed. No retries permitted until 2026-11-03 18:26:10.019281551 +0000 UTC m=+4110.816994328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz" (UID: "124f0414-5bef-4ff1-87e5-ce3b1739e468") : secret "agc-cred-7afe86be4901" not found
Nov 03 18:26:06.022200 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:26:06.022162    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bb010d_98bf_46b4_83b7_a99004ee7a50.slice/crio-a6ec8ae150e9477c2ec14c5148780d1034bc250675006fa6116df28a76fcb44a WatchSource:0}: Error finding container a6ec8ae150e9477c2ec14c5148780d1034bc250675006fa6116df28a76fcb44a: Status 404 returned error can't find the container with id a6ec8ae150e9477c2ec14c5148780d1034bc250675006fa6116df28a76fcb44a
Nov 03 18:26:06.023348 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:06.023324    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"]
Nov 03 18:26:06.429190 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:06.429104    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6" event={"ID":"70bb010d-98bf-46b4-83b7-a99004ee7a50","Type":"ContainerStarted","Data":"e1d8b42d798dea6883f40c7f4ce04514effb1efa3884e29775b3a1c26c631a0b"}
Nov 03 18:26:06.429190 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:06.429161    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6" event={"ID":"70bb010d-98bf-46b4-83b7-a99004ee7a50","Type":"ContainerStarted","Data":"a6ec8ae150e9477c2ec14c5148780d1034bc250675006fa6116df28a76fcb44a"}
Nov 03 18:26:06.429447 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:06.429381    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"
Nov 03 18:26:06.550727 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:06.550668    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6" podStartSLOduration=2.550651523 podStartE2EDuration="2.550651523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:26:06.549009351 +0000 UTC m=+4107.346722158" watchObservedRunningTime="2026-11-03 18:26:06.550651523 +0000 UTC m=+4107.348364308"
Nov 03 18:26:06.572187 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:06.572163    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5"
Nov 03 18:26:06.727020 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:06.726988    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46ff3966-c839-4d0b-880f-0dd889cc54c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ff3966-c839-4d0b-880f-0dd889cc54c3-maas-api-xks-ca\") pod \"46ff3966-c839-4d0b-880f-0dd889cc54c3\" (UID: \"46ff3966-c839-4d0b-880f-0dd889cc54c3\") "
Nov 03 18:26:06.727454 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:06.727082    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ff3966-c839-4d0b-880f-0dd889cc54c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ff3966-c839-4d0b-880f-0dd889cc54c3-maas-api-openshift-ca\") pod \"46ff3966-c839-4d0b-880f-0dd889cc54c3\" (UID: \"46ff3966-c839-4d0b-880f-0dd889cc54c3\") "
Nov 03 18:26:06.727454 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:06.727116    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ff3966-c839-4d0b-880f-0dd889cc54c3-kube-api-access-8d8v5\" (UniqueName: \"kubernetes.io/projected/46ff3966-c839-4d0b-880f-0dd889cc54c3-kube-api-access-8d8v5\") pod \"46ff3966-c839-4d0b-880f-0dd889cc54c3\" (UID: \"46ff3966-c839-4d0b-880f-0dd889cc54c3\") "
Nov 03 18:26:06.727583 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:06.727486    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ff3966-c839-4d0b-880f-0dd889cc54c3-maas-api-openshift-ca" pod "46ff3966-c839-4d0b-880f-0dd889cc54c3" (UID: "46ff3966-c839-4d0b-880f-0dd889cc54c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:26:06.729134 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:06.729116    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ff3966-c839-4d0b-880f-0dd889cc54c3-maas-api-xks-ca" pod "46ff3966-c839-4d0b-880f-0dd889cc54c3" (UID: "46ff3966-c839-4d0b-880f-0dd889cc54c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:26:06.729230 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:06.729212    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ff3966-c839-4d0b-880f-0dd889cc54c3-kube-api-access-8d8v5" pod "46ff3966-c839-4d0b-880f-0dd889cc54c3" (UID: "46ff3966-c839-4d0b-880f-0dd889cc54c3"). InnerVolumeSpecName "kube-api-access-8d8v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:26:06.828349 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:06.828318    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ff3966-c839-4d0b-880f-0dd889cc54c3-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:06.828349 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:06.828343    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ff3966-c839-4d0b-880f-0dd889cc54c3-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:06.828349 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:06.828375    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d8v5\" (UniqueName: \"kubernetes.io/projected/46ff3966-c839-4d0b-880f-0dd889cc54c3-kube-api-access-8d8v5\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:06.860228 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:06.860192    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:26:07.436079 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:07.436045    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5"
Nov 03 18:26:07.436252 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:07.436077    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5" event={"ID":"46ff3966-c839-4d0b-880f-0dd889cc54c3","Type":"ContainerDied","Data":"1255e4df8d89f59c6969d232d2aa71bdb0bf4b380d498f6798e6d0b51d299112"}
Nov 03 18:26:07.436252 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:07.436125    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1255e4df8d89f59c6969d232d2aa71bdb0bf4b380d498f6798e6d0b51d299112"
Nov 03 18:26:07.620638 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:07.620513    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r" podUID="598f1c49-a188-4e50-9c6f-75b4468a126e" containerName="payload-processing" containerID="cri-o://ab0b546f8cc1cd6852e66d1267803de873d018a42acf1309ca904490924c8c99" gracePeriod=25
Nov 03 18:26:07.874543 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:07.874478    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"
Nov 03 18:26:08.042296 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:08.042266    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/598f1c49-a188-4e50-9c6f-75b4468a126e-kube-api-access-c7s9b\" (UniqueName: \"kubernetes.io/projected/598f1c49-a188-4e50-9c6f-75b4468a126e-kube-api-access-c7s9b\") pod \"598f1c49-a188-4e50-9c6f-75b4468a126e\" (UID: \"598f1c49-a188-4e50-9c6f-75b4468a126e\") "
Nov 03 18:26:08.042496 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:08.042349    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/598f1c49-a188-4e50-9c6f-75b4468a126e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/598f1c49-a188-4e50-9c6f-75b4468a126e-routing-overlay-volume\") pod \"598f1c49-a188-4e50-9c6f-75b4468a126e\" (UID: \"598f1c49-a188-4e50-9c6f-75b4468a126e\") "
Nov 03 18:26:08.042496 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:08.042392    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/598f1c49-a188-4e50-9c6f-75b4468a126e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/598f1c49-a188-4e50-9c6f-75b4468a126e-plugins-config-volume\") pod \"598f1c49-a188-4e50-9c6f-75b4468a126e\" (UID: \"598f1c49-a188-4e50-9c6f-75b4468a126e\") "
Nov 03 18:26:08.042496 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:08.042419    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/598f1c49-a188-4e50-9c6f-75b4468a126e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/598f1c49-a188-4e50-9c6f-75b4468a126e-provider-credentials\") pod \"598f1c49-a188-4e50-9c6f-75b4468a126e\" (UID: \"598f1c49-a188-4e50-9c6f-75b4468a126e\") "
Nov 03 18:26:08.042821 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:08.042722    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/598f1c49-a188-4e50-9c6f-75b4468a126e-routing-overlay-volume" pod "598f1c49-a188-4e50-9c6f-75b4468a126e" (UID: "598f1c49-a188-4e50-9c6f-75b4468a126e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:26:08.042821 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:08.042784    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/598f1c49-a188-4e50-9c6f-75b4468a126e-plugins-config-volume" pod "598f1c49-a188-4e50-9c6f-75b4468a126e" (UID: "598f1c49-a188-4e50-9c6f-75b4468a126e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:26:08.042980 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:08.042852    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/598f1c49-a188-4e50-9c6f-75b4468a126e-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:08.044565 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:08.044542    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598f1c49-a188-4e50-9c6f-75b4468a126e-provider-credentials" pod "598f1c49-a188-4e50-9c6f-75b4468a126e" (UID: "598f1c49-a188-4e50-9c6f-75b4468a126e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:26:08.044664 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:08.044587    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598f1c49-a188-4e50-9c6f-75b4468a126e-kube-api-access-c7s9b" pod "598f1c49-a188-4e50-9c6f-75b4468a126e" (UID: "598f1c49-a188-4e50-9c6f-75b4468a126e"). InnerVolumeSpecName "kube-api-access-c7s9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:26:08.054010 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:08.053983    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:08.144381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:08.144269    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7s9b\" (UniqueName: \"kubernetes.io/projected/598f1c49-a188-4e50-9c6f-75b4468a126e-kube-api-access-c7s9b\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:08.144381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:08.144298    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/598f1c49-a188-4e50-9c6f-75b4468a126e-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:08.144381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:08.144313    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/598f1c49-a188-4e50-9c6f-75b4468a126e-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:08.442751 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:08.442671    6059 generic.go:356] "Generic (PLEG): container finished" podID="598f1c49-a188-4e50-9c6f-75b4468a126e" containerID="ab0b546f8cc1cd6852e66d1267803de873d018a42acf1309ca904490924c8c99" exitCode=0
Nov 03 18:26:08.442751 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:08.442722    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"
Nov 03 18:26:08.442751 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:08.442730    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r" event={"ID":"598f1c49-a188-4e50-9c6f-75b4468a126e","Type":"ContainerDied","Data":"ab0b546f8cc1cd6852e66d1267803de873d018a42acf1309ca904490924c8c99"}
Nov 03 18:26:08.442987 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:08.442778    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r" event={"ID":"598f1c49-a188-4e50-9c6f-75b4468a126e","Type":"ContainerDied","Data":"faca15a4f1ec2c20214c37819119dc9606fd966da951ab3a9cf293b80a88b209"}
Nov 03 18:26:08.442987 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:08.442802    6059 scope.go:122] "RemoveContainer" containerID="ab0b546f8cc1cd6852e66d1267803de873d018a42acf1309ca904490924c8c99"
Nov 03 18:26:08.452515 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:08.452498    6059 scope.go:122] "RemoveContainer" containerID="ab0b546f8cc1cd6852e66d1267803de873d018a42acf1309ca904490924c8c99"
Nov 03 18:26:08.452773 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:08.452744    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0b546f8cc1cd6852e66d1267803de873d018a42acf1309ca904490924c8c99\": container with ID starting with ab0b546f8cc1cd6852e66d1267803de873d018a42acf1309ca904490924c8c99 not found: ID does not exist" containerID="ab0b546f8cc1cd6852e66d1267803de873d018a42acf1309ca904490924c8c99"
Nov 03 18:26:08.452854 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:08.452784    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0b546f8cc1cd6852e66d1267803de873d018a42acf1309ca904490924c8c99"} err="failed to get container status \"ab0b546f8cc1cd6852e66d1267803de873d018a42acf1309ca904490924c8c99\": rpc error: code = NotFound desc = could not find container \"ab0b546f8cc1cd6852e66d1267803de873d018a42acf1309ca904490924c8c99\": container with ID starting with ab0b546f8cc1cd6852e66d1267803de873d018a42acf1309ca904490924c8c99 not found: ID does not exist"
Nov 03 18:26:08.612178 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:08.612138    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"]
Nov 03 18:26:08.663731 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:08.663698    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-c7f54zr9r"]
Nov 03 18:26:09.336594 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:09.336561    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="598f1c49-a188-4e50-9c6f-75b4468a126e" path="/var/lib/kubelet/pods/598f1c49-a188-4e50-9c6f-75b4468a126e/volumes"
Nov 03 18:26:09.997152 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:09.997117    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz" podUID="124f0414-5bef-4ff1-87e5-ce3b1739e468" containerName="payload-processing" containerID="cri-o://d741ea0b361b354f238c3cccd5b9ac8b239314531ffcf5d6f92b9a1d0ba4f71b" gracePeriod=25
Nov 03 18:26:10.063723 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:10.063694    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-7afe86be4901: secret "agc-cred-7afe86be4901" not found
Nov 03 18:26:10.063723 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:10.063724    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz: secret "agc-cred-7afe86be4901" not found
Nov 03 18:26:10.063895 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:10.063782    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-provider-credentials podName:124f0414-5bef-4ff1-87e5-ce3b1739e468 nodeName:}" failed. No retries permitted until 2026-11-03 18:26:18.063767117 +0000 UTC m=+4118.861479876 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz" (UID: "124f0414-5bef-4ff1-87e5-ce3b1739e468") : secret "agc-cred-7afe86be4901" not found
Nov 03 18:26:10.245280 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:10.245256    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"
Nov 03 18:26:10.265617 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:10.265567    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/124f0414-5bef-4ff1-87e5-ce3b1739e468-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/124f0414-5bef-4ff1-87e5-ce3b1739e468-routing-overlay-volume\") pod \"124f0414-5bef-4ff1-87e5-ce3b1739e468\" (UID: \"124f0414-5bef-4ff1-87e5-ce3b1739e468\") "
Nov 03 18:26:10.265713 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:10.265649    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/124f0414-5bef-4ff1-87e5-ce3b1739e468-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/124f0414-5bef-4ff1-87e5-ce3b1739e468-plugins-config-volume\") pod \"124f0414-5bef-4ff1-87e5-ce3b1739e468\" (UID: \"124f0414-5bef-4ff1-87e5-ce3b1739e468\") "
Nov 03 18:26:10.265713 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:10.265700    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-kube-api-access-ln2tl\" (UniqueName: \"kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-kube-api-access-ln2tl\") pod \"124f0414-5bef-4ff1-87e5-ce3b1739e468\" (UID: \"124f0414-5bef-4ff1-87e5-ce3b1739e468\") "
Nov 03 18:26:10.265836 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:10.265765    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-provider-credentials\" (UniqueName: \"kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-provider-credentials\") pod \"124f0414-5bef-4ff1-87e5-ce3b1739e468\" (UID: \"124f0414-5bef-4ff1-87e5-ce3b1739e468\") "
Nov 03 18:26:10.265976 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:10.265949    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/124f0414-5bef-4ff1-87e5-ce3b1739e468-routing-overlay-volume" pod "124f0414-5bef-4ff1-87e5-ce3b1739e468" (UID: "124f0414-5bef-4ff1-87e5-ce3b1739e468"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:26:10.266075 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:10.266055    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/124f0414-5bef-4ff1-87e5-ce3b1739e468-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:10.266129 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:10.266065    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/124f0414-5bef-4ff1-87e5-ce3b1739e468-plugins-config-volume" pod "124f0414-5bef-4ff1-87e5-ce3b1739e468" (UID: "124f0414-5bef-4ff1-87e5-ce3b1739e468"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:26:10.267798 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:10.267777    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-kube-api-access-ln2tl" pod "124f0414-5bef-4ff1-87e5-ce3b1739e468" (UID: "124f0414-5bef-4ff1-87e5-ce3b1739e468"). InnerVolumeSpecName "kube-api-access-ln2tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:26:10.267889 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:10.267809    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-provider-credentials" pod "124f0414-5bef-4ff1-87e5-ce3b1739e468" (UID: "124f0414-5bef-4ff1-87e5-ce3b1739e468"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:26:10.367259 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:10.367234    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln2tl\" (UniqueName: \"kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-kube-api-access-ln2tl\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:10.367259 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:10.367259    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/124f0414-5bef-4ff1-87e5-ce3b1739e468-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:10.367653 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:10.367269    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/124f0414-5bef-4ff1-87e5-ce3b1739e468-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:10.456735 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:10.456700    6059 generic.go:356] "Generic (PLEG): container finished" podID="124f0414-5bef-4ff1-87e5-ce3b1739e468" containerID="d741ea0b361b354f238c3cccd5b9ac8b239314531ffcf5d6f92b9a1d0ba4f71b" exitCode=0
Nov 03 18:26:10.456916 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:10.456752    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"
Nov 03 18:26:10.456916 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:10.456767    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz" event={"ID":"124f0414-5bef-4ff1-87e5-ce3b1739e468","Type":"ContainerDied","Data":"d741ea0b361b354f238c3cccd5b9ac8b239314531ffcf5d6f92b9a1d0ba4f71b"}
Nov 03 18:26:10.456916 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:10.456810    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz" event={"ID":"124f0414-5bef-4ff1-87e5-ce3b1739e468","Type":"ContainerDied","Data":"0a5ef8f3c509adba2ef7a544e5768b72ebccc4d18c105d4bce35bd4e8f0f1c17"}
Nov 03 18:26:10.456916 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:10.456831    6059 scope.go:122] "RemoveContainer" containerID="d741ea0b361b354f238c3cccd5b9ac8b239314531ffcf5d6f92b9a1d0ba4f71b"
Nov 03 18:26:10.466624 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:10.466604    6059 scope.go:122] "RemoveContainer" containerID="d741ea0b361b354f238c3cccd5b9ac8b239314531ffcf5d6f92b9a1d0ba4f71b"
Nov 03 18:26:10.466890 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:10.466872    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d741ea0b361b354f238c3cccd5b9ac8b239314531ffcf5d6f92b9a1d0ba4f71b\": container with ID starting with d741ea0b361b354f238c3cccd5b9ac8b239314531ffcf5d6f92b9a1d0ba4f71b not found: ID does not exist" containerID="d741ea0b361b354f238c3cccd5b9ac8b239314531ffcf5d6f92b9a1d0ba4f71b"
Nov 03 18:26:10.466950 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:10.466902    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d741ea0b361b354f238c3cccd5b9ac8b239314531ffcf5d6f92b9a1d0ba4f71b"} err="failed to get container status \"d741ea0b361b354f238c3cccd5b9ac8b239314531ffcf5d6f92b9a1d0ba4f71b\": rpc error: code = NotFound desc = could not find container \"d741ea0b361b354f238c3cccd5b9ac8b239314531ffcf5d6f92b9a1d0ba4f71b\": container with ID starting with d741ea0b361b354f238c3cccd5b9ac8b239314531ffcf5d6f92b9a1d0ba4f71b not found: ID does not exist"
Nov 03 18:26:10.627253 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:10.627174    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"]
Nov 03 18:26:10.631939 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:10.631914    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-8674g66fz"]
Nov 03 18:26:11.334580 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:11.334548    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="124f0414-5bef-4ff1-87e5-ce3b1739e468" path="/var/lib/kubelet/pods/124f0414-5bef-4ff1-87e5-ce3b1739e468/volumes"
Nov 03 18:26:13.052688 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:13.052626    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:14.713040 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.713002    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff6d6a8e-v7tvz"]
Nov 03 18:26:14.713514 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.713498    6059 cpu_manager.go:414] "containerMap: removing container" podUID="124f0414-5bef-4ff1-87e5-ce3b1739e468" containerName="payload-processing"
Nov 03 18:26:14.713567 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.713515    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="124f0414-5bef-4ff1-87e5-ce3b1739e468" containerName="payload-processing"
Nov 03 18:26:14.713567 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.713543    6059 cpu_manager.go:414] "containerMap: removing container" podUID="46ff3966-c839-4d0b-880f-0dd889cc54c3" containerName="revoke-keys"
Nov 03 18:26:14.713567 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.713548    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ff3966-c839-4d0b-880f-0dd889cc54c3" containerName="revoke-keys"
Nov 03 18:26:14.713567 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.713563    6059 cpu_manager.go:414] "containerMap: removing container" podUID="598f1c49-a188-4e50-9c6f-75b4468a126e" containerName="payload-processing"
Nov 03 18:26:14.713567 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.713567    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="598f1c49-a188-4e50-9c6f-75b4468a126e" containerName="payload-processing"
Nov 03 18:26:14.713727 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.713640    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="598f1c49-a188-4e50-9c6f-75b4468a126e" containerName="payload-processing"
Nov 03 18:26:14.713727 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.713651    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="124f0414-5bef-4ff1-87e5-ce3b1739e468" containerName="payload-processing"
Nov 03 18:26:14.713727 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.713662    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ff3966-c839-4d0b-880f-0dd889cc54c3" containerName="revoke-keys"
Nov 03 18:26:14.716889 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.716874    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6d6a8e-v7tvz"
Nov 03 18:26:14.773442 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.773419    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:26:14.807951 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.807916    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f24b5a6-eb74-4310-8cea-8645fd109b8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff6d6a8e-v7tvz\" (UID: \"0f24b5a6-eb74-4310-8cea-8645fd109b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6d6a8e-v7tvz"
Nov 03 18:26:14.808116 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.808062    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnnw5\" (UniqueName: \"kubernetes.io/projected/0f24b5a6-eb74-4310-8cea-8645fd109b8e-kube-api-access-tnnw5\") pod \"maas-api-revoke-sub-ff6d6a8e-v7tvz\" (UID: \"0f24b5a6-eb74-4310-8cea-8645fd109b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6d6a8e-v7tvz"
Nov 03 18:26:14.808196 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.808178    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f24b5a6-eb74-4310-8cea-8645fd109b8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff6d6a8e-v7tvz\" (UID: \"0f24b5a6-eb74-4310-8cea-8645fd109b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6d6a8e-v7tvz"
Nov 03 18:26:14.825154 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.825080    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:26:14.825154 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.825082    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:26:14.825154 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.825140    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 18:26:14.827182 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.827165    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff6d6a8e-v7tvz"]
Nov 03 18:26:14.848233 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.848209    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:14.909629 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.909603    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnnw5\" (UniqueName: \"kubernetes.io/projected/0f24b5a6-eb74-4310-8cea-8645fd109b8e-kube-api-access-tnnw5\") pod \"maas-api-revoke-sub-ff6d6a8e-v7tvz\" (UID: \"0f24b5a6-eb74-4310-8cea-8645fd109b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6d6a8e-v7tvz"
Nov 03 18:26:14.909807 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.909649    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f24b5a6-eb74-4310-8cea-8645fd109b8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff6d6a8e-v7tvz\" (UID: \"0f24b5a6-eb74-4310-8cea-8645fd109b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6d6a8e-v7tvz"
Nov 03 18:26:14.909868 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.909828    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f24b5a6-eb74-4310-8cea-8645fd109b8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff6d6a8e-v7tvz\" (UID: \"0f24b5a6-eb74-4310-8cea-8645fd109b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6d6a8e-v7tvz"
Nov 03 18:26:14.910548 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.910524    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f24b5a6-eb74-4310-8cea-8645fd109b8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff6d6a8e-v7tvz\" (UID: \"0f24b5a6-eb74-4310-8cea-8645fd109b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6d6a8e-v7tvz"
Nov 03 18:26:14.912194 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.912167    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f24b5a6-eb74-4310-8cea-8645fd109b8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff6d6a8e-v7tvz\" (UID: \"0f24b5a6-eb74-4310-8cea-8645fd109b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6d6a8e-v7tvz"
Nov 03 18:26:14.972447 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:14.972379    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnnw5\" (UniqueName: \"kubernetes.io/projected/0f24b5a6-eb74-4310-8cea-8645fd109b8e-kube-api-access-tnnw5\") pod \"maas-api-revoke-sub-ff6d6a8e-v7tvz\" (UID: \"0f24b5a6-eb74-4310-8cea-8645fd109b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6d6a8e-v7tvz"
Nov 03 18:26:15.030036 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:15.030005    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6d6a8e-v7tvz"
Nov 03 18:26:15.399168 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:26:15.399133    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f24b5a6_eb74_4310_8cea_8645fd109b8e.slice/crio-584fecedbf6b8d68e8b75491e68233133b9f772197f10b440aa292cef4ba48a1 WatchSource:0}: Error finding container 584fecedbf6b8d68e8b75491e68233133b9f772197f10b440aa292cef4ba48a1: Status 404 returned error can't find the container with id 584fecedbf6b8d68e8b75491e68233133b9f772197f10b440aa292cef4ba48a1
Nov 03 18:26:15.402414 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:15.402385    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff6d6a8e-v7tvz"]
Nov 03 18:26:15.424896 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:15.424869    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg" podUID="2d396930-b2b9-45df-aa1d-042dd368fef3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:26:15.488058 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:15.488023    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6d6a8e-v7tvz" event={"ID":"0f24b5a6-eb74-4310-8cea-8645fd109b8e","Type":"ContainerStarted","Data":"584fecedbf6b8d68e8b75491e68233133b9f772197f10b440aa292cef4ba48a1"}
Nov 03 18:26:16.495957 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:16.495923    6059 generic.go:356] "Generic (PLEG): container finished" podID="0f24b5a6-eb74-4310-8cea-8645fd109b8e" containerID="9c7c5804008f716ca0833f3afd72e0f1f3c525b7672020ffa6fd78870dbd71c7" exitCode=0
Nov 03 18:26:16.496305 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:16.496013    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6d6a8e-v7tvz" event={"ID":"0f24b5a6-eb74-4310-8cea-8645fd109b8e","Type":"ContainerDied","Data":"9c7c5804008f716ca0833f3afd72e0f1f3c525b7672020ffa6fd78870dbd71c7"}
Nov 03 18:26:17.439188 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:17.439149    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6" podUID="70bb010d-98bf-46b4-83b7-a99004ee7a50" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:26:17.641907 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:17.641884    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6d6a8e-v7tvz"
Nov 03 18:26:17.739098 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:17.739064    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f24b5a6-eb74-4310-8cea-8645fd109b8e-kube-api-access-tnnw5\" (UniqueName: \"kubernetes.io/projected/0f24b5a6-eb74-4310-8cea-8645fd109b8e-kube-api-access-tnnw5\") pod \"0f24b5a6-eb74-4310-8cea-8645fd109b8e\" (UID: \"0f24b5a6-eb74-4310-8cea-8645fd109b8e\") "
Nov 03 18:26:17.739307 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:17.739112    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f24b5a6-eb74-4310-8cea-8645fd109b8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f24b5a6-eb74-4310-8cea-8645fd109b8e-maas-api-openshift-ca\") pod \"0f24b5a6-eb74-4310-8cea-8645fd109b8e\" (UID: \"0f24b5a6-eb74-4310-8cea-8645fd109b8e\") "
Nov 03 18:26:17.739307 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:17.739239    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f24b5a6-eb74-4310-8cea-8645fd109b8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f24b5a6-eb74-4310-8cea-8645fd109b8e-maas-api-xks-ca\") pod \"0f24b5a6-eb74-4310-8cea-8645fd109b8e\" (UID: \"0f24b5a6-eb74-4310-8cea-8645fd109b8e\") "
Nov 03 18:26:17.739509 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:17.739484    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f24b5a6-eb74-4310-8cea-8645fd109b8e-maas-api-openshift-ca" pod "0f24b5a6-eb74-4310-8cea-8645fd109b8e" (UID: "0f24b5a6-eb74-4310-8cea-8645fd109b8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:26:17.739635 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:17.739617    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f24b5a6-eb74-4310-8cea-8645fd109b8e-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:17.741200 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:17.741180    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f24b5a6-eb74-4310-8cea-8645fd109b8e-maas-api-xks-ca" pod "0f24b5a6-eb74-4310-8cea-8645fd109b8e" (UID: "0f24b5a6-eb74-4310-8cea-8645fd109b8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:26:17.741285 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:17.741266    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f24b5a6-eb74-4310-8cea-8645fd109b8e-kube-api-access-tnnw5" pod "0f24b5a6-eb74-4310-8cea-8645fd109b8e" (UID: "0f24b5a6-eb74-4310-8cea-8645fd109b8e"). InnerVolumeSpecName "kube-api-access-tnnw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:26:17.840824 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:17.840792    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnnw5\" (UniqueName: \"kubernetes.io/projected/0f24b5a6-eb74-4310-8cea-8645fd109b8e-kube-api-access-tnnw5\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:17.840824 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:17.840820    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f24b5a6-eb74-4310-8cea-8645fd109b8e-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:17.855173 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:17.855145    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:18.508673 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:18.508641    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6d6a8e-v7tvz"
Nov 03 18:26:18.508829 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:18.508673    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff6d6a8e-v7tvz" event={"ID":"0f24b5a6-eb74-4310-8cea-8645fd109b8e","Type":"ContainerDied","Data":"584fecedbf6b8d68e8b75491e68233133b9f772197f10b440aa292cef4ba48a1"}
Nov 03 18:26:18.508829 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:18.508743    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="584fecedbf6b8d68e8b75491e68233133b9f772197f10b440aa292cef4ba48a1"
Nov 03 18:26:22.659746 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:22.659702    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:25.423340 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:25.423301    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg" podUID="2d396930-b2b9-45df-aa1d-042dd368fef3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:26:27.420958 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:27.420919    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:27.438324 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:27.438295    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6" podUID="70bb010d-98bf-46b4-83b7-a99004ee7a50" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:26:27.614854 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:27.614823    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:27.797190 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:27.797154    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:28.371491 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.371456    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg"]
Nov 03 18:26:28.371946 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.371932    6059 cpu_manager.go:414] "containerMap: removing container" podUID="0f24b5a6-eb74-4310-8cea-8645fd109b8e" containerName="revoke-keys"
Nov 03 18:26:28.371994 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.371948    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f24b5a6-eb74-4310-8cea-8645fd109b8e" containerName="revoke-keys"
Nov 03 18:26:28.372036 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.372023    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f24b5a6-eb74-4310-8cea-8645fd109b8e" containerName="revoke-keys"
Nov 03 18:26:28.375883 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.375861    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg"
Nov 03 18:26:28.377484 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.377458    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"]
Nov 03 18:26:28.378710 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.378683    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg" podUID="2d396930-b2b9-45df-aa1d-042dd368fef3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:26:28.422598 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.422561    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"]
Nov 03 18:26:28.426592 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.426570    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"
Nov 03 18:26:28.433817 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.433792    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:26:28.434087 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.434064    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:26:28.434087 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.434080    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 18:26:28.444794 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:28.444771    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-6244ce06c11a: secret "agc-cred-6244ce06c11a" not found
Nov 03 18:26:28.444794 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:28.444793    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-7afe86be4901: secret "agc-cred-7afe86be4901" not found
Nov 03 18:26:28.444975 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:28.444806    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-d13ad376f3be: secret "agc-cred-d13ad376f3be" not found
Nov 03 18:26:28.444975 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:28.444826    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2: [secret "agc-cred-6244ce06c11a" not found, secret "agc-cred-7afe86be4901" not found, secret "agc-cred-d13ad376f3be" not found]
Nov 03 18:26:28.444975 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.444831    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a59cf8bf-640b-4a56-8f65-6e1ebf380caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d2fd950-9bkhg\" (UID: \"a59cf8bf-640b-4a56-8f65-6e1ebf380caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg"
Nov 03 18:26:28.444975 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:28.444884    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-provider-credentials podName:ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6 nodeName:}" failed. No retries permitted until 2026-11-03 18:26:28.944863279 +0000 UTC m=+4129.742576038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2" (UID: "ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6") : [secret "agc-cred-6244ce06c11a" not found, secret "agc-cred-7afe86be4901" not found, secret "agc-cred-d13ad376f3be" not found]
Nov 03 18:26:28.445253 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.445024    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a59cf8bf-640b-4a56-8f65-6e1ebf380caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d2fd950-9bkhg\" (UID: \"a59cf8bf-640b-4a56-8f65-6e1ebf380caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg"
Nov 03 18:26:28.445253 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.445151    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnhpq\" (UniqueName: \"kubernetes.io/projected/a59cf8bf-640b-4a56-8f65-6e1ebf380caf-kube-api-access-qnhpq\") pod \"maas-api-revoke-sub-2d2fd950-9bkhg\" (UID: \"a59cf8bf-640b-4a56-8f65-6e1ebf380caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg"
Nov 03 18:26:28.471873 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.471839    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:26:28.474336 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.474310    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg"]
Nov 03 18:26:28.487852 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.487823    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"]
Nov 03 18:26:28.546070 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.546039    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a59cf8bf-640b-4a56-8f65-6e1ebf380caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d2fd950-9bkhg\" (UID: \"a59cf8bf-640b-4a56-8f65-6e1ebf380caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg"
Nov 03 18:26:28.546246 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.546090    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a59cf8bf-640b-4a56-8f65-6e1ebf380caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d2fd950-9bkhg\" (UID: \"a59cf8bf-640b-4a56-8f65-6e1ebf380caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg"
Nov 03 18:26:28.546246 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.546130    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnhpq\" (UniqueName: \"kubernetes.io/projected/a59cf8bf-640b-4a56-8f65-6e1ebf380caf-kube-api-access-qnhpq\") pod \"maas-api-revoke-sub-2d2fd950-9bkhg\" (UID: \"a59cf8bf-640b-4a56-8f65-6e1ebf380caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg"
Nov 03 18:26:28.546246 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.546178    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr\" (UID: \"7cba1dd3-36bd-4832-a4cb-cd3537ac1147\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"
Nov 03 18:26:28.546246 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.546206    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr\" (UID: \"7cba1dd3-36bd-4832-a4cb-cd3537ac1147\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"
Nov 03 18:26:28.546246 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.546232    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4hkj\" (UniqueName: \"kubernetes.io/projected/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-kube-api-access-x4hkj\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr\" (UID: \"7cba1dd3-36bd-4832-a4cb-cd3537ac1147\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"
Nov 03 18:26:28.546512 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.546264    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr\" (UID: \"7cba1dd3-36bd-4832-a4cb-cd3537ac1147\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"
Nov 03 18:26:28.546820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.546797    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a59cf8bf-640b-4a56-8f65-6e1ebf380caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d2fd950-9bkhg\" (UID: \"a59cf8bf-640b-4a56-8f65-6e1ebf380caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg"
Nov 03 18:26:28.548419 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.548402    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a59cf8bf-640b-4a56-8f65-6e1ebf380caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d2fd950-9bkhg\" (UID: \"a59cf8bf-640b-4a56-8f65-6e1ebf380caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg"
Nov 03 18:26:28.609381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.609339    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnhpq\" (UniqueName: \"kubernetes.io/projected/a59cf8bf-640b-4a56-8f65-6e1ebf380caf-kube-api-access-qnhpq\") pod \"maas-api-revoke-sub-2d2fd950-9bkhg\" (UID: \"a59cf8bf-640b-4a56-8f65-6e1ebf380caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg"
Nov 03 18:26:28.647839 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.647762    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr\" (UID: \"7cba1dd3-36bd-4832-a4cb-cd3537ac1147\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"
Nov 03 18:26:28.647982 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.647901    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr\" (UID: \"7cba1dd3-36bd-4832-a4cb-cd3537ac1147\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"
Nov 03 18:26:28.647982 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.647921    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr\" (UID: \"7cba1dd3-36bd-4832-a4cb-cd3537ac1147\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"
Nov 03 18:26:28.647982 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.647950    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4hkj\" (UniqueName: \"kubernetes.io/projected/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-kube-api-access-x4hkj\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr\" (UID: \"7cba1dd3-36bd-4832-a4cb-cd3537ac1147\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"
Nov 03 18:26:28.648613 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.648579    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr\" (UID: \"7cba1dd3-36bd-4832-a4cb-cd3537ac1147\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"
Nov 03 18:26:28.648761 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.648657    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr\" (UID: \"7cba1dd3-36bd-4832-a4cb-cd3537ac1147\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"
Nov 03 18:26:28.650350 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.650324    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr\" (UID: \"7cba1dd3-36bd-4832-a4cb-cd3537ac1147\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"
Nov 03 18:26:28.689254 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.689221    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg"
Nov 03 18:26:28.711151 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.711124    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4hkj\" (UniqueName: \"kubernetes.io/projected/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-kube-api-access-x4hkj\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr\" (UID: \"7cba1dd3-36bd-4832-a4cb-cd3537ac1147\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"
Nov 03 18:26:28.740105 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:28.740077    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"
Nov 03 18:26:28.952687 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:28.952612    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-6244ce06c11a: secret "agc-cred-6244ce06c11a" not found
Nov 03 18:26:28.952687 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:28.952643    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-7afe86be4901: secret "agc-cred-7afe86be4901" not found
Nov 03 18:26:28.952687 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:28.952650    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-d13ad376f3be: secret "agc-cred-d13ad376f3be" not found
Nov 03 18:26:28.952687 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:28.952663    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2: [secret "agc-cred-6244ce06c11a" not found, secret "agc-cred-7afe86be4901" not found, secret "agc-cred-d13ad376f3be" not found]
Nov 03 18:26:28.952928 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:28.952717    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-provider-credentials podName:ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6 nodeName:}" failed. No retries permitted until 2026-11-03 18:26:29.952702359 +0000 UTC m=+4130.750415138 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2" (UID: "ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6") : [secret "agc-cred-6244ce06c11a" not found, secret "agc-cred-7afe86be4901" not found, secret "agc-cred-d13ad376f3be" not found]
Nov 03 18:26:29.062111 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:26:29.062077    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda59cf8bf_640b_4a56_8f65_6e1ebf380caf.slice/crio-d3b56cbfe49c99a701bbbc3ce3f958a042fdf1adadf889892d0bbed9369a5aa9 WatchSource:0}: Error finding container d3b56cbfe49c99a701bbbc3ce3f958a042fdf1adadf889892d0bbed9369a5aa9: Status 404 returned error can't find the container with id d3b56cbfe49c99a701bbbc3ce3f958a042fdf1adadf889892d0bbed9369a5aa9
Nov 03 18:26:29.063955 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:29.063933    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg"]
Nov 03 18:26:29.170446 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:26:29.170417    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cba1dd3_36bd_4832_a4cb_cd3537ac1147.slice/crio-aa4b660629d493db91c81eeda77955ba2496451e69990136d1a150328b8b5c47 WatchSource:0}: Error finding container aa4b660629d493db91c81eeda77955ba2496451e69990136d1a150328b8b5c47: Status 404 returned error can't find the container with id aa4b660629d493db91c81eeda77955ba2496451e69990136d1a150328b8b5c47
Nov 03 18:26:29.172487 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:29.172444    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"]
Nov 03 18:26:29.578773 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:29.578678    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr" event={"ID":"7cba1dd3-36bd-4832-a4cb-cd3537ac1147","Type":"ContainerStarted","Data":"a05465577a0977a4cf6cc9c06364c04871eb8e2e8b2422441c8065ba4f841dfc"}
Nov 03 18:26:29.578773 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:29.578742    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr" event={"ID":"7cba1dd3-36bd-4832-a4cb-cd3537ac1147","Type":"ContainerStarted","Data":"aa4b660629d493db91c81eeda77955ba2496451e69990136d1a150328b8b5c47"}
Nov 03 18:26:29.579329 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:29.578883    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"
Nov 03 18:26:29.580453 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:29.580429    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg" event={"ID":"a59cf8bf-640b-4a56-8f65-6e1ebf380caf","Type":"ContainerStarted","Data":"e2aa24d2a4f356463721f29eb6e99d7a015fff47e9fc4a5e1fb0fce0b764b3f5"}
Nov 03 18:26:29.580576 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:29.580468    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg" event={"ID":"a59cf8bf-640b-4a56-8f65-6e1ebf380caf","Type":"ContainerStarted","Data":"d3b56cbfe49c99a701bbbc3ce3f958a042fdf1adadf889892d0bbed9369a5aa9"}
Nov 03 18:26:29.700113 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:29.700060    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr" podStartSLOduration=1.700046151 podStartE2EDuration="1.700046151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:26:29.698748974 +0000 UTC m=+4130.496461766" watchObservedRunningTime="2026-11-03 18:26:29.700046151 +0000 UTC m=+4130.497758932"
Nov 03 18:26:29.819257 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:29.819202    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg" podStartSLOduration=1.819189158 podStartE2EDuration="1.819189158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:26:29.817856365 +0000 UTC m=+4130.615569147" watchObservedRunningTime="2026-11-03 18:26:29.819189158 +0000 UTC m=+4130.616901956"
Nov 03 18:26:29.962598 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:29.962568    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-6244ce06c11a: secret "agc-cred-6244ce06c11a" not found
Nov 03 18:26:29.962598 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:29.962600    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-7afe86be4901: secret "agc-cred-7afe86be4901" not found
Nov 03 18:26:29.962812 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:29.962610    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-d13ad376f3be: secret "agc-cred-d13ad376f3be" not found
Nov 03 18:26:29.962812 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:29.962626    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2: [secret "agc-cred-6244ce06c11a" not found, secret "agc-cred-7afe86be4901" not found, secret "agc-cred-d13ad376f3be" not found]
Nov 03 18:26:29.962812 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:29.962695    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-provider-credentials podName:ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6 nodeName:}" failed. No retries permitted until 2026-11-03 18:26:31.962674484 +0000 UTC m=+4132.760387246 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2" (UID: "ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6") : [secret "agc-cred-6244ce06c11a" not found, secret "agc-cred-7afe86be4901" not found, secret "agc-cred-d13ad376f3be" not found]
Nov 03 18:26:30.120720 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:30.120686    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:30.587405 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:30.587371    6059 generic.go:356] "Generic (PLEG): container finished" podID="a59cf8bf-640b-4a56-8f65-6e1ebf380caf" containerID="e2aa24d2a4f356463721f29eb6e99d7a015fff47e9fc4a5e1fb0fce0b764b3f5" exitCode=0
Nov 03 18:26:30.587800 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:30.587442    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg" event={"ID":"a59cf8bf-640b-4a56-8f65-6e1ebf380caf","Type":"ContainerDied","Data":"e2aa24d2a4f356463721f29eb6e99d7a015fff47e9fc4a5e1fb0fce0b764b3f5"}
Nov 03 18:26:30.790841 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:30.790809    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"]
Nov 03 18:26:30.792111 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:30.792084    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6" podUID="70bb010d-98bf-46b4-83b7-a99004ee7a50" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:26:30.836079 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:30.836040    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"]
Nov 03 18:26:30.840062 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:30.839967    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"
Nov 03 18:26:30.900128 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:30.900094    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"]
Nov 03 18:26:30.974019 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:30.973988    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cvwz\" (UniqueName: \"kubernetes.io/projected/89785e67-bd91-4e22-9e7b-8e88ce58268f-kube-api-access-9cvwz\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b\" (UID: \"89785e67-bd91-4e22-9e7b-8e88ce58268f\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"
Nov 03 18:26:30.974236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:30.974043    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89785e67-bd91-4e22-9e7b-8e88ce58268f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b\" (UID: \"89785e67-bd91-4e22-9e7b-8e88ce58268f\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"
Nov 03 18:26:30.974236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:30.974064    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/89785e67-bd91-4e22-9e7b-8e88ce58268f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b\" (UID: \"89785e67-bd91-4e22-9e7b-8e88ce58268f\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"
Nov 03 18:26:30.974236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:30.974219    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/89785e67-bd91-4e22-9e7b-8e88ce58268f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b\" (UID: \"89785e67-bd91-4e22-9e7b-8e88ce58268f\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"
Nov 03 18:26:31.075424 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:31.075385    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cvwz\" (UniqueName: \"kubernetes.io/projected/89785e67-bd91-4e22-9e7b-8e88ce58268f-kube-api-access-9cvwz\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b\" (UID: \"89785e67-bd91-4e22-9e7b-8e88ce58268f\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"
Nov 03 18:26:31.075599 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:31.075460    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89785e67-bd91-4e22-9e7b-8e88ce58268f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b\" (UID: \"89785e67-bd91-4e22-9e7b-8e88ce58268f\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"
Nov 03 18:26:31.075599 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:31.075478    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/89785e67-bd91-4e22-9e7b-8e88ce58268f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b\" (UID: \"89785e67-bd91-4e22-9e7b-8e88ce58268f\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"
Nov 03 18:26:31.075599 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:31.075517    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/89785e67-bd91-4e22-9e7b-8e88ce58268f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b\" (UID: \"89785e67-bd91-4e22-9e7b-8e88ce58268f\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"
Nov 03 18:26:31.076250 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:31.076223    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/89785e67-bd91-4e22-9e7b-8e88ce58268f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b\" (UID: \"89785e67-bd91-4e22-9e7b-8e88ce58268f\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"
Nov 03 18:26:31.076390 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:31.076296    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89785e67-bd91-4e22-9e7b-8e88ce58268f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b\" (UID: \"89785e67-bd91-4e22-9e7b-8e88ce58268f\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"
Nov 03 18:26:31.077992 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:31.077973    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/89785e67-bd91-4e22-9e7b-8e88ce58268f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b\" (UID: \"89785e67-bd91-4e22-9e7b-8e88ce58268f\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"
Nov 03 18:26:31.138552 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:31.138475    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cvwz\" (UniqueName: \"kubernetes.io/projected/89785e67-bd91-4e22-9e7b-8e88ce58268f-kube-api-access-9cvwz\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b\" (UID: \"89785e67-bd91-4e22-9e7b-8e88ce58268f\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"
Nov 03 18:26:31.153208 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:31.153167    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"
Nov 03 18:26:31.527396 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:26:31.527327    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89785e67_bd91_4e22_9e7b_8e88ce58268f.slice/crio-342e32980b9d6342d9cca20ddc389de75276e9aeb43125e52925f0297b61cf7e WatchSource:0}: Error finding container 342e32980b9d6342d9cca20ddc389de75276e9aeb43125e52925f0297b61cf7e: Status 404 returned error can't find the container with id 342e32980b9d6342d9cca20ddc389de75276e9aeb43125e52925f0297b61cf7e
Nov 03 18:26:31.529861 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:31.529836    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"]
Nov 03 18:26:31.594266 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:31.594241    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b" event={"ID":"89785e67-bd91-4e22-9e7b-8e88ce58268f","Type":"ContainerStarted","Data":"342e32980b9d6342d9cca20ddc389de75276e9aeb43125e52925f0297b61cf7e"}
Nov 03 18:26:31.711813 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:31.711787    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg"
Nov 03 18:26:31.785614 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:31.785550    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a59cf8bf-640b-4a56-8f65-6e1ebf380caf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a59cf8bf-640b-4a56-8f65-6e1ebf380caf-maas-api-openshift-ca\") pod \"a59cf8bf-640b-4a56-8f65-6e1ebf380caf\" (UID: \"a59cf8bf-640b-4a56-8f65-6e1ebf380caf\") "
Nov 03 18:26:31.785614 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:31.785602    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a59cf8bf-640b-4a56-8f65-6e1ebf380caf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a59cf8bf-640b-4a56-8f65-6e1ebf380caf-maas-api-xks-ca\") pod \"a59cf8bf-640b-4a56-8f65-6e1ebf380caf\" (UID: \"a59cf8bf-640b-4a56-8f65-6e1ebf380caf\") "
Nov 03 18:26:31.785807 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:31.785623    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a59cf8bf-640b-4a56-8f65-6e1ebf380caf-kube-api-access-qnhpq\" (UniqueName: \"kubernetes.io/projected/a59cf8bf-640b-4a56-8f65-6e1ebf380caf-kube-api-access-qnhpq\") pod \"a59cf8bf-640b-4a56-8f65-6e1ebf380caf\" (UID: \"a59cf8bf-640b-4a56-8f65-6e1ebf380caf\") "
Nov 03 18:26:31.785884 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:31.785863    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a59cf8bf-640b-4a56-8f65-6e1ebf380caf-maas-api-openshift-ca" pod "a59cf8bf-640b-4a56-8f65-6e1ebf380caf" (UID: "a59cf8bf-640b-4a56-8f65-6e1ebf380caf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:26:31.786015 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:31.786001    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a59cf8bf-640b-4a56-8f65-6e1ebf380caf-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:31.787542 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:31.787523    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a59cf8bf-640b-4a56-8f65-6e1ebf380caf-maas-api-xks-ca" pod "a59cf8bf-640b-4a56-8f65-6e1ebf380caf" (UID: "a59cf8bf-640b-4a56-8f65-6e1ebf380caf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:26:31.787654 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:31.787629    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59cf8bf-640b-4a56-8f65-6e1ebf380caf-kube-api-access-qnhpq" pod "a59cf8bf-640b-4a56-8f65-6e1ebf380caf" (UID: "a59cf8bf-640b-4a56-8f65-6e1ebf380caf"). InnerVolumeSpecName "kube-api-access-qnhpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:26:31.887150 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:31.887116    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a59cf8bf-640b-4a56-8f65-6e1ebf380caf-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:31.887150 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:31.887143    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnhpq\" (UniqueName: \"kubernetes.io/projected/a59cf8bf-640b-4a56-8f65-6e1ebf380caf-kube-api-access-qnhpq\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:31.988036 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:31.987999    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-6244ce06c11a: secret "agc-cred-6244ce06c11a" not found
Nov 03 18:26:31.988036 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:31.988030    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-7afe86be4901: secret "agc-cred-7afe86be4901" not found
Nov 03 18:26:31.988036 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:31.988039    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-d13ad376f3be: secret "agc-cred-d13ad376f3be" not found
Nov 03 18:26:31.988280 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:31.988052    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2: [secret "agc-cred-6244ce06c11a" not found, secret "agc-cred-7afe86be4901" not found, secret "agc-cred-d13ad376f3be" not found]
Nov 03 18:26:31.988280 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:31.988106    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-provider-credentials podName:ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6 nodeName:}" failed. No retries permitted until 2026-11-03 18:26:35.98809019 +0000 UTC m=+4136.785802953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2" (UID: "ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6") : [secret "agc-cred-6244ce06c11a" not found, secret "agc-cred-7afe86be4901" not found, secret "agc-cred-d13ad376f3be" not found]
Nov 03 18:26:32.345122 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:32.345022    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:32.608913 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:32.608808    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b" event={"ID":"89785e67-bd91-4e22-9e7b-8e88ce58268f","Type":"ContainerStarted","Data":"3c2d93b3bbb599c8b77dd41e760bcffdca0fdbe3bcf7004252d649f40478e113"}
Nov 03 18:26:32.608913 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:32.608905    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"
Nov 03 18:26:32.610331 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:32.610305    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg"
Nov 03 18:26:32.610331 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:32.610315    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg" event={"ID":"a59cf8bf-640b-4a56-8f65-6e1ebf380caf","Type":"ContainerDied","Data":"d3b56cbfe49c99a701bbbc3ce3f958a042fdf1adadf889892d0bbed9369a5aa9"}
Nov 03 18:26:32.610514 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:32.610372    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3b56cbfe49c99a701bbbc3ce3f958a042fdf1adadf889892d0bbed9369a5aa9"
Nov 03 18:26:32.730278 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:32.730227    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b" podStartSLOduration=2.730214176 podStartE2EDuration="2.730214176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:26:32.72834529 +0000 UTC m=+4133.526058072" watchObservedRunningTime="2026-11-03 18:26:32.730214176 +0000 UTC m=+4133.527926956"
Nov 03 18:26:33.378618 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:33.378580    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg" podUID="2d396930-b2b9-45df-aa1d-042dd368fef3" containerName="payload-processing" containerID="cri-o://e41c73121a42d43acb1d9c40c6bc183cc2e401fc4b8ad526f51539f1a12b787d" gracePeriod=25
Nov 03 18:26:33.617833 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:33.617803    6059 generic.go:356] "Generic (PLEG): container finished" podID="2d396930-b2b9-45df-aa1d-042dd368fef3" containerID="e41c73121a42d43acb1d9c40c6bc183cc2e401fc4b8ad526f51539f1a12b787d" exitCode=0
Nov 03 18:26:33.618140 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:33.617866    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg" event={"ID":"2d396930-b2b9-45df-aa1d-042dd368fef3","Type":"ContainerDied","Data":"e41c73121a42d43acb1d9c40c6bc183cc2e401fc4b8ad526f51539f1a12b787d"}
Nov 03 18:26:33.640962 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:33.640910    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"
Nov 03 18:26:33.806544 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:33.806508    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d396930-b2b9-45df-aa1d-042dd368fef3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d396930-b2b9-45df-aa1d-042dd368fef3-provider-credentials\") pod \"2d396930-b2b9-45df-aa1d-042dd368fef3\" (UID: \"2d396930-b2b9-45df-aa1d-042dd368fef3\") "
Nov 03 18:26:33.806706 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:33.806598    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d396930-b2b9-45df-aa1d-042dd368fef3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d396930-b2b9-45df-aa1d-042dd368fef3-plugins-config-volume\") pod \"2d396930-b2b9-45df-aa1d-042dd368fef3\" (UID: \"2d396930-b2b9-45df-aa1d-042dd368fef3\") "
Nov 03 18:26:33.806706 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:33.806642    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d396930-b2b9-45df-aa1d-042dd368fef3-kube-api-access-bbrnf\" (UniqueName: \"kubernetes.io/projected/2d396930-b2b9-45df-aa1d-042dd368fef3-kube-api-access-bbrnf\") pod \"2d396930-b2b9-45df-aa1d-042dd368fef3\" (UID: \"2d396930-b2b9-45df-aa1d-042dd368fef3\") "
Nov 03 18:26:33.806706 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:33.806674    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d396930-b2b9-45df-aa1d-042dd368fef3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d396930-b2b9-45df-aa1d-042dd368fef3-routing-overlay-volume\") pod \"2d396930-b2b9-45df-aa1d-042dd368fef3\" (UID: \"2d396930-b2b9-45df-aa1d-042dd368fef3\") "
Nov 03 18:26:33.807034 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:33.807001    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d396930-b2b9-45df-aa1d-042dd368fef3-plugins-config-volume" pod "2d396930-b2b9-45df-aa1d-042dd368fef3" (UID: "2d396930-b2b9-45df-aa1d-042dd368fef3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:26:33.807158 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:33.807053    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d396930-b2b9-45df-aa1d-042dd368fef3-routing-overlay-volume" pod "2d396930-b2b9-45df-aa1d-042dd368fef3" (UID: "2d396930-b2b9-45df-aa1d-042dd368fef3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:26:33.807245 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:33.807229    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d396930-b2b9-45df-aa1d-042dd368fef3-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:33.807306 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:33.807252    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d396930-b2b9-45df-aa1d-042dd368fef3-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:33.808741 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:33.808715    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d396930-b2b9-45df-aa1d-042dd368fef3-provider-credentials" pod "2d396930-b2b9-45df-aa1d-042dd368fef3" (UID: "2d396930-b2b9-45df-aa1d-042dd368fef3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:26:33.808830 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:33.808739    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d396930-b2b9-45df-aa1d-042dd368fef3-kube-api-access-bbrnf" pod "2d396930-b2b9-45df-aa1d-042dd368fef3" (UID: "2d396930-b2b9-45df-aa1d-042dd368fef3"). InnerVolumeSpecName "kube-api-access-bbrnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:26:33.908278 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:33.908207    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d396930-b2b9-45df-aa1d-042dd368fef3-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:33.908278 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:33.908233    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbrnf\" (UniqueName: \"kubernetes.io/projected/2d396930-b2b9-45df-aa1d-042dd368fef3-kube-api-access-bbrnf\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:34.625261 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:34.625230    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"
Nov 03 18:26:34.625823 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:34.625264    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg" event={"ID":"2d396930-b2b9-45df-aa1d-042dd368fef3","Type":"ContainerDied","Data":"446b6c9d1e1bd492f1116c8f4d24a553b4746227ba770995f01a8f7bd84a1a18"}
Nov 03 18:26:34.625823 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:34.625322    6059 scope.go:122] "RemoveContainer" containerID="e41c73121a42d43acb1d9c40c6bc183cc2e401fc4b8ad526f51539f1a12b787d"
Nov 03 18:26:34.794978 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:34.794940    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"]
Nov 03 18:26:34.846527 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:34.846489    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-5759mtmxg"]
Nov 03 18:26:35.335002 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:35.334972    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d396930-b2b9-45df-aa1d-042dd368fef3" path="/var/lib/kubelet/pods/2d396930-b2b9-45df-aa1d-042dd368fef3/volumes"
Nov 03 18:26:35.791471 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:35.791431    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6" podUID="70bb010d-98bf-46b4-83b7-a99004ee7a50" containerName="payload-processing" containerID="cri-o://e1d8b42d798dea6883f40c7f4ce04514effb1efa3884e29775b3a1c26c631a0b" gracePeriod=25
Nov 03 18:26:36.029552 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:36.029526    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-6244ce06c11a: secret "agc-cred-6244ce06c11a" not found
Nov 03 18:26:36.029552 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:36.029553    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-7afe86be4901: secret "agc-cred-7afe86be4901" not found
Nov 03 18:26:36.029712 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:36.029560    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-d13ad376f3be: secret "agc-cred-d13ad376f3be" not found
Nov 03 18:26:36.029712 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:36.029573    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2: [secret "agc-cred-6244ce06c11a" not found, secret "agc-cred-7afe86be4901" not found, secret "agc-cred-d13ad376f3be" not found]
Nov 03 18:26:36.029712 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:36.029629    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-provider-credentials podName:ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6 nodeName:}" failed. No retries permitted until 2026-11-03 18:26:44.029612644 +0000 UTC m=+4144.827325420 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2" (UID: "ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6") : [secret "agc-cred-6244ce06c11a" not found, secret "agc-cred-7afe86be4901" not found, secret "agc-cred-d13ad376f3be" not found]
Nov 03 18:26:36.044550 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:36.044499    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"
Nov 03 18:26:36.130547 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:36.130515    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70bb010d-98bf-46b4-83b7-a99004ee7a50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70bb010d-98bf-46b4-83b7-a99004ee7a50-plugins-config-volume\") pod \"70bb010d-98bf-46b4-83b7-a99004ee7a50\" (UID: \"70bb010d-98bf-46b4-83b7-a99004ee7a50\") "
Nov 03 18:26:36.130723 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:36.130570    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70bb010d-98bf-46b4-83b7-a99004ee7a50-kube-api-access-xglxn\" (UniqueName: \"kubernetes.io/projected/70bb010d-98bf-46b4-83b7-a99004ee7a50-kube-api-access-xglxn\") pod \"70bb010d-98bf-46b4-83b7-a99004ee7a50\" (UID: \"70bb010d-98bf-46b4-83b7-a99004ee7a50\") "
Nov 03 18:26:36.130723 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:36.130606    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70bb010d-98bf-46b4-83b7-a99004ee7a50-provider-credentials\" (UniqueName: \"kubernetes.io/projected/70bb010d-98bf-46b4-83b7-a99004ee7a50-provider-credentials\") pod \"70bb010d-98bf-46b4-83b7-a99004ee7a50\" (UID: \"70bb010d-98bf-46b4-83b7-a99004ee7a50\") "
Nov 03 18:26:36.130723 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:36.130637    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70bb010d-98bf-46b4-83b7-a99004ee7a50-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70bb010d-98bf-46b4-83b7-a99004ee7a50-routing-overlay-volume\") pod \"70bb010d-98bf-46b4-83b7-a99004ee7a50\" (UID: \"70bb010d-98bf-46b4-83b7-a99004ee7a50\") "
Nov 03 18:26:36.131090 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:36.131056    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70bb010d-98bf-46b4-83b7-a99004ee7a50-plugins-config-volume" pod "70bb010d-98bf-46b4-83b7-a99004ee7a50" (UID: "70bb010d-98bf-46b4-83b7-a99004ee7a50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:26:36.131090 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:36.131077    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70bb010d-98bf-46b4-83b7-a99004ee7a50-routing-overlay-volume" pod "70bb010d-98bf-46b4-83b7-a99004ee7a50" (UID: "70bb010d-98bf-46b4-83b7-a99004ee7a50"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:26:36.132685 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:36.132659    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bb010d-98bf-46b4-83b7-a99004ee7a50-kube-api-access-xglxn" pod "70bb010d-98bf-46b4-83b7-a99004ee7a50" (UID: "70bb010d-98bf-46b4-83b7-a99004ee7a50"). InnerVolumeSpecName "kube-api-access-xglxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:26:36.132760 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:36.132720    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bb010d-98bf-46b4-83b7-a99004ee7a50-provider-credentials" pod "70bb010d-98bf-46b4-83b7-a99004ee7a50" (UID: "70bb010d-98bf-46b4-83b7-a99004ee7a50"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:26:36.231997 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:36.231957    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70bb010d-98bf-46b4-83b7-a99004ee7a50-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:36.231997 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:36.231989    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xglxn\" (UniqueName: \"kubernetes.io/projected/70bb010d-98bf-46b4-83b7-a99004ee7a50-kube-api-access-xglxn\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:36.232203 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:36.232007    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70bb010d-98bf-46b4-83b7-a99004ee7a50-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:36.232203 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:36.232018    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70bb010d-98bf-46b4-83b7-a99004ee7a50-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:36.639172 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:36.639133    6059 generic.go:356] "Generic (PLEG): container finished" podID="70bb010d-98bf-46b4-83b7-a99004ee7a50" containerID="e1d8b42d798dea6883f40c7f4ce04514effb1efa3884e29775b3a1c26c631a0b" exitCode=0
Nov 03 18:26:36.639392 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:36.639180    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"
Nov 03 18:26:36.639392 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:36.639223    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6" event={"ID":"70bb010d-98bf-46b4-83b7-a99004ee7a50","Type":"ContainerDied","Data":"e1d8b42d798dea6883f40c7f4ce04514effb1efa3884e29775b3a1c26c631a0b"}
Nov 03 18:26:36.639392 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:36.639277    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6" event={"ID":"70bb010d-98bf-46b4-83b7-a99004ee7a50","Type":"ContainerDied","Data":"a6ec8ae150e9477c2ec14c5148780d1034bc250675006fa6116df28a76fcb44a"}
Nov 03 18:26:36.639392 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:36.639300    6059 scope.go:122] "RemoveContainer" containerID="e1d8b42d798dea6883f40c7f4ce04514effb1efa3884e29775b3a1c26c631a0b"
Nov 03 18:26:36.649241 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:36.649226    6059 scope.go:122] "RemoveContainer" containerID="e1d8b42d798dea6883f40c7f4ce04514effb1efa3884e29775b3a1c26c631a0b"
Nov 03 18:26:36.649508 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:36.649493    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1d8b42d798dea6883f40c7f4ce04514effb1efa3884e29775b3a1c26c631a0b\": container with ID starting with e1d8b42d798dea6883f40c7f4ce04514effb1efa3884e29775b3a1c26c631a0b not found: ID does not exist" containerID="e1d8b42d798dea6883f40c7f4ce04514effb1efa3884e29775b3a1c26c631a0b"
Nov 03 18:26:36.649558 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:36.649515    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1d8b42d798dea6883f40c7f4ce04514effb1efa3884e29775b3a1c26c631a0b"} err="failed to get container status \"e1d8b42d798dea6883f40c7f4ce04514effb1efa3884e29775b3a1c26c631a0b\": rpc error: code = NotFound desc = could not find container \"e1d8b42d798dea6883f40c7f4ce04514effb1efa3884e29775b3a1c26c631a0b\": container with ID starting with e1d8b42d798dea6883f40c7f4ce04514effb1efa3884e29775b3a1c26c631a0b not found: ID does not exist"
Nov 03 18:26:36.811104 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:36.811070    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"]
Nov 03 18:26:36.815057 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:36.815029    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6587sscz6"]
Nov 03 18:26:36.860506 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:36.860468    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:26:37.153695 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:37.153652    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:37.334943 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:37.334902    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70bb010d-98bf-46b4-83b7-a99004ee7a50" path="/var/lib/kubelet/pods/70bb010d-98bf-46b4-83b7-a99004ee7a50/volumes"
Nov 03 18:26:40.072889 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.072852    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9604928-z2z5r"]
Nov 03 18:26:40.073409 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.073393    6059 cpu_manager.go:414] "containerMap: removing container" podUID="70bb010d-98bf-46b4-83b7-a99004ee7a50" containerName="payload-processing"
Nov 03 18:26:40.073409 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.073409    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70bb010d-98bf-46b4-83b7-a99004ee7a50" containerName="payload-processing"
Nov 03 18:26:40.073500 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.073432    6059 cpu_manager.go:414] "containerMap: removing container" podUID="2d396930-b2b9-45df-aa1d-042dd368fef3" containerName="payload-processing"
Nov 03 18:26:40.073500 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.073438    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d396930-b2b9-45df-aa1d-042dd368fef3" containerName="payload-processing"
Nov 03 18:26:40.073500 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.073457    6059 cpu_manager.go:414] "containerMap: removing container" podUID="a59cf8bf-640b-4a56-8f65-6e1ebf380caf" containerName="revoke-keys"
Nov 03 18:26:40.073500 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.073462    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a59cf8bf-640b-4a56-8f65-6e1ebf380caf" containerName="revoke-keys"
Nov 03 18:26:40.073628 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.073538    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a59cf8bf-640b-4a56-8f65-6e1ebf380caf" containerName="revoke-keys"
Nov 03 18:26:40.073628 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.073552    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="70bb010d-98bf-46b4-83b7-a99004ee7a50" containerName="payload-processing"
Nov 03 18:26:40.073628 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.073560    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d396930-b2b9-45df-aa1d-042dd368fef3" containerName="payload-processing"
Nov 03 18:26:40.076925 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.076910    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9604928-z2z5r"
Nov 03 18:26:40.134504 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.134468    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:26:40.170153 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.170120    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e84c1e6-3809-4ce6-9430-7c7ec1f26a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9604928-z2z5r\" (UID: \"4e84c1e6-3809-4ce6-9430-7c7ec1f26a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9604928-z2z5r"
Nov 03 18:26:40.170282 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.170260    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e84c1e6-3809-4ce6-9430-7c7ec1f26a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9604928-z2z5r\" (UID: \"4e84c1e6-3809-4ce6-9430-7c7ec1f26a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9604928-z2z5r"
Nov 03 18:26:40.170344 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.170285    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj6hh\" (UniqueName: \"kubernetes.io/projected/4e84c1e6-3809-4ce6-9430-7c7ec1f26a33-kube-api-access-vj6hh\") pod \"maas-api-revoke-sub-c9604928-z2z5r\" (UID: \"4e84c1e6-3809-4ce6-9430-7c7ec1f26a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9604928-z2z5r"
Nov 03 18:26:40.185758 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.185736    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:26:40.185961 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.185944    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\""
Nov 03 18:26:40.186013 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.185958    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:26:40.188239 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.188221    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9604928-z2z5r"]
Nov 03 18:26:40.271519 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.271491    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e84c1e6-3809-4ce6-9430-7c7ec1f26a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9604928-z2z5r\" (UID: \"4e84c1e6-3809-4ce6-9430-7c7ec1f26a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9604928-z2z5r"
Nov 03 18:26:40.271680 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.271531    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj6hh\" (UniqueName: \"kubernetes.io/projected/4e84c1e6-3809-4ce6-9430-7c7ec1f26a33-kube-api-access-vj6hh\") pod \"maas-api-revoke-sub-c9604928-z2z5r\" (UID: \"4e84c1e6-3809-4ce6-9430-7c7ec1f26a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9604928-z2z5r"
Nov 03 18:26:40.271680 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.271597    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e84c1e6-3809-4ce6-9430-7c7ec1f26a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9604928-z2z5r\" (UID: \"4e84c1e6-3809-4ce6-9430-7c7ec1f26a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9604928-z2z5r"
Nov 03 18:26:40.272326 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.272308    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e84c1e6-3809-4ce6-9430-7c7ec1f26a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9604928-z2z5r\" (UID: \"4e84c1e6-3809-4ce6-9430-7c7ec1f26a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9604928-z2z5r"
Nov 03 18:26:40.273868 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.273845    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e84c1e6-3809-4ce6-9430-7c7ec1f26a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9604928-z2z5r\" (UID: \"4e84c1e6-3809-4ce6-9430-7c7ec1f26a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9604928-z2z5r"
Nov 03 18:26:40.334404 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.334326    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj6hh\" (UniqueName: \"kubernetes.io/projected/4e84c1e6-3809-4ce6-9430-7c7ec1f26a33-kube-api-access-vj6hh\") pod \"maas-api-revoke-sub-c9604928-z2z5r\" (UID: \"4e84c1e6-3809-4ce6-9430-7c7ec1f26a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9604928-z2z5r"
Nov 03 18:26:40.389533 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.389506    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9604928-z2z5r"
Nov 03 18:26:40.590491 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.590413    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr" podUID="7cba1dd3-36bd-4832-a4cb-cd3537ac1147" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:26:40.762488 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:26:40.762459    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e84c1e6_3809_4ce6_9430_7c7ec1f26a33.slice/crio-8d06cf4297233afe6569f6e854feb61bd3c5c07836f7d5e68de92148d3727a97 WatchSource:0}: Error finding container 8d06cf4297233afe6569f6e854feb61bd3c5c07836f7d5e68de92148d3727a97: Status 404 returned error can't find the container with id 8d06cf4297233afe6569f6e854feb61bd3c5c07836f7d5e68de92148d3727a97
Nov 03 18:26:40.764348 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:40.764329    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9604928-z2z5r"]
Nov 03 18:26:41.670200 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:41.670171    6059 generic.go:356] "Generic (PLEG): container finished" podID="4e84c1e6-3809-4ce6-9430-7c7ec1f26a33" containerID="759ab4504c17b7357802114eb867f975ed4085f32c345c1f74e0afd25492e9be" exitCode=0
Nov 03 18:26:41.670540 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:41.670198    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9604928-z2z5r" event={"ID":"4e84c1e6-3809-4ce6-9430-7c7ec1f26a33","Type":"ContainerDied","Data":"759ab4504c17b7357802114eb867f975ed4085f32c345c1f74e0afd25492e9be"}
Nov 03 18:26:41.670540 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:41.670249    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9604928-z2z5r" event={"ID":"4e84c1e6-3809-4ce6-9430-7c7ec1f26a33","Type":"ContainerStarted","Data":"8d06cf4297233afe6569f6e854feb61bd3c5c07836f7d5e68de92148d3727a97"}
Nov 03 18:26:42.069909 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:42.069875    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:42.810226 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:42.810200    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9604928-z2z5r"
Nov 03 18:26:42.907436 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:42.907404    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e84c1e6-3809-4ce6-9430-7c7ec1f26a33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e84c1e6-3809-4ce6-9430-7c7ec1f26a33-maas-api-xks-ca\") pod \"4e84c1e6-3809-4ce6-9430-7c7ec1f26a33\" (UID: \"4e84c1e6-3809-4ce6-9430-7c7ec1f26a33\") "
Nov 03 18:26:42.907605 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:42.907480    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e84c1e6-3809-4ce6-9430-7c7ec1f26a33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e84c1e6-3809-4ce6-9430-7c7ec1f26a33-maas-api-openshift-ca\") pod \"4e84c1e6-3809-4ce6-9430-7c7ec1f26a33\" (UID: \"4e84c1e6-3809-4ce6-9430-7c7ec1f26a33\") "
Nov 03 18:26:42.907605 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:42.907507    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e84c1e6-3809-4ce6-9430-7c7ec1f26a33-kube-api-access-vj6hh\" (UniqueName: \"kubernetes.io/projected/4e84c1e6-3809-4ce6-9430-7c7ec1f26a33-kube-api-access-vj6hh\") pod \"4e84c1e6-3809-4ce6-9430-7c7ec1f26a33\" (UID: \"4e84c1e6-3809-4ce6-9430-7c7ec1f26a33\") "
Nov 03 18:26:42.907865 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:42.907839    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e84c1e6-3809-4ce6-9430-7c7ec1f26a33-maas-api-openshift-ca" pod "4e84c1e6-3809-4ce6-9430-7c7ec1f26a33" (UID: "4e84c1e6-3809-4ce6-9430-7c7ec1f26a33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:26:42.909632 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:42.909608    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e84c1e6-3809-4ce6-9430-7c7ec1f26a33-kube-api-access-vj6hh" pod "4e84c1e6-3809-4ce6-9430-7c7ec1f26a33" (UID: "4e84c1e6-3809-4ce6-9430-7c7ec1f26a33"). InnerVolumeSpecName "kube-api-access-vj6hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:26:42.909754 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:42.909629    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e84c1e6-3809-4ce6-9430-7c7ec1f26a33-maas-api-xks-ca" pod "4e84c1e6-3809-4ce6-9430-7c7ec1f26a33" (UID: "4e84c1e6-3809-4ce6-9430-7c7ec1f26a33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:26:43.008642 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:43.008613    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e84c1e6-3809-4ce6-9430-7c7ec1f26a33-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:43.008642 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:43.008640    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj6hh\" (UniqueName: \"kubernetes.io/projected/4e84c1e6-3809-4ce6-9430-7c7ec1f26a33-kube-api-access-vj6hh\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:43.008642 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:43.008651    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e84c1e6-3809-4ce6-9430-7c7ec1f26a33-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:43.620998 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:43.620902    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b" podUID="89785e67-bd91-4e22-9e7b-8e88ce58268f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:26:43.682208 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:43.682179    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9604928-z2z5r"
Nov 03 18:26:43.682411 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:43.682207    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9604928-z2z5r" event={"ID":"4e84c1e6-3809-4ce6-9430-7c7ec1f26a33","Type":"ContainerDied","Data":"8d06cf4297233afe6569f6e854feb61bd3c5c07836f7d5e68de92148d3727a97"}
Nov 03 18:26:43.682411 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:43.682245    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d06cf4297233afe6569f6e854feb61bd3c5c07836f7d5e68de92148d3727a97"
Nov 03 18:26:44.120545 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:44.120508    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-6244ce06c11a: secret "agc-cred-6244ce06c11a" not found
Nov 03 18:26:44.120545 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:44.120542    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-7afe86be4901: secret "agc-cred-7afe86be4901" not found
Nov 03 18:26:44.120545 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:44.120551    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-d13ad376f3be: secret "agc-cred-d13ad376f3be" not found
Nov 03 18:26:44.121021 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:44.120568    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2: [secret "agc-cred-6244ce06c11a" not found, secret "agc-cred-7afe86be4901" not found, secret "agc-cred-d13ad376f3be" not found]
Nov 03 18:26:44.121021 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:44.120627    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-provider-credentials podName:ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6 nodeName:}" failed. No retries permitted until 2026-11-03 18:27:00.120610598 +0000 UTC m=+4160.918323360 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2" (UID: "ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6") : [secret "agc-cred-6244ce06c11a" not found, secret "agc-cred-7afe86be4901" not found, secret "agc-cred-d13ad376f3be" not found]
Nov 03 18:26:47.053559 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:47.053527    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:50.589273 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:50.589234    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr" podUID="7cba1dd3-36bd-4832-a4cb-cd3537ac1147" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:26:51.769587 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:51.769512    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:52.866106 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:52.866070    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"]
Nov 03 18:26:52.871810 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:52.871784    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"]
Nov 03 18:26:52.872968 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:52.872942    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr" podUID="7cba1dd3-36bd-4832-a4cb-cd3537ac1147" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:26:53.026199 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:53.026136    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:53.208130 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:53.208090    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:53.619820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:53.619726    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b" podUID="89785e67-bd91-4e22-9e7b-8e88ce58268f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:26:53.754445 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:53.754399    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:54.144585 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:54.144548    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:55.434622 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:55.434584    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"]
Nov 03 18:26:55.436136 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:55.436109    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b" podUID="89785e67-bd91-4e22-9e7b-8e88ce58268f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:26:55.485698 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:55.485672    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"]
Nov 03 18:26:55.609627 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:55.609594    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:56.546131 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:56.546102    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:57.866772 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:57.866727    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2" podUID="d3d58540-7d6f-459b-bcc5-e21804a1085b" containerName="payload-processing" containerID="cri-o://d8dce6c0533e6ffe62ed5d68122bf313a776dac706a268dbddb1ed1b79e9859b" gracePeriod=25
Nov 03 18:26:57.872261 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:57.872222    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr" podUID="7cba1dd3-36bd-4832-a4cb-cd3537ac1147" containerName="payload-processing" containerID="cri-o://a05465577a0977a4cf6cc9c06364c04871eb8e2e8b2422441c8065ba4f841dfc" gracePeriod=25
Nov 03 18:26:58.007752 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:26:58.007723    6059 logging.go:55] [core] [Channel #8420 SubChannel #8421]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.250:50052", ServerName: "10.133.0.250:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.250:50052: connect: connection refused"
Nov 03 18:26:58.150602 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.150576    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"
Nov 03 18:26:58.153382 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.153343    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"
Nov 03 18:26:58.259929 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.259888    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-plugins-config-volume\") pod \"7cba1dd3-36bd-4832-a4cb-cd3537ac1147\" (UID: \"7cba1dd3-36bd-4832-a4cb-cd3537ac1147\") "
Nov 03 18:26:58.260138 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.259940    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d58540-7d6f-459b-bcc5-e21804a1085b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3d58540-7d6f-459b-bcc5-e21804a1085b-routing-overlay-volume\") pod \"d3d58540-7d6f-459b-bcc5-e21804a1085b\" (UID: \"d3d58540-7d6f-459b-bcc5-e21804a1085b\") "
Nov 03 18:26:58.260138 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.259976    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-kube-api-access-x4hkj\" (UniqueName: \"kubernetes.io/projected/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-kube-api-access-x4hkj\") pod \"7cba1dd3-36bd-4832-a4cb-cd3537ac1147\" (UID: \"7cba1dd3-36bd-4832-a4cb-cd3537ac1147\") "
Nov 03 18:26:58.260273 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.260151    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-routing-overlay-volume\") pod \"7cba1dd3-36bd-4832-a4cb-cd3537ac1147\" (UID: \"7cba1dd3-36bd-4832-a4cb-cd3537ac1147\") "
Nov 03 18:26:58.260273 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.260217    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d58540-7d6f-459b-bcc5-e21804a1085b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d58540-7d6f-459b-bcc5-e21804a1085b-plugins-config-volume\") pod \"d3d58540-7d6f-459b-bcc5-e21804a1085b\" (UID: \"d3d58540-7d6f-459b-bcc5-e21804a1085b\") "
Nov 03 18:26:58.260426 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.260303    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d58540-7d6f-459b-bcc5-e21804a1085b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3d58540-7d6f-459b-bcc5-e21804a1085b-provider-credentials\") pod \"d3d58540-7d6f-459b-bcc5-e21804a1085b\" (UID: \"d3d58540-7d6f-459b-bcc5-e21804a1085b\") "
Nov 03 18:26:58.260426 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.260411    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d58540-7d6f-459b-bcc5-e21804a1085b-routing-overlay-volume" pod "d3d58540-7d6f-459b-bcc5-e21804a1085b" (UID: "d3d58540-7d6f-459b-bcc5-e21804a1085b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:26:58.260542 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.260449    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d58540-7d6f-459b-bcc5-e21804a1085b-kube-api-access-thnzn\" (UniqueName: \"kubernetes.io/projected/d3d58540-7d6f-459b-bcc5-e21804a1085b-kube-api-access-thnzn\") pod \"d3d58540-7d6f-459b-bcc5-e21804a1085b\" (UID: \"d3d58540-7d6f-459b-bcc5-e21804a1085b\") "
Nov 03 18:26:58.260542 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.260484    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-plugins-config-volume" pod "7cba1dd3-36bd-4832-a4cb-cd3537ac1147" (UID: "7cba1dd3-36bd-4832-a4cb-cd3537ac1147"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:26:58.260542 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.260531    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-provider-credentials\") pod \"7cba1dd3-36bd-4832-a4cb-cd3537ac1147\" (UID: \"7cba1dd3-36bd-4832-a4cb-cd3537ac1147\") "
Nov 03 18:26:58.260768 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.260532    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-routing-overlay-volume" pod "7cba1dd3-36bd-4832-a4cb-cd3537ac1147" (UID: "7cba1dd3-36bd-4832-a4cb-cd3537ac1147"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:26:58.260768 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.260644    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d58540-7d6f-459b-bcc5-e21804a1085b-plugins-config-volume" pod "d3d58540-7d6f-459b-bcc5-e21804a1085b" (UID: "d3d58540-7d6f-459b-bcc5-e21804a1085b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:26:58.260925 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.260903    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:58.260998 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.260929    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3d58540-7d6f-459b-bcc5-e21804a1085b-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:58.260998 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.260946    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:58.260998 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.260961    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d58540-7d6f-459b-bcc5-e21804a1085b-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:58.262272 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.262246    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-kube-api-access-x4hkj" pod "7cba1dd3-36bd-4832-a4cb-cd3537ac1147" (UID: "7cba1dd3-36bd-4832-a4cb-cd3537ac1147"). InnerVolumeSpecName "kube-api-access-x4hkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:26:58.262395 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.262376    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-provider-credentials" pod "7cba1dd3-36bd-4832-a4cb-cd3537ac1147" (UID: "7cba1dd3-36bd-4832-a4cb-cd3537ac1147"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:26:58.262702 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.262680    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d58540-7d6f-459b-bcc5-e21804a1085b-provider-credentials" pod "d3d58540-7d6f-459b-bcc5-e21804a1085b" (UID: "d3d58540-7d6f-459b-bcc5-e21804a1085b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:26:58.262787 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.262771    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d58540-7d6f-459b-bcc5-e21804a1085b-kube-api-access-thnzn" pod "d3d58540-7d6f-459b-bcc5-e21804a1085b" (UID: "d3d58540-7d6f-459b-bcc5-e21804a1085b"). InnerVolumeSpecName "kube-api-access-thnzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:26:58.362346 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.362302    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3d58540-7d6f-459b-bcc5-e21804a1085b-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:58.362346 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.362337    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thnzn\" (UniqueName: \"kubernetes.io/projected/d3d58540-7d6f-459b-bcc5-e21804a1085b-kube-api-access-thnzn\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:58.362346 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.362348    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:58.362624 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.362383    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4hkj\" (UniqueName: \"kubernetes.io/projected/7cba1dd3-36bd-4832-a4cb-cd3537ac1147-kube-api-access-x4hkj\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:26:58.767221 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.767180    6059 generic.go:356] "Generic (PLEG): container finished" podID="d3d58540-7d6f-459b-bcc5-e21804a1085b" containerID="d8dce6c0533e6ffe62ed5d68122bf313a776dac706a268dbddb1ed1b79e9859b" exitCode=0
Nov 03 18:26:58.767462 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.767230    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"
Nov 03 18:26:58.767462 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.767261    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2" event={"ID":"d3d58540-7d6f-459b-bcc5-e21804a1085b","Type":"ContainerDied","Data":"d8dce6c0533e6ffe62ed5d68122bf313a776dac706a268dbddb1ed1b79e9859b"}
Nov 03 18:26:58.767462 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.767325    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2" event={"ID":"d3d58540-7d6f-459b-bcc5-e21804a1085b","Type":"ContainerDied","Data":"5d900d1e9f7af8b45b1b80d7f9a62425cc40fc1a3185895b69c12d56f540c164"}
Nov 03 18:26:58.767462 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.767379    6059 scope.go:122] "RemoveContainer" containerID="d8dce6c0533e6ffe62ed5d68122bf313a776dac706a268dbddb1ed1b79e9859b"
Nov 03 18:26:58.768706 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.768683    6059 generic.go:356] "Generic (PLEG): container finished" podID="7cba1dd3-36bd-4832-a4cb-cd3537ac1147" containerID="a05465577a0977a4cf6cc9c06364c04871eb8e2e8b2422441c8065ba4f841dfc" exitCode=0
Nov 03 18:26:58.768810 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.768731    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"
Nov 03 18:26:58.768810 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.768759    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr" event={"ID":"7cba1dd3-36bd-4832-a4cb-cd3537ac1147","Type":"ContainerDied","Data":"a05465577a0977a4cf6cc9c06364c04871eb8e2e8b2422441c8065ba4f841dfc"}
Nov 03 18:26:58.768810 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.768802    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr" event={"ID":"7cba1dd3-36bd-4832-a4cb-cd3537ac1147","Type":"ContainerDied","Data":"aa4b660629d493db91c81eeda77955ba2496451e69990136d1a150328b8b5c47"}
Nov 03 18:26:58.776629 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.776611    6059 scope.go:122] "RemoveContainer" containerID="d8dce6c0533e6ffe62ed5d68122bf313a776dac706a268dbddb1ed1b79e9859b"
Nov 03 18:26:58.776933 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:58.776914    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8dce6c0533e6ffe62ed5d68122bf313a776dac706a268dbddb1ed1b79e9859b\": container with ID starting with d8dce6c0533e6ffe62ed5d68122bf313a776dac706a268dbddb1ed1b79e9859b not found: ID does not exist" containerID="d8dce6c0533e6ffe62ed5d68122bf313a776dac706a268dbddb1ed1b79e9859b"
Nov 03 18:26:58.777017 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.776946    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8dce6c0533e6ffe62ed5d68122bf313a776dac706a268dbddb1ed1b79e9859b"} err="failed to get container status \"d8dce6c0533e6ffe62ed5d68122bf313a776dac706a268dbddb1ed1b79e9859b\": rpc error: code = NotFound desc = could not find container \"d8dce6c0533e6ffe62ed5d68122bf313a776dac706a268dbddb1ed1b79e9859b\": container with ID starting with d8dce6c0533e6ffe62ed5d68122bf313a776dac706a268dbddb1ed1b79e9859b not found: ID does not exist"
Nov 03 18:26:58.777017 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.776968    6059 scope.go:122] "RemoveContainer" containerID="a05465577a0977a4cf6cc9c06364c04871eb8e2e8b2422441c8065ba4f841dfc"
Nov 03 18:26:58.785952 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.785935    6059 scope.go:122] "RemoveContainer" containerID="a05465577a0977a4cf6cc9c06364c04871eb8e2e8b2422441c8065ba4f841dfc"
Nov 03 18:26:58.786204 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:26:58.786187    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a05465577a0977a4cf6cc9c06364c04871eb8e2e8b2422441c8065ba4f841dfc\": container with ID starting with a05465577a0977a4cf6cc9c06364c04871eb8e2e8b2422441c8065ba4f841dfc not found: ID does not exist" containerID="a05465577a0977a4cf6cc9c06364c04871eb8e2e8b2422441c8065ba4f841dfc"
Nov 03 18:26:58.786250 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.786210    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a05465577a0977a4cf6cc9c06364c04871eb8e2e8b2422441c8065ba4f841dfc"} err="failed to get container status \"a05465577a0977a4cf6cc9c06364c04871eb8e2e8b2422441c8065ba4f841dfc\": rpc error: code = NotFound desc = could not find container \"a05465577a0977a4cf6cc9c06364c04871eb8e2e8b2422441c8065ba4f841dfc\": container with ID starting with a05465577a0977a4cf6cc9c06364c04871eb8e2e8b2422441c8065ba4f841dfc not found: ID does not exist"
Nov 03 18:26:58.936615 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.936578    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"]
Nov 03 18:26:58.988328 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:58.988290    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2"]
Nov 03 18:26:59.007804 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:59.007762    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6b87mjjx2" podUID="d3d58540-7d6f-459b-bcc5-e21804a1085b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.250:50052\" within 1s: context deadline exceeded"
Nov 03 18:26:59.054883 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:59.054808    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"]
Nov 03 18:26:59.064786 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:59.064755    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-6fd829nrr"]
Nov 03 18:26:59.335604 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:59.335526    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cba1dd3-36bd-4832-a4cb-cd3537ac1147" path="/var/lib/kubelet/pods/7cba1dd3-36bd-4832-a4cb-cd3537ac1147/volumes"
Nov 03 18:26:59.335925 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:59.335912    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3d58540-7d6f-459b-bcc5-e21804a1085b" path="/var/lib/kubelet/pods/d3d58540-7d6f-459b-bcc5-e21804a1085b/volumes"
Nov 03 18:26:59.727277 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:59.727242    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:26:59.903865 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:26:59.903812    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:00.178891 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:27:00.178809    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-6244ce06c11a: secret "agc-cred-6244ce06c11a" not found
Nov 03 18:27:00.178891 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:27:00.178838    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-7afe86be4901: secret "agc-cred-7afe86be4901" not found
Nov 03 18:27:00.178891 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:27:00.178845    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-d13ad376f3be: secret "agc-cred-d13ad376f3be" not found
Nov 03 18:27:00.178891 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:27:00.178851    6059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-8c9b0f/agc-cred-f2f8f7423350: secret "agc-cred-f2f8f7423350" not found
Nov 03 18:27:00.178891 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:27:00.178860    6059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2: [secret "agc-cred-6244ce06c11a" not found, secret "agc-cred-7afe86be4901" not found, secret "agc-cred-d13ad376f3be" not found, secret "agc-cred-f2f8f7423350" not found]
Nov 03 18:27:00.179458 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:27:00.178919    6059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-provider-credentials podName:ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6 nodeName:}" failed. No retries permitted until 2026-11-03 18:27:32.178904285 +0000 UTC m=+4192.976617043 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2" (UID: "ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6") : [secret "agc-cred-6244ce06c11a" not found, secret "agc-cred-7afe86be4901" not found, secret "agc-cred-d13ad376f3be" not found, secret "agc-cred-f2f8f7423350" not found]
Nov 03 18:27:00.257268 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.257235    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:00.435223 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.435137    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b" podUID="89785e67-bd91-4e22-9e7b-8e88ce58268f" containerName="payload-processing" containerID="cri-o://3c2d93b3bbb599c8b77dd41e760bcffdca0fdbe3bcf7004252d649f40478e113" gracePeriod=25
Nov 03 18:27:00.486128 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.486090    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2" podUID="ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6" containerName="payload-processing" containerID="cri-o://ffa91730e1e7f2f8f83672dc60b3a8e9275b02e197c9c131191222308dd987cb" gracePeriod=25
Nov 03 18:27:00.722867 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.722831    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"
Nov 03 18:27:00.755008 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.754980    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"
Nov 03 18:27:00.783466 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.783431    6059 generic.go:356] "Generic (PLEG): container finished" podID="ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6" containerID="ffa91730e1e7f2f8f83672dc60b3a8e9275b02e197c9c131191222308dd987cb" exitCode=0
Nov 03 18:27:00.783635 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.783476    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"
Nov 03 18:27:00.783635 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.783520    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2" event={"ID":"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6","Type":"ContainerDied","Data":"ffa91730e1e7f2f8f83672dc60b3a8e9275b02e197c9c131191222308dd987cb"}
Nov 03 18:27:00.783635 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.783584    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2" event={"ID":"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6","Type":"ContainerDied","Data":"0b980be967a90ec5b668b3e2bc31f9d75bbb2a3f6427bef91ef2d581a5bbddda"}
Nov 03 18:27:00.783635 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.783601    6059 scope.go:122] "RemoveContainer" containerID="ffa91730e1e7f2f8f83672dc60b3a8e9275b02e197c9c131191222308dd987cb"
Nov 03 18:27:00.784488 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.784466    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89785e67-bd91-4e22-9e7b-8e88ce58268f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89785e67-bd91-4e22-9e7b-8e88ce58268f-plugins-config-volume\") pod \"89785e67-bd91-4e22-9e7b-8e88ce58268f\" (UID: \"89785e67-bd91-4e22-9e7b-8e88ce58268f\") "
Nov 03 18:27:00.784748 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.784637    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89785e67-bd91-4e22-9e7b-8e88ce58268f-kube-api-access-9cvwz\" (UniqueName: \"kubernetes.io/projected/89785e67-bd91-4e22-9e7b-8e88ce58268f-kube-api-access-9cvwz\") pod \"89785e67-bd91-4e22-9e7b-8e88ce58268f\" (UID: \"89785e67-bd91-4e22-9e7b-8e88ce58268f\") "
Nov 03 18:27:00.784748 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.784691    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89785e67-bd91-4e22-9e7b-8e88ce58268f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/89785e67-bd91-4e22-9e7b-8e88ce58268f-routing-overlay-volume\") pod \"89785e67-bd91-4e22-9e7b-8e88ce58268f\" (UID: \"89785e67-bd91-4e22-9e7b-8e88ce58268f\") "
Nov 03 18:27:00.784748 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.784734    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89785e67-bd91-4e22-9e7b-8e88ce58268f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/89785e67-bd91-4e22-9e7b-8e88ce58268f-provider-credentials\") pod \"89785e67-bd91-4e22-9e7b-8e88ce58268f\" (UID: \"89785e67-bd91-4e22-9e7b-8e88ce58268f\") "
Nov 03 18:27:00.785034 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.785009    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89785e67-bd91-4e22-9e7b-8e88ce58268f-plugins-config-volume" pod "89785e67-bd91-4e22-9e7b-8e88ce58268f" (UID: "89785e67-bd91-4e22-9e7b-8e88ce58268f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:27:00.785098 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.785032    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89785e67-bd91-4e22-9e7b-8e88ce58268f-routing-overlay-volume" pod "89785e67-bd91-4e22-9e7b-8e88ce58268f" (UID: "89785e67-bd91-4e22-9e7b-8e88ce58268f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:27:00.785251 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.785229    6059 generic.go:356] "Generic (PLEG): container finished" podID="89785e67-bd91-4e22-9e7b-8e88ce58268f" containerID="3c2d93b3bbb599c8b77dd41e760bcffdca0fdbe3bcf7004252d649f40478e113" exitCode=0
Nov 03 18:27:00.785331 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.785290    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"
Nov 03 18:27:00.785425 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.785287    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b" event={"ID":"89785e67-bd91-4e22-9e7b-8e88ce58268f","Type":"ContainerDied","Data":"3c2d93b3bbb599c8b77dd41e760bcffdca0fdbe3bcf7004252d649f40478e113"}
Nov 03 18:27:00.785486 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.785452    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b" event={"ID":"89785e67-bd91-4e22-9e7b-8e88ce58268f","Type":"ContainerDied","Data":"342e32980b9d6342d9cca20ddc389de75276e9aeb43125e52925f0297b61cf7e"}
Nov 03 18:27:00.786981 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.786953    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89785e67-bd91-4e22-9e7b-8e88ce58268f-provider-credentials" pod "89785e67-bd91-4e22-9e7b-8e88ce58268f" (UID: "89785e67-bd91-4e22-9e7b-8e88ce58268f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:27:00.787060 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.786982    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89785e67-bd91-4e22-9e7b-8e88ce58268f-kube-api-access-9cvwz" pod "89785e67-bd91-4e22-9e7b-8e88ce58268f" (UID: "89785e67-bd91-4e22-9e7b-8e88ce58268f"). InnerVolumeSpecName "kube-api-access-9cvwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:27:00.794324 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.794299    6059 scope.go:122] "RemoveContainer" containerID="ffa91730e1e7f2f8f83672dc60b3a8e9275b02e197c9c131191222308dd987cb"
Nov 03 18:27:00.794677 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:27:00.794656    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffa91730e1e7f2f8f83672dc60b3a8e9275b02e197c9c131191222308dd987cb\": container with ID starting with ffa91730e1e7f2f8f83672dc60b3a8e9275b02e197c9c131191222308dd987cb not found: ID does not exist" containerID="ffa91730e1e7f2f8f83672dc60b3a8e9275b02e197c9c131191222308dd987cb"
Nov 03 18:27:00.794747 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.794685    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa91730e1e7f2f8f83672dc60b3a8e9275b02e197c9c131191222308dd987cb"} err="failed to get container status \"ffa91730e1e7f2f8f83672dc60b3a8e9275b02e197c9c131191222308dd987cb\": rpc error: code = NotFound desc = could not find container \"ffa91730e1e7f2f8f83672dc60b3a8e9275b02e197c9c131191222308dd987cb\": container with ID starting with ffa91730e1e7f2f8f83672dc60b3a8e9275b02e197c9c131191222308dd987cb not found: ID does not exist"
Nov 03 18:27:00.794747 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.794701    6059 scope.go:122] "RemoveContainer" containerID="3c2d93b3bbb599c8b77dd41e760bcffdca0fdbe3bcf7004252d649f40478e113"
Nov 03 18:27:00.803321 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.803303    6059 scope.go:122] "RemoveContainer" containerID="3c2d93b3bbb599c8b77dd41e760bcffdca0fdbe3bcf7004252d649f40478e113"
Nov 03 18:27:00.803659 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:27:00.803633    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2d93b3bbb599c8b77dd41e760bcffdca0fdbe3bcf7004252d649f40478e113\": container with ID starting with 3c2d93b3bbb599c8b77dd41e760bcffdca0fdbe3bcf7004252d649f40478e113 not found: ID does not exist" containerID="3c2d93b3bbb599c8b77dd41e760bcffdca0fdbe3bcf7004252d649f40478e113"
Nov 03 18:27:00.803706 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.803667    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2d93b3bbb599c8b77dd41e760bcffdca0fdbe3bcf7004252d649f40478e113"} err="failed to get container status \"3c2d93b3bbb599c8b77dd41e760bcffdca0fdbe3bcf7004252d649f40478e113\": rpc error: code = NotFound desc = could not find container \"3c2d93b3bbb599c8b77dd41e760bcffdca0fdbe3bcf7004252d649f40478e113\": container with ID starting with 3c2d93b3bbb599c8b77dd41e760bcffdca0fdbe3bcf7004252d649f40478e113 not found: ID does not exist"
Nov 03 18:27:00.885684 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.885648    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-provider-credentials\") pod \"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6\" (UID: \"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6\") "
Nov 03 18:27:00.885871 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.885746    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-kube-api-access-q7xjm\" (UniqueName: \"kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-kube-api-access-q7xjm\") pod \"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6\" (UID: \"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6\") "
Nov 03 18:27:00.885871 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.885772    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-routing-overlay-volume\") pod \"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6\" (UID: \"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6\") "
Nov 03 18:27:00.885960 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.885890    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-plugins-config-volume\") pod \"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6\" (UID: \"ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6\") "
Nov 03 18:27:00.886234 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.886208    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-routing-overlay-volume" pod "ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6" (UID: "ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:27:00.886343 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.886216    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cvwz\" (UniqueName: \"kubernetes.io/projected/89785e67-bd91-4e22-9e7b-8e88ce58268f-kube-api-access-9cvwz\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:00.886343 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.886259    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/89785e67-bd91-4e22-9e7b-8e88ce58268f-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:00.886343 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.886269    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/89785e67-bd91-4e22-9e7b-8e88ce58268f-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:00.886343 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.886278    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89785e67-bd91-4e22-9e7b-8e88ce58268f-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:00.886343 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.886282    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-plugins-config-volume" pod "ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6" (UID: "ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:27:00.887833 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.887809    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-provider-credentials" pod "ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6" (UID: "ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:27:00.887938 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.887854    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-kube-api-access-q7xjm" pod "ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6" (UID: "ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6"). InnerVolumeSpecName "kube-api-access-q7xjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:27:00.986986 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.986948    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:00.986986 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.986982    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:00.986986 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.986994    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7xjm\" (UniqueName: \"kubernetes.io/projected/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-kube-api-access-q7xjm\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:00.987246 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:00.987009    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:01.211656 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:01.211617    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"]
Nov 03 18:27:01.215660 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:01.215625    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6d974r6s2"]
Nov 03 18:27:01.341058 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:01.340961    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6" path="/var/lib/kubelet/pods/ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6/volumes"
Nov 03 18:27:01.341294 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:01.341269    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"]
Nov 03 18:27:01.341385 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:01.341300    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-58f4rlm5b"]
Nov 03 18:27:01.457476 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:01.457442    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:03.335158 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:03.335122    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89785e67-bd91-4e22-9e7b-8e88ce58268f" path="/var/lib/kubelet/pods/89785e67-bd91-4e22-9e7b-8e88ce58268f/volumes"
Nov 03 18:27:05.419741 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:05.419706    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp"]
Nov 03 18:27:05.699215 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:05.699125    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:05.937448 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:05.937414    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:06.495259 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:06.495226    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:06.861068 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:27:06.860980    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:27:07.140850 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:07.140754    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:10.420452 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:10.420398    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp" podUID="be0c0d23-e8b2-43aa-a7a2-921718dde1b1" containerName="payload-pre-processing" containerID="cri-o://ddd5efa584a6f5effac4ccf7cf21bf988e88d6ae2c87fce3a871144fac29a206" gracePeriod=25
Nov 03 18:27:11.451129 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:11.451097    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:13.038453 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:13.038418    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:15.657730 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:15.657696    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:16.815415 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:16.815350    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp" podUID="be0c0d23-e8b2-43aa-a7a2-921718dde1b1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:27:17.455035 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:17.454997    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:18.654931 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:18.654886    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:18.837048 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:18.837014    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:20.247802 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:20.247768    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:20.895922 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:20.895872    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"]
Nov 03 18:27:20.896162 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:20.896118    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk" podUID="3b813b37-7f95-4138-a42d-bb5af19d7e09" containerName="istio-proxy" containerID="cri-o://b08716f414dd05f8458ae06a0a338f6cb3d95b906d01c0391411ef870bded263" gracePeriod=30
Nov 03 18:27:20.929856 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:20.929816    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:21.718350 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:21.718315    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:22.129840 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:22.129766    6059 patch_prober.go:29] interesting pod/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:27:22.129983 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:22.129833    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk" podUID="3b813b37-7f95-4138-a42d-bb5af19d7e09" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:27:22.526240 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:22.526208    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:26.149771 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.149742    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 18:27:26.227965 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.227921    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-workload-certs\") pod \"3b813b37-7f95-4138-a42d-bb5af19d7e09\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") "
Nov 03 18:27:26.228159 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.228001    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-envoy\") pod \"3b813b37-7f95-4138-a42d-bb5af19d7e09\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") "
Nov 03 18:27:26.228159 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.228041    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b813b37-7f95-4138-a42d-bb5af19d7e09-kube-api-access-44nm5\" (UniqueName: \"kubernetes.io/projected/3b813b37-7f95-4138-a42d-bb5af19d7e09-kube-api-access-44nm5\") pod \"3b813b37-7f95-4138-a42d-bb5af19d7e09\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") "
Nov 03 18:27:26.228159 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.228080    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-workload-socket\") pod \"3b813b37-7f95-4138-a42d-bb5af19d7e09\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") "
Nov 03 18:27:26.228381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.228188    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-credential-socket\") pod \"3b813b37-7f95-4138-a42d-bb5af19d7e09\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") "
Nov 03 18:27:26.228381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.228219    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-token\" (UniqueName: \"kubernetes.io/projected/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-token\") pod \"3b813b37-7f95-4138-a42d-bb5af19d7e09\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") "
Nov 03 18:27:26.228381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.228185    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-workload-certs" pod "3b813b37-7f95-4138-a42d-bb5af19d7e09" (UID: "3b813b37-7f95-4138-a42d-bb5af19d7e09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:27:26.228381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.228292    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-data\") pod \"3b813b37-7f95-4138-a42d-bb5af19d7e09\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") "
Nov 03 18:27:26.228381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.228329    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-podinfo\") pod \"3b813b37-7f95-4138-a42d-bb5af19d7e09\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") "
Nov 03 18:27:26.228655 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.228384    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b813b37-7f95-4138-a42d-bb5af19d7e09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b813b37-7f95-4138-a42d-bb5af19d7e09-istiod-ca-cert\") pod \"3b813b37-7f95-4138-a42d-bb5af19d7e09\" (UID: \"3b813b37-7f95-4138-a42d-bb5af19d7e09\") "
Nov 03 18:27:26.228655 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.228398    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-workload-socket" pod "3b813b37-7f95-4138-a42d-bb5af19d7e09" (UID: "3b813b37-7f95-4138-a42d-bb5af19d7e09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:27:26.228655 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.228447    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-credential-socket" pod "3b813b37-7f95-4138-a42d-bb5af19d7e09" (UID: "3b813b37-7f95-4138-a42d-bb5af19d7e09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:27:26.228655 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.228568    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-data" pod "3b813b37-7f95-4138-a42d-bb5af19d7e09" (UID: "3b813b37-7f95-4138-a42d-bb5af19d7e09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:27:26.228896 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.228806    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:26.228896 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.228828    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:26.228896 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.228844    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:26.228896 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.228847    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b813b37-7f95-4138-a42d-bb5af19d7e09-istiod-ca-cert" pod "3b813b37-7f95-4138-a42d-bb5af19d7e09" (UID: "3b813b37-7f95-4138-a42d-bb5af19d7e09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:27:26.228896 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.228859    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:26.230328 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.230304    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-envoy" pod "3b813b37-7f95-4138-a42d-bb5af19d7e09" (UID: "3b813b37-7f95-4138-a42d-bb5af19d7e09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:27:26.230511 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.230346    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b813b37-7f95-4138-a42d-bb5af19d7e09-kube-api-access-44nm5" pod "3b813b37-7f95-4138-a42d-bb5af19d7e09" (UID: "3b813b37-7f95-4138-a42d-bb5af19d7e09"). InnerVolumeSpecName "kube-api-access-44nm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:27:26.230511 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.230446    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-token" pod "3b813b37-7f95-4138-a42d-bb5af19d7e09" (UID: "3b813b37-7f95-4138-a42d-bb5af19d7e09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:27:26.230600 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.230523    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-podinfo" pod "3b813b37-7f95-4138-a42d-bb5af19d7e09" (UID: "3b813b37-7f95-4138-a42d-bb5af19d7e09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:27:26.329512 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.329480    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:26.329512 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.329506    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:26.329512 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.329517    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b813b37-7f95-4138-a42d-bb5af19d7e09-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:26.329765 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.329525    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b813b37-7f95-4138-a42d-bb5af19d7e09-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:26.329765 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.329534    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44nm5\" (UniqueName: \"kubernetes.io/projected/3b813b37-7f95-4138-a42d-bb5af19d7e09-kube-api-access-44nm5\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:26.815575 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.815540    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp" podUID="be0c0d23-e8b2-43aa-a7a2-921718dde1b1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:27:26.935344 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.935308    6059 generic.go:356] "Generic (PLEG): container finished" podID="3b813b37-7f95-4138-a42d-bb5af19d7e09" containerID="b08716f414dd05f8458ae06a0a338f6cb3d95b906d01c0391411ef870bded263" exitCode=0
Nov 03 18:27:26.935561 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.935389    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"
Nov 03 18:27:26.935561 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.935391    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk" event={"ID":"3b813b37-7f95-4138-a42d-bb5af19d7e09","Type":"ContainerDied","Data":"b08716f414dd05f8458ae06a0a338f6cb3d95b906d01c0391411ef870bded263"}
Nov 03 18:27:26.935561 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.935453    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk" event={"ID":"3b813b37-7f95-4138-a42d-bb5af19d7e09","Type":"ContainerDied","Data":"758ab79bff010b0cad8a0b6fe441cf5a40a37f857e745e4f69eb8d52cad49557"}
Nov 03 18:27:26.935561 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.935480    6059 scope.go:122] "RemoveContainer" containerID="b08716f414dd05f8458ae06a0a338f6cb3d95b906d01c0391411ef870bded263"
Nov 03 18:27:26.945504 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.945482    6059 scope.go:122] "RemoveContainer" containerID="9a2e2e2e41accf8a19e9da57a36975a41c081d143904704ca8e494637fd408f0"
Nov 03 18:27:26.954854 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.954833    6059 scope.go:122] "RemoveContainer" containerID="b08716f414dd05f8458ae06a0a338f6cb3d95b906d01c0391411ef870bded263"
Nov 03 18:27:26.955136 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:27:26.955116    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b08716f414dd05f8458ae06a0a338f6cb3d95b906d01c0391411ef870bded263\": container with ID starting with b08716f414dd05f8458ae06a0a338f6cb3d95b906d01c0391411ef870bded263 not found: ID does not exist" containerID="b08716f414dd05f8458ae06a0a338f6cb3d95b906d01c0391411ef870bded263"
Nov 03 18:27:26.955190 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.955147    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08716f414dd05f8458ae06a0a338f6cb3d95b906d01c0391411ef870bded263"} err="failed to get container status \"b08716f414dd05f8458ae06a0a338f6cb3d95b906d01c0391411ef870bded263\": rpc error: code = NotFound desc = could not find container \"b08716f414dd05f8458ae06a0a338f6cb3d95b906d01c0391411ef870bded263\": container with ID starting with b08716f414dd05f8458ae06a0a338f6cb3d95b906d01c0391411ef870bded263 not found: ID does not exist"
Nov 03 18:27:26.955190 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.955167    6059 scope.go:122] "RemoveContainer" containerID="9a2e2e2e41accf8a19e9da57a36975a41c081d143904704ca8e494637fd408f0"
Nov 03 18:27:26.955433 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:27:26.955410    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a2e2e2e41accf8a19e9da57a36975a41c081d143904704ca8e494637fd408f0\": container with ID starting with 9a2e2e2e41accf8a19e9da57a36975a41c081d143904704ca8e494637fd408f0 not found: ID does not exist" containerID="9a2e2e2e41accf8a19e9da57a36975a41c081d143904704ca8e494637fd408f0"
Nov 03 18:27:26.955489 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:26.955440    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a2e2e2e41accf8a19e9da57a36975a41c081d143904704ca8e494637fd408f0"} err="failed to get container status \"9a2e2e2e41accf8a19e9da57a36975a41c081d143904704ca8e494637fd408f0\": rpc error: code = NotFound desc = could not find container \"9a2e2e2e41accf8a19e9da57a36975a41c081d143904704ca8e494637fd408f0\": container with ID starting with 9a2e2e2e41accf8a19e9da57a36975a41c081d143904704ca8e494637fd408f0 not found: ID does not exist"
Nov 03 18:27:27.080649 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:27.080552    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25"]
Nov 03 18:27:27.158369 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:27.158317    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"]
Nov 03 18:27:27.162523 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:27.162498    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-029ca0-openshift-default-665dcdc7cc-q24xk"]
Nov 03 18:27:27.190869 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:27.190835    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5"]
Nov 03 18:27:27.335168 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:27.335089    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b813b37-7f95-4138-a42d-bb5af19d7e09" path="/var/lib/kubelet/pods/3b813b37-7f95-4138-a42d-bb5af19d7e09/volumes"
Nov 03 18:27:27.443077 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:27.443039    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:27.629229 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:27.629113    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:28.920066 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:28.920024    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:30.666278 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:30.666252    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp"
Nov 03 18:27:30.768209 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:30.768179    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be0c0d23-e8b2-43aa-a7a2-921718dde1b1-kube-api-access-xd88r\" (UniqueName: \"kubernetes.io/projected/be0c0d23-e8b2-43aa-a7a2-921718dde1b1-kube-api-access-xd88r\") pod \"be0c0d23-e8b2-43aa-a7a2-921718dde1b1\" (UID: \"be0c0d23-e8b2-43aa-a7a2-921718dde1b1\") "
Nov 03 18:27:30.768409 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:30.768320    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be0c0d23-e8b2-43aa-a7a2-921718dde1b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be0c0d23-e8b2-43aa-a7a2-921718dde1b1-plugins-config-volume\") pod \"be0c0d23-e8b2-43aa-a7a2-921718dde1b1\" (UID: \"be0c0d23-e8b2-43aa-a7a2-921718dde1b1\") "
Nov 03 18:27:30.768736 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:30.768708    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be0c0d23-e8b2-43aa-a7a2-921718dde1b1-plugins-config-volume" pod "be0c0d23-e8b2-43aa-a7a2-921718dde1b1" (UID: "be0c0d23-e8b2-43aa-a7a2-921718dde1b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:27:30.770188 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:30.770167    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0c0d23-e8b2-43aa-a7a2-921718dde1b1-kube-api-access-xd88r" pod "be0c0d23-e8b2-43aa-a7a2-921718dde1b1" (UID: "be0c0d23-e8b2-43aa-a7a2-921718dde1b1"). InnerVolumeSpecName "kube-api-access-xd88r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:27:30.870095 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:30.870064    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be0c0d23-e8b2-43aa-a7a2-921718dde1b1-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:30.870095 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:30.870093    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd88r\" (UniqueName: \"kubernetes.io/projected/be0c0d23-e8b2-43aa-a7a2-921718dde1b1-kube-api-access-xd88r\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:30.959687 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:30.959650    6059 generic.go:356] "Generic (PLEG): container finished" podID="be0c0d23-e8b2-43aa-a7a2-921718dde1b1" containerID="ddd5efa584a6f5effac4ccf7cf21bf988e88d6ae2c87fce3a871144fac29a206" exitCode=0
Nov 03 18:27:30.959839 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:30.959703    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp"
Nov 03 18:27:30.959839 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:30.959733    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp" event={"ID":"be0c0d23-e8b2-43aa-a7a2-921718dde1b1","Type":"ContainerDied","Data":"ddd5efa584a6f5effac4ccf7cf21bf988e88d6ae2c87fce3a871144fac29a206"}
Nov 03 18:27:30.959839 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:30.959789    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp" event={"ID":"be0c0d23-e8b2-43aa-a7a2-921718dde1b1","Type":"ContainerDied","Data":"5c370a91ad0206d5388bad2b49528ed91462a4757fc8c23a20861c67f0eda568"}
Nov 03 18:27:30.959839 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:30.959811    6059 scope.go:122] "RemoveContainer" containerID="ddd5efa584a6f5effac4ccf7cf21bf988e88d6ae2c87fce3a871144fac29a206"
Nov 03 18:27:30.969787 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:30.969766    6059 scope.go:122] "RemoveContainer" containerID="ddd5efa584a6f5effac4ccf7cf21bf988e88d6ae2c87fce3a871144fac29a206"
Nov 03 18:27:30.970057 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:27:30.970034    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddd5efa584a6f5effac4ccf7cf21bf988e88d6ae2c87fce3a871144fac29a206\": container with ID starting with ddd5efa584a6f5effac4ccf7cf21bf988e88d6ae2c87fce3a871144fac29a206 not found: ID does not exist" containerID="ddd5efa584a6f5effac4ccf7cf21bf988e88d6ae2c87fce3a871144fac29a206"
Nov 03 18:27:30.970150 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:30.970061    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddd5efa584a6f5effac4ccf7cf21bf988e88d6ae2c87fce3a871144fac29a206"} err="failed to get container status \"ddd5efa584a6f5effac4ccf7cf21bf988e88d6ae2c87fce3a871144fac29a206\": rpc error: code = NotFound desc = could not find container \"ddd5efa584a6f5effac4ccf7cf21bf988e88d6ae2c87fce3a871144fac29a206\": container with ID starting with ddd5efa584a6f5effac4ccf7cf21bf988e88d6ae2c87fce3a871144fac29a206 not found: ID does not exist"
Nov 03 18:27:31.129729 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:31.129637    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp"]
Nov 03 18:27:31.133479 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:31.133451    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-029ca0-67d8d49879-zjpxp"]
Nov 03 18:27:31.335985 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:31.335942    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be0c0d23-e8b2-43aa-a7a2-921718dde1b1" path="/var/lib/kubelet/pods/be0c0d23-e8b2-43aa-a7a2-921718dde1b1/volumes"
Nov 03 18:27:32.081685 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:32.081612    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25" podUID="1341c5b9-3ac7-4b13-8026-cb22e3fea9c8" containerName="payload-processing" containerID="cri-o://78e671491c9123a83d3a64432972863734f42fb24d56011f4c9c8d2abfbde142" gracePeriod=25
Nov 03 18:27:32.191953 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:32.191888    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5" podUID="8c4f4203-c72f-408c-9da3-f83a246aaeda" containerName="payload-pre-processing" containerID="cri-o://0f5ff96bc28bbdbb043a11dbc592635db7d096704e38ca8e38b7a40c05271e28" gracePeriod=25
Nov 03 18:27:32.654074 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:32.654035    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:34.526455 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:34.526419    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:36.854555 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:36.854522    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:36.860746 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:27:36.860718    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:27:37.924098 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:37.924058    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5" podUID="8c4f4203-c72f-408c-9da3-f83a246aaeda" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:27:37.924583 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:37.924061    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25" podUID="1341c5b9-3ac7-4b13-8026-cb22e3fea9c8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:27:39.981650 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:39.981618    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:27:39.998500 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:39.998465    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:27:39.999833 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:39.999809    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:27:40.021671 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:40.021647    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:27:40.330439 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:40.330324    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:40.508706 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:40.508665    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:41.893349 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:41.893310    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:42.383719 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:42.383676    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:44.738586 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:44.738555    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-n4k4d_d8bf1633-07f3-4660-9268-cd96bfee740a/kuadrant-console-plugin/0.log"
Nov 03 18:27:45.942804 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:45.942765    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:47.594301 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:47.594271    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5fdd44f589-9c6qf_e9f4c7ae-adf2-4c9a-a9be-a8b5829b69d2/manager/0.log"
Nov 03 18:27:47.923600 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:47.923515    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5" podUID="8c4f4203-c72f-408c-9da3-f83a246aaeda" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:27:47.923771 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:47.923528    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25" podUID="1341c5b9-3ac7-4b13-8026-cb22e3fea9c8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:27:52.323898 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:52.323864    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25"
Nov 03 18:27:52.438714 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:52.438692    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5"
Nov 03 18:27:52.496992 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:52.496965    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1341c5b9-3ac7-4b13-8026-cb22e3fea9c8-kube-api-access-mx48x\" (UniqueName: \"kubernetes.io/projected/1341c5b9-3ac7-4b13-8026-cb22e3fea9c8-kube-api-access-mx48x\") pod \"1341c5b9-3ac7-4b13-8026-cb22e3fea9c8\" (UID: \"1341c5b9-3ac7-4b13-8026-cb22e3fea9c8\") "
Nov 03 18:27:52.497126 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:52.497055    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1341c5b9-3ac7-4b13-8026-cb22e3fea9c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1341c5b9-3ac7-4b13-8026-cb22e3fea9c8-plugins-config-volume\") pod \"1341c5b9-3ac7-4b13-8026-cb22e3fea9c8\" (UID: \"1341c5b9-3ac7-4b13-8026-cb22e3fea9c8\") "
Nov 03 18:27:52.497447 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:52.497419    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1341c5b9-3ac7-4b13-8026-cb22e3fea9c8-plugins-config-volume" pod "1341c5b9-3ac7-4b13-8026-cb22e3fea9c8" (UID: "1341c5b9-3ac7-4b13-8026-cb22e3fea9c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:27:52.498974 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:52.498953    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1341c5b9-3ac7-4b13-8026-cb22e3fea9c8-kube-api-access-mx48x" pod "1341c5b9-3ac7-4b13-8026-cb22e3fea9c8" (UID: "1341c5b9-3ac7-4b13-8026-cb22e3fea9c8"). InnerVolumeSpecName "kube-api-access-mx48x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:27:52.597784 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:52.597755    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c4f4203-c72f-408c-9da3-f83a246aaeda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c4f4203-c72f-408c-9da3-f83a246aaeda-plugins-config-volume\") pod \"8c4f4203-c72f-408c-9da3-f83a246aaeda\" (UID: \"8c4f4203-c72f-408c-9da3-f83a246aaeda\") "
Nov 03 18:27:52.597969 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:52.597809    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c4f4203-c72f-408c-9da3-f83a246aaeda-kube-api-access-h2gvg\" (UniqueName: \"kubernetes.io/projected/8c4f4203-c72f-408c-9da3-f83a246aaeda-kube-api-access-h2gvg\") pod \"8c4f4203-c72f-408c-9da3-f83a246aaeda\" (UID: \"8c4f4203-c72f-408c-9da3-f83a246aaeda\") "
Nov 03 18:27:52.598046 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:52.598003    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx48x\" (UniqueName: \"kubernetes.io/projected/1341c5b9-3ac7-4b13-8026-cb22e3fea9c8-kube-api-access-mx48x\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:52.598046 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:52.598013    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1341c5b9-3ac7-4b13-8026-cb22e3fea9c8-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:52.598182 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:52.598159    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c4f4203-c72f-408c-9da3-f83a246aaeda-plugins-config-volume" pod "8c4f4203-c72f-408c-9da3-f83a246aaeda" (UID: "8c4f4203-c72f-408c-9da3-f83a246aaeda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:27:52.599783 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:52.599762    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4f4203-c72f-408c-9da3-f83a246aaeda-kube-api-access-h2gvg" pod "8c4f4203-c72f-408c-9da3-f83a246aaeda" (UID: "8c4f4203-c72f-408c-9da3-f83a246aaeda"). InnerVolumeSpecName "kube-api-access-h2gvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:27:52.698896 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:52.698804    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c4f4203-c72f-408c-9da3-f83a246aaeda-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:52.698896 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:52.698835    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2gvg\" (UniqueName: \"kubernetes.io/projected/8c4f4203-c72f-408c-9da3-f83a246aaeda-kube-api-access-h2gvg\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:27:53.092651 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:53.092616    6059 generic.go:356] "Generic (PLEG): container finished" podID="1341c5b9-3ac7-4b13-8026-cb22e3fea9c8" containerID="78e671491c9123a83d3a64432972863734f42fb24d56011f4c9c8d2abfbde142" exitCode=0
Nov 03 18:27:53.092850 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:53.092669    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25"
Nov 03 18:27:53.092850 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:53.092706    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25" event={"ID":"1341c5b9-3ac7-4b13-8026-cb22e3fea9c8","Type":"ContainerDied","Data":"78e671491c9123a83d3a64432972863734f42fb24d56011f4c9c8d2abfbde142"}
Nov 03 18:27:53.092850 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:53.092771    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25" event={"ID":"1341c5b9-3ac7-4b13-8026-cb22e3fea9c8","Type":"ContainerDied","Data":"a5ef8c3f691068c96cdd0461145a70c3d70c485cd0d649e1f8f7de2e4149e09f"}
Nov 03 18:27:53.092850 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:53.092799    6059 scope.go:122] "RemoveContainer" containerID="78e671491c9123a83d3a64432972863734f42fb24d56011f4c9c8d2abfbde142"
Nov 03 18:27:53.094313 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:53.094279    6059 generic.go:356] "Generic (PLEG): container finished" podID="8c4f4203-c72f-408c-9da3-f83a246aaeda" containerID="0f5ff96bc28bbdbb043a11dbc592635db7d096704e38ca8e38b7a40c05271e28" exitCode=0
Nov 03 18:27:53.094441 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:53.094330    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5"
Nov 03 18:27:53.094441 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:53.094424    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5" event={"ID":"8c4f4203-c72f-408c-9da3-f83a246aaeda","Type":"ContainerDied","Data":"0f5ff96bc28bbdbb043a11dbc592635db7d096704e38ca8e38b7a40c05271e28"}
Nov 03 18:27:53.094580 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:53.094465    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5" event={"ID":"8c4f4203-c72f-408c-9da3-f83a246aaeda","Type":"ContainerDied","Data":"92965ca1fe811e2b202d77835d2504339074603fd9b4533db99d901608cb6bea"}
Nov 03 18:27:53.104803 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:53.104745    6059 scope.go:122] "RemoveContainer" containerID="78e671491c9123a83d3a64432972863734f42fb24d56011f4c9c8d2abfbde142"
Nov 03 18:27:53.105123 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:27:53.105095    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e671491c9123a83d3a64432972863734f42fb24d56011f4c9c8d2abfbde142\": container with ID starting with 78e671491c9123a83d3a64432972863734f42fb24d56011f4c9c8d2abfbde142 not found: ID does not exist" containerID="78e671491c9123a83d3a64432972863734f42fb24d56011f4c9c8d2abfbde142"
Nov 03 18:27:53.105221 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:53.105130    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e671491c9123a83d3a64432972863734f42fb24d56011f4c9c8d2abfbde142"} err="failed to get container status \"78e671491c9123a83d3a64432972863734f42fb24d56011f4c9c8d2abfbde142\": rpc error: code = NotFound desc = could not find container \"78e671491c9123a83d3a64432972863734f42fb24d56011f4c9c8d2abfbde142\": container with ID starting with 78e671491c9123a83d3a64432972863734f42fb24d56011f4c9c8d2abfbde142 not found: ID does not exist"
Nov 03 18:27:53.105221 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:53.105149    6059 scope.go:122] "RemoveContainer" containerID="0f5ff96bc28bbdbb043a11dbc592635db7d096704e38ca8e38b7a40c05271e28"
Nov 03 18:27:53.113842 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:53.113824    6059 scope.go:122] "RemoveContainer" containerID="0f5ff96bc28bbdbb043a11dbc592635db7d096704e38ca8e38b7a40c05271e28"
Nov 03 18:27:53.114109 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:27:53.114087    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f5ff96bc28bbdbb043a11dbc592635db7d096704e38ca8e38b7a40c05271e28\": container with ID starting with 0f5ff96bc28bbdbb043a11dbc592635db7d096704e38ca8e38b7a40c05271e28 not found: ID does not exist" containerID="0f5ff96bc28bbdbb043a11dbc592635db7d096704e38ca8e38b7a40c05271e28"
Nov 03 18:27:53.114174 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:53.114116    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5ff96bc28bbdbb043a11dbc592635db7d096704e38ca8e38b7a40c05271e28"} err="failed to get container status \"0f5ff96bc28bbdbb043a11dbc592635db7d096704e38ca8e38b7a40c05271e28\": rpc error: code = NotFound desc = could not find container \"0f5ff96bc28bbdbb043a11dbc592635db7d096704e38ca8e38b7a40c05271e28\": container with ID starting with 0f5ff96bc28bbdbb043a11dbc592635db7d096704e38ca8e38b7a40c05271e28 not found: ID does not exist"
Nov 03 18:27:53.263369 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:53.263322    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5"]
Nov 03 18:27:53.267315 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:53.267290    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8c9b0f-856c4599dc-zgdh5"]
Nov 03 18:27:53.334448 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:53.334418    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c4f4203-c72f-408c-9da3-f83a246aaeda" path="/var/lib/kubelet/pods/8c4f4203-c72f-408c-9da3-f83a246aaeda/volumes"
Nov 03 18:27:53.380781 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:53.380700    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25"]
Nov 03 18:27:53.384912 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:53.384889    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8c9b0f-5c88c87c77-dcp25"]
Nov 03 18:27:54.174827 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.174794    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:54.667706 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.667666    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"]
Nov 03 18:27:54.668274 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668252    6059 cpu_manager.go:414] "containerMap: removing container" podUID="3b813b37-7f95-4138-a42d-bb5af19d7e09" containerName="istio-proxy"
Nov 03 18:27:54.668399 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668276    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b813b37-7f95-4138-a42d-bb5af19d7e09" containerName="istio-proxy"
Nov 03 18:27:54.668399 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668302    6059 cpu_manager.go:414] "containerMap: removing container" podUID="d3d58540-7d6f-459b-bcc5-e21804a1085b" containerName="payload-processing"
Nov 03 18:27:54.668399 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668311    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d58540-7d6f-459b-bcc5-e21804a1085b" containerName="payload-processing"
Nov 03 18:27:54.668399 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668322    6059 cpu_manager.go:414] "containerMap: removing container" podUID="4e84c1e6-3809-4ce6-9430-7c7ec1f26a33" containerName="revoke-keys"
Nov 03 18:27:54.668399 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668330    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e84c1e6-3809-4ce6-9430-7c7ec1f26a33" containerName="revoke-keys"
Nov 03 18:27:54.668399 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668347    6059 cpu_manager.go:414] "containerMap: removing container" podUID="be0c0d23-e8b2-43aa-a7a2-921718dde1b1" containerName="payload-pre-processing"
Nov 03 18:27:54.668399 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668370    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be0c0d23-e8b2-43aa-a7a2-921718dde1b1" containerName="payload-pre-processing"
Nov 03 18:27:54.668399 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668400    6059 cpu_manager.go:414] "containerMap: removing container" podUID="1341c5b9-3ac7-4b13-8026-cb22e3fea9c8" containerName="payload-processing"
Nov 03 18:27:54.668820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668407    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1341c5b9-3ac7-4b13-8026-cb22e3fea9c8" containerName="payload-processing"
Nov 03 18:27:54.668820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668416    6059 cpu_manager.go:414] "containerMap: removing container" podUID="8c4f4203-c72f-408c-9da3-f83a246aaeda" containerName="payload-pre-processing"
Nov 03 18:27:54.668820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668423    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c4f4203-c72f-408c-9da3-f83a246aaeda" containerName="payload-pre-processing"
Nov 03 18:27:54.668820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668442    6059 cpu_manager.go:414] "containerMap: removing container" podUID="89785e67-bd91-4e22-9e7b-8e88ce58268f" containerName="payload-processing"
Nov 03 18:27:54.668820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668451    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89785e67-bd91-4e22-9e7b-8e88ce58268f" containerName="payload-processing"
Nov 03 18:27:54.668820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668466    6059 cpu_manager.go:414] "containerMap: removing container" podUID="ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6" containerName="payload-processing"
Nov 03 18:27:54.668820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668473    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6" containerName="payload-processing"
Nov 03 18:27:54.668820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668485    6059 cpu_manager.go:414] "containerMap: removing container" podUID="7cba1dd3-36bd-4832-a4cb-cd3537ac1147" containerName="payload-processing"
Nov 03 18:27:54.668820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668492    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cba1dd3-36bd-4832-a4cb-cd3537ac1147" containerName="payload-processing"
Nov 03 18:27:54.668820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668592    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="be0c0d23-e8b2-43aa-a7a2-921718dde1b1" containerName="payload-pre-processing"
Nov 03 18:27:54.668820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668610    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1341c5b9-3ac7-4b13-8026-cb22e3fea9c8" containerName="payload-processing"
Nov 03 18:27:54.668820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668619    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c4f4203-c72f-408c-9da3-f83a246aaeda" containerName="payload-pre-processing"
Nov 03 18:27:54.668820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668631    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="89785e67-bd91-4e22-9e7b-8e88ce58268f" containerName="payload-processing"
Nov 03 18:27:54.668820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668641    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce33491c-941e-44dd-a4ad-d8ca6a3dbbe6" containerName="payload-processing"
Nov 03 18:27:54.668820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668651    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cba1dd3-36bd-4832-a4cb-cd3537ac1147" containerName="payload-processing"
Nov 03 18:27:54.668820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668659    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b813b37-7f95-4138-a42d-bb5af19d7e09" containerName="istio-proxy"
Nov 03 18:27:54.668820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668671    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d58540-7d6f-459b-bcc5-e21804a1085b" containerName="payload-processing"
Nov 03 18:27:54.668820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.668679    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e84c1e6-3809-4ce6-9430-7c7ec1f26a33" containerName="revoke-keys"
Nov 03 18:27:54.673320 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.673299    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.727706 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.727670    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-007400fa-openshift-default-dockercfg-qm7dp\""
Nov 03 18:27:54.781256 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.781225    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"]
Nov 03 18:27:54.822109 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.822070    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-envoy\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.822109 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.822115    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-data\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.822333 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.822228    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vktbn\" (UniqueName: \"kubernetes.io/projected/4efe012c-c87d-4544-8f34-8fd3495e19da-kube-api-access-vktbn\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.822333 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.822315    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-workload-certs\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.822470 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.822382    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-token\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.822470 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.822418    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4efe012c-c87d-4544-8f34-8fd3495e19da-istiod-ca-cert\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.822470 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.822439    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-credential-socket\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.822594 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.822491    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-podinfo\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.822594 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.822518    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-workload-socket\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.923246 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.923157    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-podinfo\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.923246 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.923194    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-workload-socket\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.923530 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.923334    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-envoy\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.923530 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.923391    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-data\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.923530 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.923432    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vktbn\" (UniqueName: \"kubernetes.io/projected/4efe012c-c87d-4544-8f34-8fd3495e19da-kube-api-access-vktbn\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.923530 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.923512    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-workload-certs\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.923764 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.923559    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-token\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.923764 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.923594    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4efe012c-c87d-4544-8f34-8fd3495e19da-istiod-ca-cert\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.923764 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.923614    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-credential-socket\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.924237 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.924213    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-data\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.924399 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.924380    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-workload-certs\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.924469 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.924402    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-workload-socket\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.924763 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.924739    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4efe012c-c87d-4544-8f34-8fd3495e19da-istiod-ca-cert\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.924944 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.924926    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-credential-socket\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.926096 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.926073    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-envoy\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.926309 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.926288    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-podinfo\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.987083 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.987047    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-token\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:54.987299 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:54.987279    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vktbn\" (UniqueName: \"kubernetes.io/projected/4efe012c-c87d-4544-8f34-8fd3495e19da-kube-api-access-vktbn\") pod \"e2e-ait-007400fa-openshift-default-575fc57557-hqh6h\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") " pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:55.286133 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:55.286093    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:55.335392 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:55.335339    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1341c5b9-3ac7-4b13-8026-cb22e3fea9c8" path="/var/lib/kubelet/pods/1341c5b9-3ac7-4b13-8026-cb22e3fea9c8/volumes"
Nov 03 18:27:55.714067 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:27:55.714031    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efe012c_c87d_4544_8f34_8fd3495e19da.slice/crio-5949405c9bfe61e38b8e3c5f2e099b1599d77b67352f8fa02e0ab6897903f0b6 WatchSource:0}: Error finding container 5949405c9bfe61e38b8e3c5f2e099b1599d77b67352f8fa02e0ab6897903f0b6: Status 404 returned error can't find the container with id 5949405c9bfe61e38b8e3c5f2e099b1599d77b67352f8fa02e0ab6897903f0b6
Nov 03 18:27:55.715177 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:55.715151    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"]
Nov 03 18:27:55.716284 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:55.716254    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:27:55.716345 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:55.716318    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:27:55.716392 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:55.716381    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:27:56.007217 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:56.007108    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:56.114371 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:56.114330    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h" event={"ID":"4efe012c-c87d-4544-8f34-8fd3495e19da","Type":"ContainerStarted","Data":"4345be318e96ec4d7db7d875fd5b987b92b25dbf1c5a3ab5db8c54540e2d84ac"}
Nov 03 18:27:56.114555 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:56.114406    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h" event={"ID":"4efe012c-c87d-4544-8f34-8fd3495e19da","Type":"ContainerStarted","Data":"5949405c9bfe61e38b8e3c5f2e099b1599d77b67352f8fa02e0ab6897903f0b6"}
Nov 03 18:27:56.287198 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:56.287106    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:56.291813 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:56.291748    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h" podStartSLOduration=2.291726937 podStartE2EDuration="2.291726937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:27:56.290005019 +0000 UTC m=+4217.087717800" watchObservedRunningTime="2026-11-03 18:27:56.291726937 +0000 UTC m=+4217.089439719"
Nov 03 18:27:56.293481 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:56.293457    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:57.120105 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:57.120067    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:57.121161 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:57.121146    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:27:58.393299 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:58.393261    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:27:58.583196 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:27:58.583162    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:01.533770 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:01.533736    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:01.815644 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:01.815543    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:02.101411 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.101304    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4"]
Nov 03 18:28:02.101749 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.101735    6059 cpu_manager.go:414] "containerMap: removing container" podUID="3b813b37-7f95-4138-a42d-bb5af19d7e09" containerName="istio-proxy"
Nov 03 18:28:02.101802 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.101751    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b813b37-7f95-4138-a42d-bb5af19d7e09" containerName="istio-proxy"
Nov 03 18:28:02.101843 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.101837    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b813b37-7f95-4138-a42d-bb5af19d7e09" containerName="istio-proxy"
Nov 03 18:28:02.105438 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.105409    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4"
Nov 03 18:28:02.190099 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.190063    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b31ac2-ad6f-4ed0-9431-7a076bfb664c-plugins-config-volume\") pod \"payload-processing-e2e-ait-007400fa-6c47968787-w22f4\" (UID: \"51b31ac2-ad6f-4ed0-9431-7a076bfb664c\") " pod="openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4"
Nov 03 18:28:02.190304 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.190179    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjwvc\" (UniqueName: \"kubernetes.io/projected/51b31ac2-ad6f-4ed0-9431-7a076bfb664c-kube-api-access-sjwvc\") pod \"payload-processing-e2e-ait-007400fa-6c47968787-w22f4\" (UID: \"51b31ac2-ad6f-4ed0-9431-7a076bfb664c\") " pod="openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4"
Nov 03 18:28:02.211374 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.211334    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-007400fa\""
Nov 03 18:28:02.211534 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.211402    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-007400fa-dockercfg-7wld8\""
Nov 03 18:28:02.213437 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.213413    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4"]
Nov 03 18:28:02.265155 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.265124    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv"]
Nov 03 18:28:02.268967 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.268947    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv"
Nov 03 18:28:02.291129 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.291097    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdzt6\" (UniqueName: \"kubernetes.io/projected/266e536b-696a-4d8e-83dd-996685cc3d59-kube-api-access-tdzt6\") pod \"payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv\" (UID: \"266e536b-696a-4d8e-83dd-996685cc3d59\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv"
Nov 03 18:28:02.291241 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.291155    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjwvc\" (UniqueName: \"kubernetes.io/projected/51b31ac2-ad6f-4ed0-9431-7a076bfb664c-kube-api-access-sjwvc\") pod \"payload-processing-e2e-ait-007400fa-6c47968787-w22f4\" (UID: \"51b31ac2-ad6f-4ed0-9431-7a076bfb664c\") " pod="openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4"
Nov 03 18:28:02.291296 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.291282    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b31ac2-ad6f-4ed0-9431-7a076bfb664c-plugins-config-volume\") pod \"payload-processing-e2e-ait-007400fa-6c47968787-w22f4\" (UID: \"51b31ac2-ad6f-4ed0-9431-7a076bfb664c\") " pod="openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4"
Nov 03 18:28:02.291375 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.291309    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266e536b-696a-4d8e-83dd-996685cc3d59-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv\" (UID: \"266e536b-696a-4d8e-83dd-996685cc3d59\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv"
Nov 03 18:28:02.292013 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.291995    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b31ac2-ad6f-4ed0-9431-7a076bfb664c-plugins-config-volume\") pod \"payload-processing-e2e-ait-007400fa-6c47968787-w22f4\" (UID: \"51b31ac2-ad6f-4ed0-9431-7a076bfb664c\") " pod="openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4"
Nov 03 18:28:02.315697 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.315669    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv"]
Nov 03 18:28:02.354269 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.354198    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjwvc\" (UniqueName: \"kubernetes.io/projected/51b31ac2-ad6f-4ed0-9431-7a076bfb664c-kube-api-access-sjwvc\") pod \"payload-processing-e2e-ait-007400fa-6c47968787-w22f4\" (UID: \"51b31ac2-ad6f-4ed0-9431-7a076bfb664c\") " pod="openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4"
Nov 03 18:28:02.391983 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.391948    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266e536b-696a-4d8e-83dd-996685cc3d59-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv\" (UID: \"266e536b-696a-4d8e-83dd-996685cc3d59\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv"
Nov 03 18:28:02.392157 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.392004    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdzt6\" (UniqueName: \"kubernetes.io/projected/266e536b-696a-4d8e-83dd-996685cc3d59-kube-api-access-tdzt6\") pod \"payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv\" (UID: \"266e536b-696a-4d8e-83dd-996685cc3d59\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv"
Nov 03 18:28:02.392647 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.392622    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266e536b-696a-4d8e-83dd-996685cc3d59-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv\" (UID: \"266e536b-696a-4d8e-83dd-996685cc3d59\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv"
Nov 03 18:28:02.418648 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.418614    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4"
Nov 03 18:28:02.455388 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.455337    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdzt6\" (UniqueName: \"kubernetes.io/projected/266e536b-696a-4d8e-83dd-996685cc3d59-kube-api-access-tdzt6\") pod \"payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv\" (UID: \"266e536b-696a-4d8e-83dd-996685cc3d59\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv"
Nov 03 18:28:02.581955 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.581923    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv"
Nov 03 18:28:02.788495 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:28:02.788450    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b31ac2_ad6f_4ed0_9431_7a076bfb664c.slice/crio-878bf39e8013c32438f1f1da19d43fa3fff1bdaaf8e47cd85bcf58c89f004364 WatchSource:0}: Error finding container 878bf39e8013c32438f1f1da19d43fa3fff1bdaaf8e47cd85bcf58c89f004364: Status 404 returned error can't find the container with id 878bf39e8013c32438f1f1da19d43fa3fff1bdaaf8e47cd85bcf58c89f004364
Nov 03 18:28:02.790714 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.790688    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4"]
Nov 03 18:28:02.952745 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:28:02.952711    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266e536b_696a_4d8e_83dd_996685cc3d59.slice/crio-fe2a5ecc568a59be1c3e2de80202bc5c9f61d5418e4c406cf839f8d996233823 WatchSource:0}: Error finding container fe2a5ecc568a59be1c3e2de80202bc5c9f61d5418e4c406cf839f8d996233823: Status 404 returned error can't find the container with id fe2a5ecc568a59be1c3e2de80202bc5c9f61d5418e4c406cf839f8d996233823
Nov 03 18:28:02.954996 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:02.954974    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv"]
Nov 03 18:28:03.156561 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:03.156521    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4" event={"ID":"51b31ac2-ad6f-4ed0-9431-7a076bfb664c","Type":"ContainerStarted","Data":"79b75936fddebddcd88eec1b5062655c4e0ee4eb61eb509d5197fae930a85762"}
Nov 03 18:28:03.156751 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:03.156592    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4" event={"ID":"51b31ac2-ad6f-4ed0-9431-7a076bfb664c","Type":"ContainerStarted","Data":"878bf39e8013c32438f1f1da19d43fa3fff1bdaaf8e47cd85bcf58c89f004364"}
Nov 03 18:28:03.156751 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:03.156630    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4"
Nov 03 18:28:03.158174 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:03.158145    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv" event={"ID":"266e536b-696a-4d8e-83dd-996685cc3d59","Type":"ContainerStarted","Data":"1fca3dd3fc69bac358067edd999c24a78bcc5393ac8d972d2f4e27ef70af41cf"}
Nov 03 18:28:03.158301 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:03.158196    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv" event={"ID":"266e536b-696a-4d8e-83dd-996685cc3d59","Type":"ContainerStarted","Data":"fe2a5ecc568a59be1c3e2de80202bc5c9f61d5418e4c406cf839f8d996233823"}
Nov 03 18:28:03.158301 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:03.158235    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv"
Nov 03 18:28:03.277160 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:03.277102    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4" podStartSLOduration=1.277086992 podStartE2EDuration="1.277086992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:28:03.275428147 +0000 UTC m=+4224.073140927" watchObservedRunningTime="2026-11-03 18:28:03.277086992 +0000 UTC m=+4224.074799772"
Nov 03 18:28:03.395840 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:03.395784    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv" podStartSLOduration=1.395767163 podStartE2EDuration="1.395767163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:28:03.394464184 +0000 UTC m=+4224.192176965" watchObservedRunningTime="2026-11-03 18:28:03.395767163 +0000 UTC m=+4224.193479997"
Nov 03 18:28:04.599836 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:04.599801    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:04.781733 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:04.781698    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:04.967247 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:04.967215    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:06.860483 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:28:06.860446    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:28:09.562282 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:09.562242    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:14.167278 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:14.167245    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4"
Nov 03 18:28:14.167812 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:14.167400    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv"
Nov 03 18:28:14.456898 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:14.456861    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:19.288526 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:19.288484    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:22.287247 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:22.287212    6059 generic.go:356] "Generic (PLEG): container finished" podID="4efe012c-c87d-4544-8f34-8fd3495e19da" containerID="4345be318e96ec4d7db7d875fd5b987b92b25dbf1c5a3ab5db8c54540e2d84ac" exitCode=0
Nov 03 18:28:22.287707 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:22.287291    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h" event={"ID":"4efe012c-c87d-4544-8f34-8fd3495e19da","Type":"ContainerDied","Data":"4345be318e96ec4d7db7d875fd5b987b92b25dbf1c5a3ab5db8c54540e2d84ac"}
Nov 03 18:28:22.287759 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:22.287726    6059 scope.go:122] "RemoveContainer" containerID="4345be318e96ec4d7db7d875fd5b987b92b25dbf1c5a3ab5db8c54540e2d84ac"
Nov 03 18:28:22.289003 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:22.288972    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:28:22.289094 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:22.289046    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:28:22.289094 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:22.289082    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:28:23.294780 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:23.294746    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h" event={"ID":"4efe012c-c87d-4544-8f34-8fd3495e19da","Type":"ContainerStarted","Data":"6ace34096399fd5160ee80bebbf3416c2751157a87f2b83f675346c9514d3a9e"}
Nov 03 18:28:24.287055 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:24.287013    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:28:24.288238 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:24.288209    6059 patch_prober.go:29] interesting pod/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Nov 03 18:28:24.288385 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:24.288272    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h" podUID="4efe012c-c87d-4544-8f34-8fd3495e19da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Nov 03 18:28:24.453386 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:24.448611    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:25.287191 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:25.287154    6059 patch_prober.go:29] interesting pod/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Nov 03 18:28:25.287413 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:25.287209    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h" podUID="4efe012c-c87d-4544-8f34-8fd3495e19da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Nov 03 18:28:26.287221 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:26.287182    6059 patch_prober.go:29] interesting pod/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Nov 03 18:28:26.287658 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:26.287241    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h" podUID="4efe012c-c87d-4544-8f34-8fd3495e19da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Nov 03 18:28:27.120779 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:27.120741    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:28:27.287237 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:27.287205    6059 patch_prober.go:29] interesting pod/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Nov 03 18:28:27.287723 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:27.287261    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h" podUID="4efe012c-c87d-4544-8f34-8fd3495e19da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Nov 03 18:28:28.287453 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:28.287415    6059 patch_prober.go:29] interesting pod/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Nov 03 18:28:28.287865 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:28.287474    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h" podUID="4efe012c-c87d-4544-8f34-8fd3495e19da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Nov 03 18:28:29.291193 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:29.291160    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:28:29.292182 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:29.292161    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:28:29.690818 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:29.690731    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:34.540310 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:34.540246    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:34.849577 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:34.849496    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr"]
Nov 03 18:28:34.853441 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:34.853422    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr"
Nov 03 18:28:34.913936 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:34.913908    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr"]
Nov 03 18:28:34.948065 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:34.948030    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2ac42c-964b-4610-a6ea-930287aeb637-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-007400fa-hg7lr\" (UID: \"7e2ac42c-964b-4610-a6ea-930287aeb637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr"
Nov 03 18:28:34.948217 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:34.948097    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2ac42c-964b-4610-a6ea-930287aeb637-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-007400fa-hg7lr\" (UID: \"7e2ac42c-964b-4610-a6ea-930287aeb637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr"
Nov 03 18:28:34.948264 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:34.948210    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6r56\" (UniqueName: \"kubernetes.io/projected/7e2ac42c-964b-4610-a6ea-930287aeb637-kube-api-access-m6r56\") pod \"maas-api-revoke-keys-e2e-ait-007400fa-hg7lr\" (UID: \"7e2ac42c-964b-4610-a6ea-930287aeb637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr"
Nov 03 18:28:34.962432 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:34.962414    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:28:34.962551 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:34.962433    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:28:34.962551 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:34.962448    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:28:34.962551 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:34.962512    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:28:35.048795 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:35.048766    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2ac42c-964b-4610-a6ea-930287aeb637-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-007400fa-hg7lr\" (UID: \"7e2ac42c-964b-4610-a6ea-930287aeb637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr"
Nov 03 18:28:35.048966 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:35.048847    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6r56\" (UniqueName: \"kubernetes.io/projected/7e2ac42c-964b-4610-a6ea-930287aeb637-kube-api-access-m6r56\") pod \"maas-api-revoke-keys-e2e-ait-007400fa-hg7lr\" (UID: \"7e2ac42c-964b-4610-a6ea-930287aeb637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr"
Nov 03 18:28:35.048966 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:35.048874    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2ac42c-964b-4610-a6ea-930287aeb637-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-007400fa-hg7lr\" (UID: \"7e2ac42c-964b-4610-a6ea-930287aeb637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr"
Nov 03 18:28:35.049477 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:35.049457    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2ac42c-964b-4610-a6ea-930287aeb637-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-007400fa-hg7lr\" (UID: \"7e2ac42c-964b-4610-a6ea-930287aeb637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr"
Nov 03 18:28:35.051183 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:35.051160    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2ac42c-964b-4610-a6ea-930287aeb637-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-007400fa-hg7lr\" (UID: \"7e2ac42c-964b-4610-a6ea-930287aeb637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr"
Nov 03 18:28:35.112906 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:35.112838    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6r56\" (UniqueName: \"kubernetes.io/projected/7e2ac42c-964b-4610-a6ea-930287aeb637-kube-api-access-m6r56\") pod \"maas-api-revoke-keys-e2e-ait-007400fa-hg7lr\" (UID: \"7e2ac42c-964b-4610-a6ea-930287aeb637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr"
Nov 03 18:28:35.166434 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:35.166406    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr"
Nov 03 18:28:35.535967 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:28:35.535933    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2ac42c_964b_4610_a6ea_930287aeb637.slice/crio-6ae44dfe92d554983a874bff112867ea60424ceaa7421b3abea052a5d3be0f7b WatchSource:0}: Error finding container 6ae44dfe92d554983a874bff112867ea60424ceaa7421b3abea052a5d3be0f7b: Status 404 returned error can't find the container with id 6ae44dfe92d554983a874bff112867ea60424ceaa7421b3abea052a5d3be0f7b
Nov 03 18:28:35.538298 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:35.538271    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr"]
Nov 03 18:28:36.366341 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:36.366305    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr" event={"ID":"7e2ac42c-964b-4610-a6ea-930287aeb637","Type":"ContainerStarted","Data":"c447370a82eb0c6958b8ee1c2d806bb671b2bd3540bcd17ca48678b0ac94ce61"}
Nov 03 18:28:36.366848 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:36.366379    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr" event={"ID":"7e2ac42c-964b-4610-a6ea-930287aeb637","Type":"ContainerStarted","Data":"6ae44dfe92d554983a874bff112867ea60424ceaa7421b3abea052a5d3be0f7b"}
Nov 03 18:28:36.487101 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:36.487049    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr" podStartSLOduration=2.487035137 podStartE2EDuration="2.487035137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:28:36.486009934 +0000 UTC m=+4257.283722716" watchObservedRunningTime="2026-11-03 18:28:36.487035137 +0000 UTC m=+4257.284747922"
Nov 03 18:28:36.861306 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:28:36.861266    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:28:37.372578 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:37.372541    6059 generic.go:356] "Generic (PLEG): container finished" podID="7e2ac42c-964b-4610-a6ea-930287aeb637" containerID="c447370a82eb0c6958b8ee1c2d806bb671b2bd3540bcd17ca48678b0ac94ce61" exitCode=0
Nov 03 18:28:37.372990 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:37.372575    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr" event={"ID":"7e2ac42c-964b-4610-a6ea-930287aeb637","Type":"ContainerDied","Data":"c447370a82eb0c6958b8ee1c2d806bb671b2bd3540bcd17ca48678b0ac94ce61"}
Nov 03 18:28:38.511761 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:38.511736    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr"
Nov 03 18:28:38.577658 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:38.577625    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e2ac42c-964b-4610-a6ea-930287aeb637-kube-api-access-m6r56\" (UniqueName: \"kubernetes.io/projected/7e2ac42c-964b-4610-a6ea-930287aeb637-kube-api-access-m6r56\") pod \"7e2ac42c-964b-4610-a6ea-930287aeb637\" (UID: \"7e2ac42c-964b-4610-a6ea-930287aeb637\") "
Nov 03 18:28:38.577824 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:38.577711    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e2ac42c-964b-4610-a6ea-930287aeb637-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2ac42c-964b-4610-a6ea-930287aeb637-maas-api-openshift-ca\") pod \"7e2ac42c-964b-4610-a6ea-930287aeb637\" (UID: \"7e2ac42c-964b-4610-a6ea-930287aeb637\") "
Nov 03 18:28:38.577824 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:38.577810    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e2ac42c-964b-4610-a6ea-930287aeb637-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2ac42c-964b-4610-a6ea-930287aeb637-maas-api-xks-ca\") pod \"7e2ac42c-964b-4610-a6ea-930287aeb637\" (UID: \"7e2ac42c-964b-4610-a6ea-930287aeb637\") "
Nov 03 18:28:38.578071 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:38.578051    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e2ac42c-964b-4610-a6ea-930287aeb637-maas-api-openshift-ca" pod "7e2ac42c-964b-4610-a6ea-930287aeb637" (UID: "7e2ac42c-964b-4610-a6ea-930287aeb637"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:28:38.578216 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:38.578196    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e2ac42c-964b-4610-a6ea-930287aeb637-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:38.579783 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:38.579761    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2ac42c-964b-4610-a6ea-930287aeb637-kube-api-access-m6r56" pod "7e2ac42c-964b-4610-a6ea-930287aeb637" (UID: "7e2ac42c-964b-4610-a6ea-930287aeb637"). InnerVolumeSpecName "kube-api-access-m6r56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:28:38.579896 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:38.579813    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e2ac42c-964b-4610-a6ea-930287aeb637-maas-api-xks-ca" pod "7e2ac42c-964b-4610-a6ea-930287aeb637" (UID: "7e2ac42c-964b-4610-a6ea-930287aeb637"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:28:38.679173 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:38.679089    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6r56\" (UniqueName: \"kubernetes.io/projected/7e2ac42c-964b-4610-a6ea-930287aeb637-kube-api-access-m6r56\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:38.679173 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:38.679123    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e2ac42c-964b-4610-a6ea-930287aeb637-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:39.355715 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:39.355657    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:39.385491 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:39.385452    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr" event={"ID":"7e2ac42c-964b-4610-a6ea-930287aeb637","Type":"ContainerDied","Data":"6ae44dfe92d554983a874bff112867ea60424ceaa7421b3abea052a5d3be0f7b"}
Nov 03 18:28:39.385652 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:39.385513    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ae44dfe92d554983a874bff112867ea60424ceaa7421b3abea052a5d3be0f7b"
Nov 03 18:28:39.385652 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:39.385475    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr"
Nov 03 18:28:44.166129 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:44.166096    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:46.239038 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:46.239005    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4"]
Nov 03 18:28:46.336371 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:46.336323    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv"]
Nov 03 18:28:46.608836 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:46.608750    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:46.790314 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:46.790274    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:48.868440 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:48.868405    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:51.239787 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:51.239711    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4" podUID="51b31ac2-ad6f-4ed0-9431-7a076bfb664c" containerName="payload-processing" containerID="cri-o://79b75936fddebddcd88eec1b5062655c4e0ee4eb61eb509d5197fae930a85762" gracePeriod=25
Nov 03 18:28:51.337045 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:51.336999    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv" podUID="266e536b-696a-4d8e-83dd-996685cc3d59" containerName="payload-pre-processing" containerID="cri-o://1fca3dd3fc69bac358067edd999c24a78bcc5393ac8d972d2f4e27ef70af41cf" gracePeriod=25
Nov 03 18:28:51.451845 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:51.451812    6059 generic.go:356] "Generic (PLEG): container finished" podID="266e536b-696a-4d8e-83dd-996685cc3d59" containerID="1fca3dd3fc69bac358067edd999c24a78bcc5393ac8d972d2f4e27ef70af41cf" exitCode=0
Nov 03 18:28:51.451963 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:51.451885    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv" event={"ID":"266e536b-696a-4d8e-83dd-996685cc3d59","Type":"ContainerDied","Data":"1fca3dd3fc69bac358067edd999c24a78bcc5393ac8d972d2f4e27ef70af41cf"}
Nov 03 18:28:51.453774 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:51.453748    6059 generic.go:356] "Generic (PLEG): container finished" podID="51b31ac2-ad6f-4ed0-9431-7a076bfb664c" containerID="79b75936fddebddcd88eec1b5062655c4e0ee4eb61eb509d5197fae930a85762" exitCode=0
Nov 03 18:28:51.453879 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:51.453806    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4" event={"ID":"51b31ac2-ad6f-4ed0-9431-7a076bfb664c","Type":"ContainerDied","Data":"79b75936fddebddcd88eec1b5062655c4e0ee4eb61eb509d5197fae930a85762"}
Nov 03 18:28:51.495250 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:51.495186    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4"
Nov 03 18:28:51.579232 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:51.579210    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv"
Nov 03 18:28:51.612122 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:51.612098    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51b31ac2-ad6f-4ed0-9431-7a076bfb664c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b31ac2-ad6f-4ed0-9431-7a076bfb664c-plugins-config-volume\") pod \"51b31ac2-ad6f-4ed0-9431-7a076bfb664c\" (UID: \"51b31ac2-ad6f-4ed0-9431-7a076bfb664c\") "
Nov 03 18:28:51.612257 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:51.612209    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51b31ac2-ad6f-4ed0-9431-7a076bfb664c-kube-api-access-sjwvc\" (UniqueName: \"kubernetes.io/projected/51b31ac2-ad6f-4ed0-9431-7a076bfb664c-kube-api-access-sjwvc\") pod \"51b31ac2-ad6f-4ed0-9431-7a076bfb664c\" (UID: \"51b31ac2-ad6f-4ed0-9431-7a076bfb664c\") "
Nov 03 18:28:51.612513 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:51.612487    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b31ac2-ad6f-4ed0-9431-7a076bfb664c-plugins-config-volume" pod "51b31ac2-ad6f-4ed0-9431-7a076bfb664c" (UID: "51b31ac2-ad6f-4ed0-9431-7a076bfb664c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:28:51.614092 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:51.614072    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b31ac2-ad6f-4ed0-9431-7a076bfb664c-kube-api-access-sjwvc" pod "51b31ac2-ad6f-4ed0-9431-7a076bfb664c" (UID: "51b31ac2-ad6f-4ed0-9431-7a076bfb664c"). InnerVolumeSpecName "kube-api-access-sjwvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:28:51.713162 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:51.713124    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/266e536b-696a-4d8e-83dd-996685cc3d59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266e536b-696a-4d8e-83dd-996685cc3d59-plugins-config-volume\") pod \"266e536b-696a-4d8e-83dd-996685cc3d59\" (UID: \"266e536b-696a-4d8e-83dd-996685cc3d59\") "
Nov 03 18:28:51.713341 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:51.713257    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/266e536b-696a-4d8e-83dd-996685cc3d59-kube-api-access-tdzt6\" (UniqueName: \"kubernetes.io/projected/266e536b-696a-4d8e-83dd-996685cc3d59-kube-api-access-tdzt6\") pod \"266e536b-696a-4d8e-83dd-996685cc3d59\" (UID: \"266e536b-696a-4d8e-83dd-996685cc3d59\") "
Nov 03 18:28:51.713602 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:51.713578    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/266e536b-696a-4d8e-83dd-996685cc3d59-plugins-config-volume" pod "266e536b-696a-4d8e-83dd-996685cc3d59" (UID: "266e536b-696a-4d8e-83dd-996685cc3d59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:28:51.713694 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:51.713680    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b31ac2-ad6f-4ed0-9431-7a076bfb664c-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:51.713734 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:51.713697    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266e536b-696a-4d8e-83dd-996685cc3d59-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:51.713734 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:51.713707    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjwvc\" (UniqueName: \"kubernetes.io/projected/51b31ac2-ad6f-4ed0-9431-7a076bfb664c-kube-api-access-sjwvc\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:51.715257 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:51.715234    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266e536b-696a-4d8e-83dd-996685cc3d59-kube-api-access-tdzt6" pod "266e536b-696a-4d8e-83dd-996685cc3d59" (UID: "266e536b-696a-4d8e-83dd-996685cc3d59"). InnerVolumeSpecName "kube-api-access-tdzt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:28:51.815219 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:51.815134    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdzt6\" (UniqueName: \"kubernetes.io/projected/266e536b-696a-4d8e-83dd-996685cc3d59-kube-api-access-tdzt6\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:52.461142 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:52.461108    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4"
Nov 03 18:28:52.461142 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:52.461135    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4" event={"ID":"51b31ac2-ad6f-4ed0-9431-7a076bfb664c","Type":"ContainerDied","Data":"878bf39e8013c32438f1f1da19d43fa3fff1bdaaf8e47cd85bcf58c89f004364"}
Nov 03 18:28:52.461723 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:52.461195    6059 scope.go:122] "RemoveContainer" containerID="79b75936fddebddcd88eec1b5062655c4e0ee4eb61eb509d5197fae930a85762"
Nov 03 18:28:52.462513 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:52.462463    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv"
Nov 03 18:28:52.462513 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:52.462459    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv" event={"ID":"266e536b-696a-4d8e-83dd-996685cc3d59","Type":"ContainerDied","Data":"fe2a5ecc568a59be1c3e2de80202bc5c9f61d5418e4c406cf839f8d996233823"}
Nov 03 18:28:52.471618 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:52.471598    6059 scope.go:122] "RemoveContainer" containerID="1fca3dd3fc69bac358067edd999c24a78bcc5393ac8d972d2f4e27ef70af41cf"
Nov 03 18:28:52.631125 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:52.631030    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv"]
Nov 03 18:28:52.682806 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:52.682764    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-007400fa-54c7c57b7-n8tdv"]
Nov 03 18:28:52.747998 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:52.747963    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4"]
Nov 03 18:28:52.754172 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:52.754135    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-007400fa-6c47968787-w22f4"]
Nov 03 18:28:53.077890 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:53.077851    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:53.334978 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:53.334896    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="266e536b-696a-4d8e-83dd-996685cc3d59" path="/var/lib/kubelet/pods/266e536b-696a-4d8e-83dd-996685cc3d59/volumes"
Nov 03 18:28:53.335273 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:53.335260    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51b31ac2-ad6f-4ed0-9431-7a076bfb664c" path="/var/lib/kubelet/pods/51b31ac2-ad6f-4ed0-9431-7a076bfb664c/volumes"
Nov 03 18:28:54.632601 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:54.632563    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:54.882763 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:54.882669    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:55.058479 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:55.058440    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:55.089946 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:55.089911    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr"]
Nov 03 18:28:55.141220 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:55.141134    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-007400fa-hg7lr"]
Nov 03 18:28:55.334987 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:55.334953    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e2ac42c-964b-4610-a6ea-930287aeb637" path="/var/lib/kubelet/pods/7e2ac42c-964b-4610-a6ea-930287aeb637/volumes"
Nov 03 18:28:55.935985 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:55.935954    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"]
Nov 03 18:28:55.936476 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:55.936166    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h" podUID="4efe012c-c87d-4544-8f34-8fd3495e19da" containerName="istio-proxy" containerID="cri-o://6ace34096399fd5160ee80bebbf3416c2751157a87f2b83f675346c9514d3a9e" gracePeriod=30
Nov 03 18:28:55.967719 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:55.967687    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:28:59.293390 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:59.293342    6059 patch_prober.go:29] interesting pod/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:28:59.293863 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:28:59.293410    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h" podUID="4efe012c-c87d-4544-8f34-8fd3495e19da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:29:01.184902 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.184874    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:29:01.310451 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.310416    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-credential-socket\") pod \"4efe012c-c87d-4544-8f34-8fd3495e19da\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") "
Nov 03 18:29:01.310644 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.310469    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-envoy\") pod \"4efe012c-c87d-4544-8f34-8fd3495e19da\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") "
Nov 03 18:29:01.310644 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.310518    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-podinfo\") pod \"4efe012c-c87d-4544-8f34-8fd3495e19da\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") "
Nov 03 18:29:01.310644 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.310539    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4efe012c-c87d-4544-8f34-8fd3495e19da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4efe012c-c87d-4544-8f34-8fd3495e19da-istiod-ca-cert\") pod \"4efe012c-c87d-4544-8f34-8fd3495e19da\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") "
Nov 03 18:29:01.310644 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.310560    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4efe012c-c87d-4544-8f34-8fd3495e19da-kube-api-access-vktbn\" (UniqueName: \"kubernetes.io/projected/4efe012c-c87d-4544-8f34-8fd3495e19da-kube-api-access-vktbn\") pod \"4efe012c-c87d-4544-8f34-8fd3495e19da\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") "
Nov 03 18:29:01.310644 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.310577    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-workload-certs\") pod \"4efe012c-c87d-4544-8f34-8fd3495e19da\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") "
Nov 03 18:29:01.310644 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.310605    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-workload-socket\") pod \"4efe012c-c87d-4544-8f34-8fd3495e19da\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") "
Nov 03 18:29:01.310644 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.310631    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-data\") pod \"4efe012c-c87d-4544-8f34-8fd3495e19da\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") "
Nov 03 18:29:01.310977 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.310657    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-token\" (UniqueName: \"kubernetes.io/projected/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-token\") pod \"4efe012c-c87d-4544-8f34-8fd3495e19da\" (UID: \"4efe012c-c87d-4544-8f34-8fd3495e19da\") "
Nov 03 18:29:01.310977 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.310773    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-credential-socket" pod "4efe012c-c87d-4544-8f34-8fd3495e19da" (UID: "4efe012c-c87d-4544-8f34-8fd3495e19da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:29:01.311101 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.310994    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-workload-certs" pod "4efe012c-c87d-4544-8f34-8fd3495e19da" (UID: "4efe012c-c87d-4544-8f34-8fd3495e19da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:29:01.311101 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.311034    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-data" pod "4efe012c-c87d-4544-8f34-8fd3495e19da" (UID: "4efe012c-c87d-4544-8f34-8fd3495e19da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:29:01.311101 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.311092    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:01.311255 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.311115    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:01.311255 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.311170    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-workload-socket" pod "4efe012c-c87d-4544-8f34-8fd3495e19da" (UID: "4efe012c-c87d-4544-8f34-8fd3495e19da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:29:01.311405 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.311277    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4efe012c-c87d-4544-8f34-8fd3495e19da-istiod-ca-cert" pod "4efe012c-c87d-4544-8f34-8fd3495e19da" (UID: "4efe012c-c87d-4544-8f34-8fd3495e19da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:29:01.312865 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.312838    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-podinfo" pod "4efe012c-c87d-4544-8f34-8fd3495e19da" (UID: "4efe012c-c87d-4544-8f34-8fd3495e19da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:29:01.312974 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.312936    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-envoy" pod "4efe012c-c87d-4544-8f34-8fd3495e19da" (UID: "4efe012c-c87d-4544-8f34-8fd3495e19da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:29:01.313212 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.313187    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efe012c-c87d-4544-8f34-8fd3495e19da-kube-api-access-vktbn" pod "4efe012c-c87d-4544-8f34-8fd3495e19da" (UID: "4efe012c-c87d-4544-8f34-8fd3495e19da"). InnerVolumeSpecName "kube-api-access-vktbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:29:01.313278 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.313198    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-token" pod "4efe012c-c87d-4544-8f34-8fd3495e19da" (UID: "4efe012c-c87d-4544-8f34-8fd3495e19da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:29:01.411787 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.411750    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:01.411787 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.411780    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:01.411787 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.411793    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4efe012c-c87d-4544-8f34-8fd3495e19da-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:01.412043 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.411803    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vktbn\" (UniqueName: \"kubernetes.io/projected/4efe012c-c87d-4544-8f34-8fd3495e19da-kube-api-access-vktbn\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:01.412043 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.411812    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:01.412043 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.411820    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:01.412043 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.411829    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4efe012c-c87d-4544-8f34-8fd3495e19da-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:01.515054 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.515020    6059 generic.go:356] "Generic (PLEG): container finished" podID="4efe012c-c87d-4544-8f34-8fd3495e19da" containerID="6ace34096399fd5160ee80bebbf3416c2751157a87f2b83f675346c9514d3a9e" exitCode=0
Nov 03 18:29:01.515242 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.515065    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"
Nov 03 18:29:01.515242 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.515111    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h" event={"ID":"4efe012c-c87d-4544-8f34-8fd3495e19da","Type":"ContainerDied","Data":"6ace34096399fd5160ee80bebbf3416c2751157a87f2b83f675346c9514d3a9e"}
Nov 03 18:29:01.515242 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.515169    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h" event={"ID":"4efe012c-c87d-4544-8f34-8fd3495e19da","Type":"ContainerDied","Data":"5949405c9bfe61e38b8e3c5f2e099b1599d77b67352f8fa02e0ab6897903f0b6"}
Nov 03 18:29:01.515242 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.515191    6059 scope.go:122] "RemoveContainer" containerID="6ace34096399fd5160ee80bebbf3416c2751157a87f2b83f675346c9514d3a9e"
Nov 03 18:29:01.524283 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.524265    6059 scope.go:122] "RemoveContainer" containerID="4345be318e96ec4d7db7d875fd5b987b92b25dbf1c5a3ab5db8c54540e2d84ac"
Nov 03 18:29:01.532183 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.532163    6059 scope.go:122] "RemoveContainer" containerID="6ace34096399fd5160ee80bebbf3416c2751157a87f2b83f675346c9514d3a9e"
Nov 03 18:29:01.532487 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:29:01.532461    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ace34096399fd5160ee80bebbf3416c2751157a87f2b83f675346c9514d3a9e\": container with ID starting with 6ace34096399fd5160ee80bebbf3416c2751157a87f2b83f675346c9514d3a9e not found: ID does not exist" containerID="6ace34096399fd5160ee80bebbf3416c2751157a87f2b83f675346c9514d3a9e"
Nov 03 18:29:01.532550 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.532497    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ace34096399fd5160ee80bebbf3416c2751157a87f2b83f675346c9514d3a9e"} err="failed to get container status \"6ace34096399fd5160ee80bebbf3416c2751157a87f2b83f675346c9514d3a9e\": rpc error: code = NotFound desc = could not find container \"6ace34096399fd5160ee80bebbf3416c2751157a87f2b83f675346c9514d3a9e\": container with ID starting with 6ace34096399fd5160ee80bebbf3416c2751157a87f2b83f675346c9514d3a9e not found: ID does not exist"
Nov 03 18:29:01.532550 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.532518    6059 scope.go:122] "RemoveContainer" containerID="4345be318e96ec4d7db7d875fd5b987b92b25dbf1c5a3ab5db8c54540e2d84ac"
Nov 03 18:29:01.532780 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:29:01.532764    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4345be318e96ec4d7db7d875fd5b987b92b25dbf1c5a3ab5db8c54540e2d84ac\": container with ID starting with 4345be318e96ec4d7db7d875fd5b987b92b25dbf1c5a3ab5db8c54540e2d84ac not found: ID does not exist" containerID="4345be318e96ec4d7db7d875fd5b987b92b25dbf1c5a3ab5db8c54540e2d84ac"
Nov 03 18:29:01.532826 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.532787    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4345be318e96ec4d7db7d875fd5b987b92b25dbf1c5a3ab5db8c54540e2d84ac"} err="failed to get container status \"4345be318e96ec4d7db7d875fd5b987b92b25dbf1c5a3ab5db8c54540e2d84ac\": rpc error: code = NotFound desc = could not find container \"4345be318e96ec4d7db7d875fd5b987b92b25dbf1c5a3ab5db8c54540e2d84ac\": container with ID starting with 4345be318e96ec4d7db7d875fd5b987b92b25dbf1c5a3ab5db8c54540e2d84ac not found: ID does not exist"
Nov 03 18:29:01.694710 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.694613    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"]
Nov 03 18:29:01.730082 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:01.730047    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-007400fa-openshift-default-575fc57557-hqh6h"]
Nov 03 18:29:02.595080 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.595045    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75c4dd589d-k799n"]
Nov 03 18:29:02.595491 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.595455    6059 cpu_manager.go:414] "containerMap: removing container" podUID="4efe012c-c87d-4544-8f34-8fd3495e19da" containerName="istio-proxy"
Nov 03 18:29:02.595491 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.595467    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4efe012c-c87d-4544-8f34-8fd3495e19da" containerName="istio-proxy"
Nov 03 18:29:02.595491 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.595488    6059 cpu_manager.go:414] "containerMap: removing container" podUID="7e2ac42c-964b-4610-a6ea-930287aeb637" containerName="revoke-keys"
Nov 03 18:29:02.595491 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.595493    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e2ac42c-964b-4610-a6ea-930287aeb637" containerName="revoke-keys"
Nov 03 18:29:02.595632 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.595502    6059 cpu_manager.go:414] "containerMap: removing container" podUID="51b31ac2-ad6f-4ed0-9431-7a076bfb664c" containerName="payload-processing"
Nov 03 18:29:02.595632 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.595507    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51b31ac2-ad6f-4ed0-9431-7a076bfb664c" containerName="payload-processing"
Nov 03 18:29:02.595632 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.595531    6059 cpu_manager.go:414] "containerMap: removing container" podUID="266e536b-696a-4d8e-83dd-996685cc3d59" containerName="payload-pre-processing"
Nov 03 18:29:02.595632 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.595536    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="266e536b-696a-4d8e-83dd-996685cc3d59" containerName="payload-pre-processing"
Nov 03 18:29:02.595632 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.595589    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4efe012c-c87d-4544-8f34-8fd3495e19da" containerName="istio-proxy"
Nov 03 18:29:02.595632 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.595596    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="266e536b-696a-4d8e-83dd-996685cc3d59" containerName="payload-pre-processing"
Nov 03 18:29:02.595632 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.595604    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4efe012c-c87d-4544-8f34-8fd3495e19da" containerName="istio-proxy"
Nov 03 18:29:02.595632 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.595612    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e2ac42c-964b-4610-a6ea-930287aeb637" containerName="revoke-keys"
Nov 03 18:29:02.595632 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.595618    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="51b31ac2-ad6f-4ed0-9431-7a076bfb664c" containerName="payload-processing"
Nov 03 18:29:02.600435 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.600417    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c4dd589d-k799n"
Nov 03 18:29:02.655004 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.654967    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 18:29:02.706885 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.706841    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 18:29:02.706885 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.706878    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hl2hg\""
Nov 03 18:29:02.707140 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.706911    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 18:29:02.709390 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.709366    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75c4dd589d-k799n"]
Nov 03 18:29:02.722448 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.722426    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f98b3b13-0a3e-4d91-894b-9d2b03f5110c-webhook-cert\") pod \"maas-controller-75c4dd589d-k799n\" (UID: \"f98b3b13-0a3e-4d91-894b-9d2b03f5110c\") " pod="opendatahub/maas-controller-75c4dd589d-k799n"
Nov 03 18:29:02.722558 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.722534    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr6km\" (UniqueName: \"kubernetes.io/projected/f98b3b13-0a3e-4d91-894b-9d2b03f5110c-kube-api-access-gr6km\") pod \"maas-controller-75c4dd589d-k799n\" (UID: \"f98b3b13-0a3e-4d91-894b-9d2b03f5110c\") " pod="opendatahub/maas-controller-75c4dd589d-k799n"
Nov 03 18:29:02.722614 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.722572    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f98b3b13-0a3e-4d91-894b-9d2b03f5110c-metrics-certs\") pod \"maas-controller-75c4dd589d-k799n\" (UID: \"f98b3b13-0a3e-4d91-894b-9d2b03f5110c\") " pod="opendatahub/maas-controller-75c4dd589d-k799n"
Nov 03 18:29:02.754623 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.754587    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:29:02.823287 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.823248    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f98b3b13-0a3e-4d91-894b-9d2b03f5110c-webhook-cert\") pod \"maas-controller-75c4dd589d-k799n\" (UID: \"f98b3b13-0a3e-4d91-894b-9d2b03f5110c\") " pod="opendatahub/maas-controller-75c4dd589d-k799n"
Nov 03 18:29:02.823503 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.823377    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr6km\" (UniqueName: \"kubernetes.io/projected/f98b3b13-0a3e-4d91-894b-9d2b03f5110c-kube-api-access-gr6km\") pod \"maas-controller-75c4dd589d-k799n\" (UID: \"f98b3b13-0a3e-4d91-894b-9d2b03f5110c\") " pod="opendatahub/maas-controller-75c4dd589d-k799n"
Nov 03 18:29:02.823503 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.823431    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f98b3b13-0a3e-4d91-894b-9d2b03f5110c-metrics-certs\") pod \"maas-controller-75c4dd589d-k799n\" (UID: \"f98b3b13-0a3e-4d91-894b-9d2b03f5110c\") " pod="opendatahub/maas-controller-75c4dd589d-k799n"
Nov 03 18:29:02.825772 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.825751    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f98b3b13-0a3e-4d91-894b-9d2b03f5110c-webhook-cert\") pod \"maas-controller-75c4dd589d-k799n\" (UID: \"f98b3b13-0a3e-4d91-894b-9d2b03f5110c\") " pod="opendatahub/maas-controller-75c4dd589d-k799n"
Nov 03 18:29:02.825864 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.825829    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f98b3b13-0a3e-4d91-894b-9d2b03f5110c-metrics-certs\") pod \"maas-controller-75c4dd589d-k799n\" (UID: \"f98b3b13-0a3e-4d91-894b-9d2b03f5110c\") " pod="opendatahub/maas-controller-75c4dd589d-k799n"
Nov 03 18:29:02.886817 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.886740    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr6km\" (UniqueName: \"kubernetes.io/projected/f98b3b13-0a3e-4d91-894b-9d2b03f5110c-kube-api-access-gr6km\") pod \"maas-controller-75c4dd589d-k799n\" (UID: \"f98b3b13-0a3e-4d91-894b-9d2b03f5110c\") " pod="opendatahub/maas-controller-75c4dd589d-k799n"
Nov 03 18:29:02.913710 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:02.913673    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c4dd589d-k799n"
Nov 03 18:29:03.287326 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:29:03.287290    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98b3b13_0a3e_4d91_894b_9d2b03f5110c.slice/crio-7f8d725882c0ca677c7a1a5de5372cf41eb5193edc5d98f80ca59a7614793fb9 WatchSource:0}: Error finding container 7f8d725882c0ca677c7a1a5de5372cf41eb5193edc5d98f80ca59a7614793fb9: Status 404 returned error can't find the container with id 7f8d725882c0ca677c7a1a5de5372cf41eb5193edc5d98f80ca59a7614793fb9
Nov 03 18:29:03.288594 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:03.288572    6059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:29:03.289082 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:03.289056    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75c4dd589d-k799n"]
Nov 03 18:29:03.334233 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:03.334201    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4efe012c-c87d-4544-8f34-8fd3495e19da" path="/var/lib/kubelet/pods/4efe012c-c87d-4544-8f34-8fd3495e19da/volumes"
Nov 03 18:29:03.528667 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:03.528633    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c4dd589d-k799n" event={"ID":"f98b3b13-0a3e-4d91-894b-9d2b03f5110c","Type":"ContainerStarted","Data":"7f8d725882c0ca677c7a1a5de5372cf41eb5193edc5d98f80ca59a7614793fb9"}
Nov 03 18:29:06.553402 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:06.553347    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c4dd589d-k799n" event={"ID":"f98b3b13-0a3e-4d91-894b-9d2b03f5110c","Type":"ContainerStarted","Data":"56d3086411e51e9ce4799cc2018084b063fe6c6f9f3522898300efb2e1b85bf8"}
Nov 03 18:29:06.553925 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:06.553548    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75c4dd589d-k799n"
Nov 03 18:29:06.689640 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:06.689587    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-75c4dd589d-k799n" podStartSLOduration=2.367589692 podStartE2EDuration="4.68957267s" totalImagesPullingTime="2.321982978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:29:06.688822391 +0000 UTC m=+4287.486535174" watchObservedRunningTime="2026-11-03 18:29:06.68957267 +0000 UTC m=+4287.487285486"
Nov 03 18:29:06.861189 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:29:06.861099    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:29:17.561046 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:17.561007    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75c4dd589d-k799n"
Nov 03 18:29:23.916804 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:23.916764    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-klgk6"]
Nov 03 18:29:23.917324 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:23.917183    6059 cpu_manager.go:414] "containerMap: removing container" podUID="4efe012c-c87d-4544-8f34-8fd3495e19da" containerName="istio-proxy"
Nov 03 18:29:23.917324 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:23.917195    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4efe012c-c87d-4544-8f34-8fd3495e19da" containerName="istio-proxy"
Nov 03 18:29:23.920416 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:23.920399    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-klgk6"
Nov 03 18:29:24.026286 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:24.026251    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:29:24.026550 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:24.026307    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:29:24.026550 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:24.026342    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:29:24.026746 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:24.026432    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:29:24.026746 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:24.026681    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk6zp\" (UniqueName: \"kubernetes.io/projected/debab382-efb3-462d-a4f5-890c3410f5f4-kube-api-access-xk6zp\") pod \"maas-api-revoke-sub-85237e45-klgk6\" (UID: \"debab382-efb3-462d-a4f5-890c3410f5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-klgk6"
Nov 03 18:29:24.026884 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:24.026865    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/debab382-efb3-462d-a4f5-890c3410f5f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-klgk6\" (UID: \"debab382-efb3-462d-a4f5-890c3410f5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-klgk6"
Nov 03 18:29:24.026947 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:24.026902    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/debab382-efb3-462d-a4f5-890c3410f5f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-klgk6\" (UID: \"debab382-efb3-462d-a4f5-890c3410f5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-klgk6"
Nov 03 18:29:24.028524 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:24.028507    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-klgk6"]
Nov 03 18:29:24.127782 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:24.127748    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/debab382-efb3-462d-a4f5-890c3410f5f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-klgk6\" (UID: \"debab382-efb3-462d-a4f5-890c3410f5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-klgk6"
Nov 03 18:29:24.127782 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:24.127789    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/debab382-efb3-462d-a4f5-890c3410f5f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-klgk6\" (UID: \"debab382-efb3-462d-a4f5-890c3410f5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-klgk6"
Nov 03 18:29:24.128036 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:24.127846    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk6zp\" (UniqueName: \"kubernetes.io/projected/debab382-efb3-462d-a4f5-890c3410f5f4-kube-api-access-xk6zp\") pod \"maas-api-revoke-sub-85237e45-klgk6\" (UID: \"debab382-efb3-462d-a4f5-890c3410f5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-klgk6"
Nov 03 18:29:24.128590 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:24.128568    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/debab382-efb3-462d-a4f5-890c3410f5f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-klgk6\" (UID: \"debab382-efb3-462d-a4f5-890c3410f5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-klgk6"
Nov 03 18:29:24.130172 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:24.130149    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/debab382-efb3-462d-a4f5-890c3410f5f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-klgk6\" (UID: \"debab382-efb3-462d-a4f5-890c3410f5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-klgk6"
Nov 03 18:29:24.191150 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:24.191079    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk6zp\" (UniqueName: \"kubernetes.io/projected/debab382-efb3-462d-a4f5-890c3410f5f4-kube-api-access-xk6zp\") pod \"maas-api-revoke-sub-85237e45-klgk6\" (UID: \"debab382-efb3-462d-a4f5-890c3410f5f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-klgk6"
Nov 03 18:29:24.232993 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:24.232965    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-klgk6"
Nov 03 18:29:24.617893 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:29:24.617866    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddebab382_efb3_462d_a4f5_890c3410f5f4.slice/crio-1735fc5184f3986c3274804bb8a249a0ed9650d22786978bd5cac5287ba10550 WatchSource:0}: Error finding container 1735fc5184f3986c3274804bb8a249a0ed9650d22786978bd5cac5287ba10550: Status 404 returned error can't find the container with id 1735fc5184f3986c3274804bb8a249a0ed9650d22786978bd5cac5287ba10550
Nov 03 18:29:24.618974 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:24.618951    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-klgk6"]
Nov 03 18:29:24.655703 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:24.655677    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-klgk6" event={"ID":"debab382-efb3-462d-a4f5-890c3410f5f4","Type":"ContainerStarted","Data":"1735fc5184f3986c3274804bb8a249a0ed9650d22786978bd5cac5287ba10550"}
Nov 03 18:29:25.663715 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:25.663682    6059 generic.go:356] "Generic (PLEG): container finished" podID="debab382-efb3-462d-a4f5-890c3410f5f4" containerID="fd068410fbc49fd38e09b09fcf13c5f21bcf99a2c1b9c11412dd2bd70dd50bcd" exitCode=0
Nov 03 18:29:25.664127 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:25.663742    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-klgk6" event={"ID":"debab382-efb3-462d-a4f5-890c3410f5f4","Type":"ContainerDied","Data":"fd068410fbc49fd38e09b09fcf13c5f21bcf99a2c1b9c11412dd2bd70dd50bcd"}
Nov 03 18:29:26.798988 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:26.798963    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-klgk6"
Nov 03 18:29:26.853461 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:26.853424    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/debab382-efb3-462d-a4f5-890c3410f5f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/debab382-efb3-462d-a4f5-890c3410f5f4-maas-api-openshift-ca\") pod \"debab382-efb3-462d-a4f5-890c3410f5f4\" (UID: \"debab382-efb3-462d-a4f5-890c3410f5f4\") "
Nov 03 18:29:26.853648 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:26.853573    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/debab382-efb3-462d-a4f5-890c3410f5f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/debab382-efb3-462d-a4f5-890c3410f5f4-maas-api-xks-ca\") pod \"debab382-efb3-462d-a4f5-890c3410f5f4\" (UID: \"debab382-efb3-462d-a4f5-890c3410f5f4\") "
Nov 03 18:29:26.853648 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:26.853625    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/debab382-efb3-462d-a4f5-890c3410f5f4-kube-api-access-xk6zp\" (UniqueName: \"kubernetes.io/projected/debab382-efb3-462d-a4f5-890c3410f5f4-kube-api-access-xk6zp\") pod \"debab382-efb3-462d-a4f5-890c3410f5f4\" (UID: \"debab382-efb3-462d-a4f5-890c3410f5f4\") "
Nov 03 18:29:26.853838 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:26.853813    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/debab382-efb3-462d-a4f5-890c3410f5f4-maas-api-openshift-ca" pod "debab382-efb3-462d-a4f5-890c3410f5f4" (UID: "debab382-efb3-462d-a4f5-890c3410f5f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:29:26.853985 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:26.853969    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/debab382-efb3-462d-a4f5-890c3410f5f4-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:26.855577 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:26.855548    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/debab382-efb3-462d-a4f5-890c3410f5f4-maas-api-xks-ca" pod "debab382-efb3-462d-a4f5-890c3410f5f4" (UID: "debab382-efb3-462d-a4f5-890c3410f5f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:29:26.855691 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:26.855667    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/debab382-efb3-462d-a4f5-890c3410f5f4-kube-api-access-xk6zp" pod "debab382-efb3-462d-a4f5-890c3410f5f4" (UID: "debab382-efb3-462d-a4f5-890c3410f5f4"). InnerVolumeSpecName "kube-api-access-xk6zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:29:26.954876 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:26.954801    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/debab382-efb3-462d-a4f5-890c3410f5f4-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:26.954876 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:26.954827    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk6zp\" (UniqueName: \"kubernetes.io/projected/debab382-efb3-462d-a4f5-890c3410f5f4-kube-api-access-xk6zp\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:27.675892 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:27.675817    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-klgk6"
Nov 03 18:29:27.675892 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:27.675854    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-klgk6" event={"ID":"debab382-efb3-462d-a4f5-890c3410f5f4","Type":"ContainerDied","Data":"1735fc5184f3986c3274804bb8a249a0ed9650d22786978bd5cac5287ba10550"}
Nov 03 18:29:27.676078 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:27.675902    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1735fc5184f3986c3274804bb8a249a0ed9650d22786978bd5cac5287ba10550"
Nov 03 18:29:35.762541 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.762504    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"]
Nov 03 18:29:35.763133 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.763112    6059 cpu_manager.go:414] "containerMap: removing container" podUID="debab382-efb3-462d-a4f5-890c3410f5f4" containerName="revoke-keys"
Nov 03 18:29:35.763219 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.763136    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="debab382-efb3-462d-a4f5-890c3410f5f4" containerName="revoke-keys"
Nov 03 18:29:35.763278 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.763245    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="debab382-efb3-462d-a4f5-890c3410f5f4" containerName="revoke-keys"
Nov 03 18:29:35.766813 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.766791    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.815111 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.815080    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"]
Nov 03 18:29:35.818995 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.818976    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:35.831962 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.831944    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-t4h2g\""
Nov 03 18:29:35.841918 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.841895    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.842016 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.841928    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.842016 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.841964    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.842108 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.842050    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.842108 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.842085    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.842186 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.842108    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.842231 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.842213    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.842270 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.842239    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcj2t\" (UniqueName: \"kubernetes.io/projected/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-kube-api-access-bcj2t\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.842331 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.842311    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.875861 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.875824    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"]
Nov 03 18:29:35.927039 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.927005    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"]
Nov 03 18:29:35.942856 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.942826    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.943023 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.942871    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:35.943023 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.942910    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:35.943023 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.942944    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.943175 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.943022    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcj2t\" (UniqueName: \"kubernetes.io/projected/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-kube-api-access-bcj2t\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.943175 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.943115    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:35.943175 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.943156    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:35.943337 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.943185    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.943337 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.943231    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:35.943492 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.943389    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.943492 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.943390    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.943614 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.943498    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.943614 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.943532    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:35.943614 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.943598    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.943810 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.943640    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:35.943810 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.943672    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:35.943810 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.943704    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.943810 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.943758    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s4jj\" (UniqueName: \"kubernetes.io/projected/d9e3abf7-c5e1-4414-858e-5f84f6a88670-kube-api-access-8s4jj\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:35.944030 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.943934    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.944030 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.943973    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.944109 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.944044    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.944154 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.944143    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.944304 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.944283    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.946025 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.946004    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:35.946263 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:35.946245    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:36.007054 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.007008    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:36.007383 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.007327    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcj2t\" (UniqueName: \"kubernetes.io/projected/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-kube-api-access-bcj2t\") pod \"maas-default-gateway-openshift-default-58c6585447-zp9n7\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:36.045540 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.045468    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:36.045540 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.045506    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s4jj\" (UniqueName: \"kubernetes.io/projected/d9e3abf7-c5e1-4414-858e-5f84f6a88670-kube-api-access-8s4jj\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:36.045540 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.045536    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:36.045788 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.045559    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:36.045788 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.045608    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:36.045788 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.045630    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:36.045788 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.045654    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:36.045788 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.045712    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:36.045788 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.045765    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:36.046111 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.046085    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:36.046182 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.046153    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:36.046249 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.046222    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:36.046304 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.046282    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:36.046458 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.046441    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:36.047927 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.047904    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:36.048067 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.048046    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:36.080160 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.080135    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:36.108801 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.108774    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:36.109006 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.108987    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s4jj\" (UniqueName: \"kubernetes.io/projected/d9e3abf7-c5e1-4414-858e-5f84f6a88670-kube-api-access-8s4jj\") pod \"maas-default-gateway-openshift-default-58c6585447-xpshn\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:36.132874 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.132570    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:36.307684 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.307598    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:29:36.388067 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.388028    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75c4dd589d-k799n"]
Nov 03 18:29:36.388260 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.388229    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-75c4dd589d-k799n" podUID="f98b3b13-0a3e-4d91-894b-9d2b03f5110c" containerName="manager" containerID="cri-o://56d3086411e51e9ce4799cc2018084b063fe6c6f9f3522898300efb2e1b85bf8" gracePeriod=30
Nov 03 18:29:36.553808 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.553770    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"]
Nov 03 18:29:36.554193 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.554168    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"]
Nov 03 18:29:36.569555 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:29:36.569488    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e3abf7_c5e1_4414_858e_5f84f6a88670.slice/crio-ff090bfc8d193186df986f99e3606860050f4f323eb5475de0b1ece095f02b29 WatchSource:0}: Error finding container ff090bfc8d193186df986f99e3606860050f4f323eb5475de0b1ece095f02b29: Status 404 returned error can't find the container with id ff090bfc8d193186df986f99e3606860050f4f323eb5475de0b1ece095f02b29
Nov 03 18:29:36.571337 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.571281    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:29:36.571475 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.571328    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:29:36.571475 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.571394    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:29:36.571475 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.571440    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:29:36.571718 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.571438    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:29:36.571811 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.571785    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:29:36.635061 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.635029    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c4dd589d-k799n"
Nov 03 18:29:36.726087 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.726047    6059 generic.go:356] "Generic (PLEG): container finished" podID="f98b3b13-0a3e-4d91-894b-9d2b03f5110c" containerID="56d3086411e51e9ce4799cc2018084b063fe6c6f9f3522898300efb2e1b85bf8" exitCode=0
Nov 03 18:29:36.726292 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.726146    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c4dd589d-k799n"
Nov 03 18:29:36.726292 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.726175    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c4dd589d-k799n" event={"ID":"f98b3b13-0a3e-4d91-894b-9d2b03f5110c","Type":"ContainerDied","Data":"56d3086411e51e9ce4799cc2018084b063fe6c6f9f3522898300efb2e1b85bf8"}
Nov 03 18:29:36.726292 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.726237    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c4dd589d-k799n" event={"ID":"f98b3b13-0a3e-4d91-894b-9d2b03f5110c","Type":"ContainerDied","Data":"7f8d725882c0ca677c7a1a5de5372cf41eb5193edc5d98f80ca59a7614793fb9"}
Nov 03 18:29:36.726292 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.726266    6059 scope.go:122] "RemoveContainer" containerID="56d3086411e51e9ce4799cc2018084b063fe6c6f9f3522898300efb2e1b85bf8"
Nov 03 18:29:36.728486 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.728450    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7" event={"ID":"60b66cb6-2bce-4c02-aa7d-7eb25f29f237","Type":"ContainerStarted","Data":"1fe6f7c31c90141bd1c0146db77a5e6a28c3f16968b2cd1cf074798481f20a93"}
Nov 03 18:29:36.728601 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.728499    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7" event={"ID":"60b66cb6-2bce-4c02-aa7d-7eb25f29f237","Type":"ContainerStarted","Data":"76282666b7f0611700c96cebfff43395bcfc0cd5e4b75558854a4fc8b41db81d"}
Nov 03 18:29:36.730971 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.730947    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn" event={"ID":"d9e3abf7-c5e1-4414-858e-5f84f6a88670","Type":"ContainerStarted","Data":"0366d3184102a5202ba42d897e05e9449e49c80f0ee47f0dc5642c67db4b9488"}
Nov 03 18:29:36.731072 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.730994    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn" event={"ID":"d9e3abf7-c5e1-4414-858e-5f84f6a88670","Type":"ContainerStarted","Data":"ff090bfc8d193186df986f99e3606860050f4f323eb5475de0b1ece095f02b29"}
Nov 03 18:29:36.738437 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.738414    6059 scope.go:122] "RemoveContainer" containerID="56d3086411e51e9ce4799cc2018084b063fe6c6f9f3522898300efb2e1b85bf8"
Nov 03 18:29:36.738750 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:29:36.738726    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56d3086411e51e9ce4799cc2018084b063fe6c6f9f3522898300efb2e1b85bf8\": container with ID starting with 56d3086411e51e9ce4799cc2018084b063fe6c6f9f3522898300efb2e1b85bf8 not found: ID does not exist" containerID="56d3086411e51e9ce4799cc2018084b063fe6c6f9f3522898300efb2e1b85bf8"
Nov 03 18:29:36.738820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.738763    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56d3086411e51e9ce4799cc2018084b063fe6c6f9f3522898300efb2e1b85bf8"} err="failed to get container status \"56d3086411e51e9ce4799cc2018084b063fe6c6f9f3522898300efb2e1b85bf8\": rpc error: code = NotFound desc = could not find container \"56d3086411e51e9ce4799cc2018084b063fe6c6f9f3522898300efb2e1b85bf8\": container with ID starting with 56d3086411e51e9ce4799cc2018084b063fe6c6f9f3522898300efb2e1b85bf8 not found: ID does not exist"
Nov 03 18:29:36.757039 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.757008    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f98b3b13-0a3e-4d91-894b-9d2b03f5110c-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f98b3b13-0a3e-4d91-894b-9d2b03f5110c-webhook-cert\") pod \"f98b3b13-0a3e-4d91-894b-9d2b03f5110c\" (UID: \"f98b3b13-0a3e-4d91-894b-9d2b03f5110c\") "
Nov 03 18:29:36.757169 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.757094    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98b3b13-0a3e-4d91-894b-9d2b03f5110c-kube-api-access-gr6km\" (UniqueName: \"kubernetes.io/projected/f98b3b13-0a3e-4d91-894b-9d2b03f5110c-kube-api-access-gr6km\") pod \"f98b3b13-0a3e-4d91-894b-9d2b03f5110c\" (UID: \"f98b3b13-0a3e-4d91-894b-9d2b03f5110c\") "
Nov 03 18:29:36.757275 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.757257    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f98b3b13-0a3e-4d91-894b-9d2b03f5110c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f98b3b13-0a3e-4d91-894b-9d2b03f5110c-metrics-certs\") pod \"f98b3b13-0a3e-4d91-894b-9d2b03f5110c\" (UID: \"f98b3b13-0a3e-4d91-894b-9d2b03f5110c\") "
Nov 03 18:29:36.760502 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.760472    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f98b3b13-0a3e-4d91-894b-9d2b03f5110c-metrics-certs" pod "f98b3b13-0a3e-4d91-894b-9d2b03f5110c" (UID: "f98b3b13-0a3e-4d91-894b-9d2b03f5110c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:29:36.760614 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.760510    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f98b3b13-0a3e-4d91-894b-9d2b03f5110c-webhook-cert" pod "f98b3b13-0a3e-4d91-894b-9d2b03f5110c" (UID: "f98b3b13-0a3e-4d91-894b-9d2b03f5110c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:29:36.760614 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.760533    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98b3b13-0a3e-4d91-894b-9d2b03f5110c-kube-api-access-gr6km" pod "f98b3b13-0a3e-4d91-894b-9d2b03f5110c" (UID: "f98b3b13-0a3e-4d91-894b-9d2b03f5110c"). InnerVolumeSpecName "kube-api-access-gr6km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:29:36.859092 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.858989    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr6km\" (UniqueName: \"kubernetes.io/projected/f98b3b13-0a3e-4d91-894b-9d2b03f5110c-kube-api-access-gr6km\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:36.859092 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.859022    6059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f98b3b13-0a3e-4d91-894b-9d2b03f5110c-metrics-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:36.859092 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.859037    6059 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f98b3b13-0a3e-4d91-894b-9d2b03f5110c-webhook-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:36.862383 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:29:36.862322    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:29:36.943085 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:36.943024    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7" podStartSLOduration=1.9430072090000001 podStartE2EDuration="1.943007209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:29:36.941854624 +0000 UTC m=+4317.739567434" watchObservedRunningTime="2026-11-03 18:29:36.943007209 +0000 UTC m=+4317.740719990"
Nov 03 18:29:37.071375 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:37.069431    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn" podStartSLOduration=2.06941065 podStartE2EDuration="2.06941065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:29:37.068287184 +0000 UTC m=+4317.865999969" watchObservedRunningTime="2026-11-03 18:29:37.06941065 +0000 UTC m=+4317.867123428"
Nov 03 18:29:37.082781 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:37.082741    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:37.086380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:37.084486    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zp9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Nov 03 18:29:37.086380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:37.084550    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7" podUID="60b66cb6-2bce-4c02-aa7d-7eb25f29f237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Nov 03 18:29:37.139536 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:37.133718    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:37.139536 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:37.137014    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xpshn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 03 18:29:37.139536 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:37.137069    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn" podUID="d9e3abf7-c5e1-4414-858e-5f84f6a88670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 03 18:29:37.184794 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:37.184757    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75c4dd589d-k799n"]
Nov 03 18:29:37.190588 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:37.190562    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75c4dd589d-k799n"]
Nov 03 18:29:37.384379 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:37.384052    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f98b3b13-0a3e-4d91-894b-9d2b03f5110c" path="/var/lib/kubelet/pods/f98b3b13-0a3e-4d91-894b-9d2b03f5110c/volumes"
Nov 03 18:29:38.082569 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.082170    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zp9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Nov 03 18:29:38.082569 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.082239    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7" podUID="60b66cb6-2bce-4c02-aa7d-7eb25f29f237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Nov 03 18:29:38.143930 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.143892    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xpshn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 03 18:29:38.144214 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.144187    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn" podUID="d9e3abf7-c5e1-4414-858e-5f84f6a88670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 03 18:29:38.498708 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.498671    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:29:38.616681 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.616647    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75c4dd589d-nztw8"]
Nov 03 18:29:38.617119 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.617106    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f98b3b13-0a3e-4d91-894b-9d2b03f5110c" containerName="manager"
Nov 03 18:29:38.617178 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.617120    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98b3b13-0a3e-4d91-894b-9d2b03f5110c" containerName="manager"
Nov 03 18:29:38.617220 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.617188    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f98b3b13-0a3e-4d91-894b-9d2b03f5110c" containerName="manager"
Nov 03 18:29:38.622061 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.622043    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c4dd589d-nztw8"
Nov 03 18:29:38.679996 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.679961    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 18:29:38.753845 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.753733    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75c4dd589d-nztw8"]
Nov 03 18:29:38.754327 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.754307    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 18:29:38.754613 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.754600    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 18:29:38.754832 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.754820    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hl2hg\""
Nov 03 18:29:38.789450 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.789411    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a-metrics-certs\") pod \"maas-controller-75c4dd589d-nztw8\" (UID: \"6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a\") " pod="opendatahub/maas-controller-75c4dd589d-nztw8"
Nov 03 18:29:38.789621 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.789458    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flrjz\" (UniqueName: \"kubernetes.io/projected/6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a-kube-api-access-flrjz\") pod \"maas-controller-75c4dd589d-nztw8\" (UID: \"6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a\") " pod="opendatahub/maas-controller-75c4dd589d-nztw8"
Nov 03 18:29:38.789621 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.789490    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a-webhook-cert\") pod \"maas-controller-75c4dd589d-nztw8\" (UID: \"6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a\") " pod="opendatahub/maas-controller-75c4dd589d-nztw8"
Nov 03 18:29:38.891053 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.891011    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a-metrics-certs\") pod \"maas-controller-75c4dd589d-nztw8\" (UID: \"6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a\") " pod="opendatahub/maas-controller-75c4dd589d-nztw8"
Nov 03 18:29:38.891245 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.891078    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flrjz\" (UniqueName: \"kubernetes.io/projected/6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a-kube-api-access-flrjz\") pod \"maas-controller-75c4dd589d-nztw8\" (UID: \"6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a\") " pod="opendatahub/maas-controller-75c4dd589d-nztw8"
Nov 03 18:29:38.891245 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.891125    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a-webhook-cert\") pod \"maas-controller-75c4dd589d-nztw8\" (UID: \"6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a\") " pod="opendatahub/maas-controller-75c4dd589d-nztw8"
Nov 03 18:29:38.893912 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.893888    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a-webhook-cert\") pod \"maas-controller-75c4dd589d-nztw8\" (UID: \"6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a\") " pod="opendatahub/maas-controller-75c4dd589d-nztw8"
Nov 03 18:29:38.894552 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.894521    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a-metrics-certs\") pod \"maas-controller-75c4dd589d-nztw8\" (UID: \"6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a\") " pod="opendatahub/maas-controller-75c4dd589d-nztw8"
Nov 03 18:29:38.958380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:38.958063    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flrjz\" (UniqueName: \"kubernetes.io/projected/6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a-kube-api-access-flrjz\") pod \"maas-controller-75c4dd589d-nztw8\" (UID: \"6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a\") " pod="opendatahub/maas-controller-75c4dd589d-nztw8"
Nov 03 18:29:39.080889 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:39.080785    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zp9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Nov 03 18:29:39.081190 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:39.081145    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7" podUID="60b66cb6-2bce-4c02-aa7d-7eb25f29f237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Nov 03 18:29:39.150384 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:39.150325    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xpshn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 03 18:29:39.150981 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:39.150949    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn" podUID="d9e3abf7-c5e1-4414-858e-5f84f6a88670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 03 18:29:39.243594 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:39.243557    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c4dd589d-nztw8"
Nov 03 18:29:39.709234 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:29:39.709204    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc700c5_cf97_4b4a_a43e_0dc69cf5b21a.slice/crio-2a577b116f19d0ee2f2428b1620f71a0269a210ee861a485f0236cb44f851d85 WatchSource:0}: Error finding container 2a577b116f19d0ee2f2428b1620f71a0269a210ee861a485f0236cb44f851d85: Status 404 returned error can't find the container with id 2a577b116f19d0ee2f2428b1620f71a0269a210ee861a485f0236cb44f851d85
Nov 03 18:29:39.711287 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:39.711263    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75c4dd589d-nztw8"]
Nov 03 18:29:39.796405 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:39.796370    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c4dd589d-nztw8" event={"ID":"6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a","Type":"ContainerStarted","Data":"2a577b116f19d0ee2f2428b1620f71a0269a210ee861a485f0236cb44f851d85"}
Nov 03 18:29:40.080924 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:40.080881    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zp9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Nov 03 18:29:40.081249 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:40.081209    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7" podUID="60b66cb6-2bce-4c02-aa7d-7eb25f29f237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Nov 03 18:29:40.137382 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:40.133768    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xpshn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 03 18:29:40.137382 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:40.133832    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn" podUID="d9e3abf7-c5e1-4414-858e-5f84f6a88670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 03 18:29:40.711891 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:40.711857    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 18:29:41.081159 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:41.081111    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zp9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Nov 03 18:29:41.081389 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:41.081178    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7" podUID="60b66cb6-2bce-4c02-aa7d-7eb25f29f237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Nov 03 18:29:41.142980 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:41.133807    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xpshn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 03 18:29:41.142980 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:41.133879    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn" podUID="d9e3abf7-c5e1-4414-858e-5f84f6a88670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 03 18:29:41.811583 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:41.811538    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c4dd589d-nztw8" event={"ID":"6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a","Type":"ContainerStarted","Data":"ba3cf997a64e5dcad0981a2a00810f0a4cbba565937b030604de2db2f5d1e122"}
Nov 03 18:29:41.812001 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:41.811637    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75c4dd589d-nztw8"
Nov 03 18:29:41.949852 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:41.949788    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-75c4dd589d-nztw8" podStartSLOduration=3.032042798 podStartE2EDuration="3.949766814s" totalImagesPullingTime="917.724016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:29:41.949001573 +0000 UTC m=+4322.746714400" watchObservedRunningTime="2026-11-03 18:29:41.949766814 +0000 UTC m=+4322.747479595"
Nov 03 18:29:42.081910 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:42.081810    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zp9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Nov 03 18:29:42.082195 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:42.082162    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7" podUID="60b66cb6-2bce-4c02-aa7d-7eb25f29f237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Nov 03 18:29:42.133821 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:42.133781    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xpshn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 03 18:29:42.134180 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:42.134111    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn" podUID="d9e3abf7-c5e1-4414-858e-5f84f6a88670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 03 18:29:43.080971 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:43.080925    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zp9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Nov 03 18:29:43.081391 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:43.080995    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7" podUID="60b66cb6-2bce-4c02-aa7d-7eb25f29f237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Nov 03 18:29:43.133092 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:43.133056    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xpshn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 03 18:29:43.133243 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:43.133121    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn" podUID="d9e3abf7-c5e1-4414-858e-5f84f6a88670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 03 18:29:44.081529 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:44.081489    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zp9n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Nov 03 18:29:44.081938 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:44.081558    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7" podUID="60b66cb6-2bce-4c02-aa7d-7eb25f29f237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Nov 03 18:29:44.133782 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:44.133743    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xpshn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 03 18:29:44.133945 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:44.133815    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn" podUID="d9e3abf7-c5e1-4414-858e-5f84f6a88670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 03 18:29:46.004425 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:46.004394    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:46.005578 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:46.005544    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:46.006247 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:46.006223    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 18:29:46.015331 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:46.015307    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:46.016063 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:46.016046    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:46.016578 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:46.016559    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 18:29:52.821518 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:52.821482    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75c4dd589d-nztw8"
Nov 03 18:29:55.228941 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:55.228894    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:29:55.404793 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:29:55.404755    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:30:00.183782 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:00.183744    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg"]
Nov 03 18:30:00.187818 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:00.187789    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg"
Nov 03 18:30:00.244519 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:00.244483    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:30:00.296452 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:00.296415    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:30:00.296645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:00.296524    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:30:00.301175 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:00.301129    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg"]
Nov 03 18:30:00.316164 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:00.316127    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/325f66dc-d7ba-44b8-a9df-06083595a26a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895510-86rrg\" (UID: \"325f66dc-d7ba-44b8-a9df-06083595a26a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg"
Nov 03 18:30:00.316343 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:00.316177    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dhlj\" (UniqueName: \"kubernetes.io/projected/325f66dc-d7ba-44b8-a9df-06083595a26a-kube-api-access-5dhlj\") pod \"maas-api-key-cleanup-29895510-86rrg\" (UID: \"325f66dc-d7ba-44b8-a9df-06083595a26a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg"
Nov 03 18:30:00.417407 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:00.417344    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dhlj\" (UniqueName: \"kubernetes.io/projected/325f66dc-d7ba-44b8-a9df-06083595a26a-kube-api-access-5dhlj\") pod \"maas-api-key-cleanup-29895510-86rrg\" (UID: \"325f66dc-d7ba-44b8-a9df-06083595a26a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg"
Nov 03 18:30:00.417620 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:00.417551    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/325f66dc-d7ba-44b8-a9df-06083595a26a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895510-86rrg\" (UID: \"325f66dc-d7ba-44b8-a9df-06083595a26a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg"
Nov 03 18:30:00.418173 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:00.418147    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/325f66dc-d7ba-44b8-a9df-06083595a26a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895510-86rrg\" (UID: \"325f66dc-d7ba-44b8-a9df-06083595a26a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg"
Nov 03 18:30:00.481272 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:00.481234    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dhlj\" (UniqueName: \"kubernetes.io/projected/325f66dc-d7ba-44b8-a9df-06083595a26a-kube-api-access-5dhlj\") pod \"maas-api-key-cleanup-29895510-86rrg\" (UID: \"325f66dc-d7ba-44b8-a9df-06083595a26a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg"
Nov 03 18:30:00.500483 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:00.500445    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg"
Nov 03 18:30:00.880073 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:30:00.880041    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod325f66dc_d7ba_44b8_a9df_06083595a26a.slice/crio-d62a8b501edb50a95d57040ce2a4872211c91167ccee38ebcb180610f8ad99b9 WatchSource:0}: Error finding container d62a8b501edb50a95d57040ce2a4872211c91167ccee38ebcb180610f8ad99b9: Status 404 returned error can't find the container with id d62a8b501edb50a95d57040ce2a4872211c91167ccee38ebcb180610f8ad99b9
Nov 03 18:30:00.881537 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:00.881501    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg"]
Nov 03 18:30:00.926740 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:00.926702    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg" event={"ID":"325f66dc-d7ba-44b8-a9df-06083595a26a","Type":"ContainerStarted","Data":"d62a8b501edb50a95d57040ce2a4872211c91167ccee38ebcb180610f8ad99b9"}
Nov 03 18:30:01.933617 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:01.933584    6059 generic.go:356] "Generic (PLEG): container finished" podID="325f66dc-d7ba-44b8-a9df-06083595a26a" containerID="509ab9094014992315a7169525ff02660e0ef7956ed970adbd6170bc85e6e31b" exitCode=60
Nov 03 18:30:01.934020 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:01.933673    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg" event={"ID":"325f66dc-d7ba-44b8-a9df-06083595a26a","Type":"ContainerDied","Data":"509ab9094014992315a7169525ff02660e0ef7956ed970adbd6170bc85e6e31b"}
Nov 03 18:30:01.934020 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:01.933921    6059 scope.go:122] "RemoveContainer" containerID="509ab9094014992315a7169525ff02660e0ef7956ed970adbd6170bc85e6e31b"
Nov 03 18:30:02.941547 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:02.941511    6059 generic.go:356] "Generic (PLEG): container finished" podID="325f66dc-d7ba-44b8-a9df-06083595a26a" containerID="1ea318b214edbf20938f48ea3b4bceee26d5f5154307a63c3fb8e68e9a4aa931" exitCode=60
Nov 03 18:30:02.942021 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:02.941605    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg" event={"ID":"325f66dc-d7ba-44b8-a9df-06083595a26a","Type":"ContainerDied","Data":"1ea318b214edbf20938f48ea3b4bceee26d5f5154307a63c3fb8e68e9a4aa931"}
Nov 03 18:30:02.942021 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:02.941669    6059 scope.go:122] "RemoveContainer" containerID="509ab9094014992315a7169525ff02660e0ef7956ed970adbd6170bc85e6e31b"
Nov 03 18:30:02.942021 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:02.941884    6059 scope.go:122] "RemoveContainer" containerID="1ea318b214edbf20938f48ea3b4bceee26d5f5154307a63c3fb8e68e9a4aa931"
Nov 03 18:30:02.942202 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:30:02.942123    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895510-86rrg_odh-ai-gateway-infra(325f66dc-d7ba-44b8-a9df-06083595a26a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg" podUID="325f66dc-d7ba-44b8-a9df-06083595a26a"
Nov 03 18:30:03.948836 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:03.948804    6059 scope.go:122] "RemoveContainer" containerID="1ea318b214edbf20938f48ea3b4bceee26d5f5154307a63c3fb8e68e9a4aa931"
Nov 03 18:30:03.949227 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:30:03.949048    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895510-86rrg_odh-ai-gateway-infra(325f66dc-d7ba-44b8-a9df-06083595a26a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg" podUID="325f66dc-d7ba-44b8-a9df-06083595a26a"
Nov 03 18:30:06.880189 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:30:06.880136    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:30:14.329656 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:14.329618    6059 scope.go:122] "RemoveContainer" containerID="1ea318b214edbf20938f48ea3b4bceee26d5f5154307a63c3fb8e68e9a4aa931"
Nov 03 18:30:15.020639 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:15.020604    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg" event={"ID":"325f66dc-d7ba-44b8-a9df-06083595a26a","Type":"ContainerStarted","Data":"782e6ecee8290d1bbe49be583aab04c6bbc20f08c164887072e355436d33a7ed"}
Nov 03 18:30:15.141393 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:15.141327    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg" podStartSLOduration=15.141311417 podStartE2EDuration="15.141311417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:30:15.140271521 +0000 UTC m=+4355.937984304" watchObservedRunningTime="2026-11-03 18:30:15.141311417 +0000 UTC m=+4355.939024198"
Nov 03 18:30:16.027230 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:16.027191    6059 generic.go:356] "Generic (PLEG): container finished" podID="325f66dc-d7ba-44b8-a9df-06083595a26a" containerID="782e6ecee8290d1bbe49be583aab04c6bbc20f08c164887072e355436d33a7ed" exitCode=60
Nov 03 18:30:16.027753 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:16.027274    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg" event={"ID":"325f66dc-d7ba-44b8-a9df-06083595a26a","Type":"ContainerDied","Data":"782e6ecee8290d1bbe49be583aab04c6bbc20f08c164887072e355436d33a7ed"}
Nov 03 18:30:16.027753 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:16.027344    6059 scope.go:122] "RemoveContainer" containerID="1ea318b214edbf20938f48ea3b4bceee26d5f5154307a63c3fb8e68e9a4aa931"
Nov 03 18:30:16.027753 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:16.027738    6059 scope.go:122] "RemoveContainer" containerID="782e6ecee8290d1bbe49be583aab04c6bbc20f08c164887072e355436d33a7ed"
Nov 03 18:30:16.028046 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:30:16.028022    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895510-86rrg_odh-ai-gateway-infra(325f66dc-d7ba-44b8-a9df-06083595a26a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg" podUID="325f66dc-d7ba-44b8-a9df-06083595a26a"
Nov 03 18:30:16.153769 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:16.153732    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg"]
Nov 03 18:30:17.185613 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:17.185587    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg"
Nov 03 18:30:17.294849 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:17.294762    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/325f66dc-d7ba-44b8-a9df-06083595a26a-kube-api-access-5dhlj\" (UniqueName: \"kubernetes.io/projected/325f66dc-d7ba-44b8-a9df-06083595a26a-kube-api-access-5dhlj\") pod \"325f66dc-d7ba-44b8-a9df-06083595a26a\" (UID: \"325f66dc-d7ba-44b8-a9df-06083595a26a\") "
Nov 03 18:30:17.294849 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:17.294841    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/325f66dc-d7ba-44b8-a9df-06083595a26a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/325f66dc-d7ba-44b8-a9df-06083595a26a-maas-api-service-ca\") pod \"325f66dc-d7ba-44b8-a9df-06083595a26a\" (UID: \"325f66dc-d7ba-44b8-a9df-06083595a26a\") "
Nov 03 18:30:17.295212 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:17.295173    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/325f66dc-d7ba-44b8-a9df-06083595a26a-maas-api-service-ca" pod "325f66dc-d7ba-44b8-a9df-06083595a26a" (UID: "325f66dc-d7ba-44b8-a9df-06083595a26a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:30:17.296894 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:17.296876    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/325f66dc-d7ba-44b8-a9df-06083595a26a-kube-api-access-5dhlj" pod "325f66dc-d7ba-44b8-a9df-06083595a26a" (UID: "325f66dc-d7ba-44b8-a9df-06083595a26a"). InnerVolumeSpecName "kube-api-access-5dhlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:30:17.395615 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:17.395576    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dhlj\" (UniqueName: \"kubernetes.io/projected/325f66dc-d7ba-44b8-a9df-06083595a26a-kube-api-access-5dhlj\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:30:17.395615 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:17.395607    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/325f66dc-d7ba-44b8-a9df-06083595a26a-maas-api-service-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:30:18.044020 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:18.043991    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg"
Nov 03 18:30:18.044020 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:18.044014    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg" event={"ID":"325f66dc-d7ba-44b8-a9df-06083595a26a","Type":"ContainerDied","Data":"d62a8b501edb50a95d57040ce2a4872211c91167ccee38ebcb180610f8ad99b9"}
Nov 03 18:30:18.044284 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:18.044082    6059 scope.go:122] "RemoveContainer" containerID="782e6ecee8290d1bbe49be583aab04c6bbc20f08c164887072e355436d33a7ed"
Nov 03 18:30:18.213992 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:18.213955    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg"]
Nov 03 18:30:18.265749 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:18.265712    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895510-86rrg"]
Nov 03 18:30:19.335951 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:19.335914    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="325f66dc-d7ba-44b8-a9df-06083595a26a" path="/var/lib/kubelet/pods/325f66dc-d7ba-44b8-a9df-06083595a26a/volumes"
Nov 03 18:30:36.050614 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.050578    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-726791df-nqhtr"]
Nov 03 18:30:36.102265 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.102231    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-726791df-nqhtr"]
Nov 03 18:30:36.240488 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.240445    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"]
Nov 03 18:30:36.240890 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.240875    6059 cpu_manager.go:414] "containerMap: removing container" podUID="325f66dc-d7ba-44b8-a9df-06083595a26a" containerName="cleanup"
Nov 03 18:30:36.240944 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.240890    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="325f66dc-d7ba-44b8-a9df-06083595a26a" containerName="cleanup"
Nov 03 18:30:36.240944 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.240929    6059 cpu_manager.go:414] "containerMap: removing container" podUID="325f66dc-d7ba-44b8-a9df-06083595a26a" containerName="cleanup"
Nov 03 18:30:36.240944 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.240935    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="325f66dc-d7ba-44b8-a9df-06083595a26a" containerName="cleanup"
Nov 03 18:30:36.240944 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.240942    6059 cpu_manager.go:414] "containerMap: removing container" podUID="325f66dc-d7ba-44b8-a9df-06083595a26a" containerName="cleanup"
Nov 03 18:30:36.241079 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.240948    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="325f66dc-d7ba-44b8-a9df-06083595a26a" containerName="cleanup"
Nov 03 18:30:36.241079 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.241040    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="325f66dc-d7ba-44b8-a9df-06083595a26a" containerName="cleanup"
Nov 03 18:30:36.241079 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.241055    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="325f66dc-d7ba-44b8-a9df-06083595a26a" containerName="cleanup"
Nov 03 18:30:36.245682 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.245653    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.276254 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.276216    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"]
Nov 03 18:30:36.276786 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.276769    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="325f66dc-d7ba-44b8-a9df-06083595a26a" containerName="cleanup"
Nov 03 18:30:36.280018 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.279999    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.307293 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.307207    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"]
Nov 03 18:30:36.372551 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.372519    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"]
Nov 03 18:30:36.403699 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.403661    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.403894 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.403713    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.403894 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.403736    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.403894 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.403815    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.403894 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.403844    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.404102 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.403913    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2159e743-a61b-4d76-9f78-9be36e825a43-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.404102 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.403953    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srnp4\" (UniqueName: \"kubernetes.io/projected/5eb79e59-86dc-47d1-830f-49e542a5b8f3-kube-api-access-srnp4\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.404102 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.403994    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2159e743-a61b-4d76-9f78-9be36e825a43-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.404102 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.404079    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2159e743-a61b-4d76-9f78-9be36e825a43-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.404278 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.404124    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.404278 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.404241    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.404350 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.404299    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.404413 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.404346    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.404449 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.404437    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.404500 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.404486    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.404539 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.404508    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.404539 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.404527    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsbjh\" (UniqueName: \"kubernetes.io/projected/2159e743-a61b-4d76-9f78-9be36e825a43-kube-api-access-wsbjh\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.404608 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.404548    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.505704 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.505660    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.505704 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.505706    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.505975 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.505749    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.505975 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.505775    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.505975 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.505797    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.505975 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.505824    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.505975 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.505851    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsbjh\" (UniqueName: \"kubernetes.io/projected/2159e743-a61b-4d76-9f78-9be36e825a43-kube-api-access-wsbjh\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.505975 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.505884    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.505975 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.505921    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.505975 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.505969    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.506463 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.505997    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.506463 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.506025    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.506463 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.506052    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.506463 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.506086    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2159e743-a61b-4d76-9f78-9be36e825a43-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.506463 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.506119    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srnp4\" (UniqueName: \"kubernetes.io/projected/5eb79e59-86dc-47d1-830f-49e542a5b8f3-kube-api-access-srnp4\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.506463 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.506119    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.506463 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.506186    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2159e743-a61b-4d76-9f78-9be36e825a43-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.506463 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.506218    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2159e743-a61b-4d76-9f78-9be36e825a43-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.506463 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.506246    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.506463 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.506301    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.506463 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.506456    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.507085 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.506492    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.507085 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.506618    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.507085 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.506653    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.507085 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.506773    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.507085 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.506930    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.507333 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.507087    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2159e743-a61b-4d76-9f78-9be36e825a43-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.507333 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.507129    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.508466 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.508438    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.508712 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.508688    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.508978 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.508959    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.509401 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.509384    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2159e743-a61b-4d76-9f78-9be36e825a43-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.569386 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.569291    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2159e743-a61b-4d76-9f78-9be36e825a43-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.569565 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.569389    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.569565 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.569485    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srnp4\" (UniqueName: \"kubernetes.io/projected/5eb79e59-86dc-47d1-830f-49e542a5b8f3-kube-api-access-srnp4\") pod \"maas-default-gateway-openshift-default-58c6585447-l2xpl\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.569646 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.569610    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsbjh\" (UniqueName: \"kubernetes.io/projected/2159e743-a61b-4d76-9f78-9be36e825a43-kube-api-access-wsbjh\") pod \"maas-default-gateway-openshift-default-58c6585447-gxxkk\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.593867 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.593828    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:36.858778 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:36.858687    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:36.862152 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:30:36.862119    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:30:37.018489 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:37.018449    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:30:37.018638 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:37.018532    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:30:37.018638 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:37.018566    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:30:37.019322 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:37.019290    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"]
Nov 03 18:30:37.162811 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:37.162705    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" event={"ID":"5eb79e59-86dc-47d1-830f-49e542a5b8f3","Type":"ContainerStarted","Data":"7b05d327a1d2a420fd123ff5f8ea128b8306222325363ab777c69fe2bca1eb9f"}
Nov 03 18:30:37.162811 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:37.162775    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" event={"ID":"5eb79e59-86dc-47d1-830f-49e542a5b8f3","Type":"ContainerStarted","Data":"9f5328318917f5308948233557571fdb2dcfa895ed7667aef175000885f26fb5"}
Nov 03 18:30:37.233685 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:30:37.233654    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2159e743_a61b_4d76_9f78_9be36e825a43.slice/crio-46390c01d533ba6fe5ef111dc75407a8b0e76ec9b76e711ff3ec12f1e1753197 WatchSource:0}: Error finding container 46390c01d533ba6fe5ef111dc75407a8b0e76ec9b76e711ff3ec12f1e1753197: Status 404 returned error can't find the container with id 46390c01d533ba6fe5ef111dc75407a8b0e76ec9b76e711ff3ec12f1e1753197
Nov 03 18:30:37.234947 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:37.234921    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"]
Nov 03 18:30:37.235941 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:37.235913    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:30:37.236021 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:37.235974    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:30:37.236021 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:37.236003    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:30:37.287130 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:37.287079    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" podStartSLOduration=1.287064423 podStartE2EDuration="1.287064423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:30:37.286132581 +0000 UTC m=+4378.083845362" watchObservedRunningTime="2026-11-03 18:30:37.287064423 +0000 UTC m=+4378.084777204"
Nov 03 18:30:37.339463 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:37.339423    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9114e7f0-5812-47c3-a08b-e77e810a8188" path="/var/lib/kubelet/pods/9114e7f0-5812-47c3-a08b-e77e810a8188/volumes"
Nov 03 18:30:37.593955 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:37.593909    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:38.188969 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:38.188020    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" event={"ID":"2159e743-a61b-4d76-9f78-9be36e825a43","Type":"ContainerStarted","Data":"a534099f4dffc13956221f54dfeab90e33cc6c87c30f1bbc8f67ee27d2c90ba1"}
Nov 03 18:30:38.188969 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:38.188082    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" event={"ID":"2159e743-a61b-4d76-9f78-9be36e825a43","Type":"ContainerStarted","Data":"46390c01d533ba6fe5ef111dc75407a8b0e76ec9b76e711ff3ec12f1e1753197"}
Nov 03 18:30:38.315223 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:38.315169    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" podStartSLOduration=2.315152188 podStartE2EDuration="2.315152188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:30:38.312575387 +0000 UTC m=+4379.110288169" watchObservedRunningTime="2026-11-03 18:30:38.315152188 +0000 UTC m=+4379.112864967"
Nov 03 18:30:38.596381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:38.596320    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-l2xpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Nov 03 18:30:38.596579 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:38.596396    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" podUID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Nov 03 18:30:38.860032 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:38.859912    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:38.862146 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:38.862113    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gxxkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Nov 03 18:30:38.862427 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:38.862398    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" podUID="2159e743-a61b-4d76-9f78-9be36e825a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Nov 03 18:30:39.594827 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:39.594789    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-l2xpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Nov 03 18:30:39.595227 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:39.594857    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" podUID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Nov 03 18:30:39.861084 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:39.860981    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gxxkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Nov 03 18:30:39.861416 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:39.861350    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" podUID="2159e743-a61b-4d76-9f78-9be36e825a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Nov 03 18:30:40.595040 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:40.595001    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-l2xpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Nov 03 18:30:40.595554 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:40.595052    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" podUID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Nov 03 18:30:40.871441 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:40.870869    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gxxkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Nov 03 18:30:40.871441 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:40.870959    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" podUID="2159e743-a61b-4d76-9f78-9be36e825a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Nov 03 18:30:41.594395 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:41.594326    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-l2xpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Nov 03 18:30:41.594586 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:41.594403    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" podUID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Nov 03 18:30:41.859879 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:41.859786    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gxxkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Nov 03 18:30:41.860482 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:41.860447    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" podUID="2159e743-a61b-4d76-9f78-9be36e825a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Nov 03 18:30:42.594536 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:42.594426    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-l2xpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Nov 03 18:30:42.594536 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:42.594498    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" podUID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Nov 03 18:30:42.860900 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:42.860802    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gxxkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Nov 03 18:30:42.861509 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:42.861477    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" podUID="2159e743-a61b-4d76-9f78-9be36e825a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Nov 03 18:30:43.594606 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:43.594499    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-l2xpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Nov 03 18:30:43.594606 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:43.594570    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" podUID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Nov 03 18:30:43.859895 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:43.859793    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gxxkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Nov 03 18:30:43.860185 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:43.860152    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" podUID="2159e743-a61b-4d76-9f78-9be36e825a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Nov 03 18:30:44.594866 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:44.594821    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-l2xpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Nov 03 18:30:44.595525 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:44.595487    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" podUID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Nov 03 18:30:44.859983 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:44.859882    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gxxkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Nov 03 18:30:44.860282 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:44.860248    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" podUID="2159e743-a61b-4d76-9f78-9be36e825a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Nov 03 18:30:45.596928 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:45.596890    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-l2xpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Nov 03 18:30:45.597616 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:45.597544    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" podUID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Nov 03 18:30:45.859887 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:45.859785    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gxxkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Nov 03 18:30:45.860178 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:45.860145    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" podUID="2159e743-a61b-4d76-9f78-9be36e825a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Nov 03 18:30:47.598437 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:47.598397    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-l2xpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:30:47.598911 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:47.598470    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" podUID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:30:47.859514 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:47.859416    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gxxkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:30:47.859514 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:47.859481    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" podUID="2159e743-a61b-4d76-9f78-9be36e825a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:30:48.050342 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.050300    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml"]
Nov 03 18:30:48.101999 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.101951    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee19937a-56dml"]
Nov 03 18:30:48.495949 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.495908    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kcmw7"]
Nov 03 18:30:48.499891 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.499862    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kcmw7"
Nov 03 18:30:48.540825 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.540780    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05846600-c826-431d-974b-3e2108fa19a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-kcmw7\" (UID: \"05846600-c826-431d-974b-3e2108fa19a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kcmw7"
Nov 03 18:30:48.541039 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.540966    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cqrk\" (UniqueName: \"kubernetes.io/projected/05846600-c826-431d-974b-3e2108fa19a2-kube-api-access-4cqrk\") pod \"maas-api-revoke-sub-f03629db-kcmw7\" (UID: \"05846600-c826-431d-974b-3e2108fa19a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kcmw7"
Nov 03 18:30:48.541039 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.541009    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05846600-c826-431d-974b-3e2108fa19a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-kcmw7\" (UID: \"05846600-c826-431d-974b-3e2108fa19a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kcmw7"
Nov 03 18:30:48.556954 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.556921    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:30:48.557160 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.556923    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:30:48.557304 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.557012    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:30:48.557304 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.557101    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:30:48.559989 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.559955    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kcmw7"]
Nov 03 18:30:48.599249 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.599200    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-l2xpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:30:48.599806 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.599325    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" podUID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:30:48.642220 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.642164    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05846600-c826-431d-974b-3e2108fa19a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-kcmw7\" (UID: \"05846600-c826-431d-974b-3e2108fa19a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kcmw7"
Nov 03 18:30:48.642484 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.642293    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cqrk\" (UniqueName: \"kubernetes.io/projected/05846600-c826-431d-974b-3e2108fa19a2-kube-api-access-4cqrk\") pod \"maas-api-revoke-sub-f03629db-kcmw7\" (UID: \"05846600-c826-431d-974b-3e2108fa19a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kcmw7"
Nov 03 18:30:48.642484 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.642350    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05846600-c826-431d-974b-3e2108fa19a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-kcmw7\" (UID: \"05846600-c826-431d-974b-3e2108fa19a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kcmw7"
Nov 03 18:30:48.642960 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.642935    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05846600-c826-431d-974b-3e2108fa19a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-kcmw7\" (UID: \"05846600-c826-431d-974b-3e2108fa19a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kcmw7"
Nov 03 18:30:48.644872 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.644842    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05846600-c826-431d-974b-3e2108fa19a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-kcmw7\" (UID: \"05846600-c826-431d-974b-3e2108fa19a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kcmw7"
Nov 03 18:30:48.705957 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.705913    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cqrk\" (UniqueName: \"kubernetes.io/projected/05846600-c826-431d-974b-3e2108fa19a2-kube-api-access-4cqrk\") pod \"maas-api-revoke-sub-f03629db-kcmw7\" (UID: \"05846600-c826-431d-974b-3e2108fa19a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kcmw7"
Nov 03 18:30:48.814599 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.814492    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kcmw7"
Nov 03 18:30:48.860816 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.860045    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gxxkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:30:48.860816 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:48.860119    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" podUID="2159e743-a61b-4d76-9f78-9be36e825a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:30:49.201373 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:30:49.201324    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05846600_c826_431d_974b_3e2108fa19a2.slice/crio-0bf2d2372ad3dd229a52b317f518620a7ce5c758e84867ee1c9de486ca7bf891 WatchSource:0}: Error finding container 0bf2d2372ad3dd229a52b317f518620a7ce5c758e84867ee1c9de486ca7bf891: Status 404 returned error can't find the container with id 0bf2d2372ad3dd229a52b317f518620a7ce5c758e84867ee1c9de486ca7bf891
Nov 03 18:30:49.204281 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:49.204252    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kcmw7"]
Nov 03 18:30:49.305336 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:49.305294    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kcmw7" event={"ID":"05846600-c826-431d-974b-3e2108fa19a2","Type":"ContainerStarted","Data":"0bf2d2372ad3dd229a52b317f518620a7ce5c758e84867ee1c9de486ca7bf891"}
Nov 03 18:30:49.359696 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:49.359642    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99f6c5fc-3a07-4154-9570-cff5b49feedd" path="/var/lib/kubelet/pods/99f6c5fc-3a07-4154-9570-cff5b49feedd/volumes"
Nov 03 18:30:49.642078 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:49.641557    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-l2xpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:30:49.642078 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:49.641630    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" podUID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:30:49.902562 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:49.902526    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gxxkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:30:49.902753 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:49.902603    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" podUID="2159e743-a61b-4d76-9f78-9be36e825a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:30:50.312083 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:50.312051    6059 generic.go:356] "Generic (PLEG): container finished" podID="05846600-c826-431d-974b-3e2108fa19a2" containerID="1293ecd1921187c07ac1ac1174eef220689ad7b53c2a4ce45ed7e540a6badc76" exitCode=0
Nov 03 18:30:50.312234 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:50.312138    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kcmw7" event={"ID":"05846600-c826-431d-974b-3e2108fa19a2","Type":"ContainerDied","Data":"1293ecd1921187c07ac1ac1174eef220689ad7b53c2a4ce45ed7e540a6badc76"}
Nov 03 18:30:50.684700 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:50.684580    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-l2xpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:30:50.684700 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:50.684658    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" podUID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:30:50.903239 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:50.903198    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gxxkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:30:50.903482 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:50.903312    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" podUID="2159e743-a61b-4d76-9f78-9be36e825a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:30:51.118573 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:51.118536    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:51.119107 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:51.119081    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:51.228350 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:51.228303    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:51.228670 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:51.228650    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:51.229701 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:51.229676    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 18:30:51.290147 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:51.290113    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 18:30:51.476263 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:51.476231    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kcmw7"
Nov 03 18:30:51.578064 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:51.578032    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05846600-c826-431d-974b-3e2108fa19a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05846600-c826-431d-974b-3e2108fa19a2-maas-api-openshift-ca\") pod \"05846600-c826-431d-974b-3e2108fa19a2\" (UID: \"05846600-c826-431d-974b-3e2108fa19a2\") "
Nov 03 18:30:51.578273 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:51.578196    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05846600-c826-431d-974b-3e2108fa19a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05846600-c826-431d-974b-3e2108fa19a2-maas-api-xks-ca\") pod \"05846600-c826-431d-974b-3e2108fa19a2\" (UID: \"05846600-c826-431d-974b-3e2108fa19a2\") "
Nov 03 18:30:51.578273 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:51.578221    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05846600-c826-431d-974b-3e2108fa19a2-kube-api-access-4cqrk\" (UniqueName: \"kubernetes.io/projected/05846600-c826-431d-974b-3e2108fa19a2-kube-api-access-4cqrk\") pod \"05846600-c826-431d-974b-3e2108fa19a2\" (UID: \"05846600-c826-431d-974b-3e2108fa19a2\") "
Nov 03 18:30:51.578700 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:51.578477    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05846600-c826-431d-974b-3e2108fa19a2-maas-api-openshift-ca" pod "05846600-c826-431d-974b-3e2108fa19a2" (UID: "05846600-c826-431d-974b-3e2108fa19a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:30:51.580250 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:51.580221    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05846600-c826-431d-974b-3e2108fa19a2-kube-api-access-4cqrk" pod "05846600-c826-431d-974b-3e2108fa19a2" (UID: "05846600-c826-431d-974b-3e2108fa19a2"). InnerVolumeSpecName "kube-api-access-4cqrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:30:51.580385 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:51.580292    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05846600-c826-431d-974b-3e2108fa19a2-maas-api-xks-ca" pod "05846600-c826-431d-974b-3e2108fa19a2" (UID: "05846600-c826-431d-974b-3e2108fa19a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:30:51.679905 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:51.679800    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05846600-c826-431d-974b-3e2108fa19a2-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:30:51.679905 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:51.679843    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cqrk\" (UniqueName: \"kubernetes.io/projected/05846600-c826-431d-974b-3e2108fa19a2-kube-api-access-4cqrk\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:30:51.679905 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:51.679859    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05846600-c826-431d-974b-3e2108fa19a2-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:30:52.325893 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:52.325859    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kcmw7"
Nov 03 18:30:52.326314 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:52.325897    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kcmw7" event={"ID":"05846600-c826-431d-974b-3e2108fa19a2","Type":"ContainerDied","Data":"0bf2d2372ad3dd229a52b317f518620a7ce5c758e84867ee1c9de486ca7bf891"}
Nov 03 18:30:52.326314 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:52.325945    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bf2d2372ad3dd229a52b317f518620a7ce5c758e84867ee1c9de486ca7bf891"
Nov 03 18:30:55.570881 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:30:55.570850    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:07.051275 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:07.051227    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5"]
Nov 03 18:31:07.053644 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:31:07.053605    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:31:07.115379 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:07.114901    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a8933b7-5h7g5"]
Nov 03 18:31:07.177501 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:07.177458    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-l2xpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:31:07.177866 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:07.177815    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" podUID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:31:07.308569 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:07.308425    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gxxkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:31:07.308892 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:07.308844    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" podUID="2159e743-a61b-4d76-9f78-9be36e825a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:31:07.386384 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:07.377899    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ff3966-c839-4d0b-880f-0dd889cc54c3" path="/var/lib/kubelet/pods/46ff3966-c839-4d0b-880f-0dd889cc54c3/volumes"
Nov 03 18:31:08.924337 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:08.924305    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zx87s"]
Nov 03 18:31:08.924778 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:08.924746    6059 cpu_manager.go:414] "containerMap: removing container" podUID="05846600-c826-431d-974b-3e2108fa19a2" containerName="revoke-keys"
Nov 03 18:31:08.924778 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:08.924757    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05846600-c826-431d-974b-3e2108fa19a2" containerName="revoke-keys"
Nov 03 18:31:08.924858 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:08.924842    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="05846600-c826-431d-974b-3e2108fa19a2" containerName="revoke-keys"
Nov 03 18:31:08.929486 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:08.929466    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zx87s"
Nov 03 18:31:09.009416 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:09.009379    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grl26\" (UniqueName: \"kubernetes.io/projected/94f9bd3a-302f-4115-88ac-6b542ece9817-kube-api-access-grl26\") pod \"maas-api-revoke-sub-0165d7d2-zx87s\" (UID: \"94f9bd3a-302f-4115-88ac-6b542ece9817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zx87s"
Nov 03 18:31:09.009636 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:09.009507    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f9bd3a-302f-4115-88ac-6b542ece9817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zx87s\" (UID: \"94f9bd3a-302f-4115-88ac-6b542ece9817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zx87s"
Nov 03 18:31:09.009636 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:09.009596    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f9bd3a-302f-4115-88ac-6b542ece9817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zx87s\" (UID: \"94f9bd3a-302f-4115-88ac-6b542ece9817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zx87s"
Nov 03 18:31:09.037060 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:09.037027    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:31:09.037060 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:09.037075    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:31:09.037413 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:09.037075    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:31:09.037413 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:09.037127    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:31:09.039422 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:09.039398    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zx87s"]
Nov 03 18:31:09.110258 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:09.110224    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f9bd3a-302f-4115-88ac-6b542ece9817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zx87s\" (UID: \"94f9bd3a-302f-4115-88ac-6b542ece9817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zx87s"
Nov 03 18:31:09.110559 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:09.110286    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f9bd3a-302f-4115-88ac-6b542ece9817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zx87s\" (UID: \"94f9bd3a-302f-4115-88ac-6b542ece9817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zx87s"
Nov 03 18:31:09.110559 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:09.110407    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grl26\" (UniqueName: \"kubernetes.io/projected/94f9bd3a-302f-4115-88ac-6b542ece9817-kube-api-access-grl26\") pod \"maas-api-revoke-sub-0165d7d2-zx87s\" (UID: \"94f9bd3a-302f-4115-88ac-6b542ece9817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zx87s"
Nov 03 18:31:09.111106 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:09.111078    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f9bd3a-302f-4115-88ac-6b542ece9817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zx87s\" (UID: \"94f9bd3a-302f-4115-88ac-6b542ece9817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zx87s"
Nov 03 18:31:09.112865 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:09.112839    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f9bd3a-302f-4115-88ac-6b542ece9817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zx87s\" (UID: \"94f9bd3a-302f-4115-88ac-6b542ece9817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zx87s"
Nov 03 18:31:09.173746 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:09.173711    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grl26\" (UniqueName: \"kubernetes.io/projected/94f9bd3a-302f-4115-88ac-6b542ece9817-kube-api-access-grl26\") pod \"maas-api-revoke-sub-0165d7d2-zx87s\" (UID: \"94f9bd3a-302f-4115-88ac-6b542ece9817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zx87s"
Nov 03 18:31:09.242962 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:09.242922    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zx87s"
Nov 03 18:31:09.619886 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:31:09.619845    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f9bd3a_302f_4115_88ac_6b542ece9817.slice/crio-8fd8b94f70a434d8d210c3be5f59a933a36e223cedd2161666a9ddb847ab0ed8 WatchSource:0}: Error finding container 8fd8b94f70a434d8d210c3be5f59a933a36e223cedd2161666a9ddb847ab0ed8: Status 404 returned error can't find the container with id 8fd8b94f70a434d8d210c3be5f59a933a36e223cedd2161666a9ddb847ab0ed8
Nov 03 18:31:09.621606 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:09.621582    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zx87s"]
Nov 03 18:31:10.539959 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:10.539924    6059 generic.go:356] "Generic (PLEG): container finished" podID="94f9bd3a-302f-4115-88ac-6b542ece9817" containerID="9eb42916fccdf7e0bb4fdefb43eadbd95e99139ed116d6249decccd24b06036d" exitCode=0
Nov 03 18:31:10.540400 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:10.540003    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zx87s" event={"ID":"94f9bd3a-302f-4115-88ac-6b542ece9817","Type":"ContainerDied","Data":"9eb42916fccdf7e0bb4fdefb43eadbd95e99139ed116d6249decccd24b06036d"}
Nov 03 18:31:10.540400 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:10.540054    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zx87s" event={"ID":"94f9bd3a-302f-4115-88ac-6b542ece9817","Type":"ContainerStarted","Data":"8fd8b94f70a434d8d210c3be5f59a933a36e223cedd2161666a9ddb847ab0ed8"}
Nov 03 18:31:11.694238 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:11.694209    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zx87s"
Nov 03 18:31:11.737512 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:11.737479    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94f9bd3a-302f-4115-88ac-6b542ece9817-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f9bd3a-302f-4115-88ac-6b542ece9817-maas-api-openshift-ca\") pod \"94f9bd3a-302f-4115-88ac-6b542ece9817\" (UID: \"94f9bd3a-302f-4115-88ac-6b542ece9817\") "
Nov 03 18:31:11.737512 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:11.737519    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94f9bd3a-302f-4115-88ac-6b542ece9817-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f9bd3a-302f-4115-88ac-6b542ece9817-maas-api-xks-ca\") pod \"94f9bd3a-302f-4115-88ac-6b542ece9817\" (UID: \"94f9bd3a-302f-4115-88ac-6b542ece9817\") "
Nov 03 18:31:11.737804 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:11.737542    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f9bd3a-302f-4115-88ac-6b542ece9817-kube-api-access-grl26\" (UniqueName: \"kubernetes.io/projected/94f9bd3a-302f-4115-88ac-6b542ece9817-kube-api-access-grl26\") pod \"94f9bd3a-302f-4115-88ac-6b542ece9817\" (UID: \"94f9bd3a-302f-4115-88ac-6b542ece9817\") "
Nov 03 18:31:11.737878 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:11.737843    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f9bd3a-302f-4115-88ac-6b542ece9817-maas-api-openshift-ca" pod "94f9bd3a-302f-4115-88ac-6b542ece9817" (UID: "94f9bd3a-302f-4115-88ac-6b542ece9817"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:31:11.737955 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:11.737936    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f9bd3a-302f-4115-88ac-6b542ece9817-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:31:11.739613 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:11.739587    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94f9bd3a-302f-4115-88ac-6b542ece9817-maas-api-xks-ca" pod "94f9bd3a-302f-4115-88ac-6b542ece9817" (UID: "94f9bd3a-302f-4115-88ac-6b542ece9817"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:31:11.739750 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:11.739733    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f9bd3a-302f-4115-88ac-6b542ece9817-kube-api-access-grl26" pod "94f9bd3a-302f-4115-88ac-6b542ece9817" (UID: "94f9bd3a-302f-4115-88ac-6b542ece9817"). InnerVolumeSpecName "kube-api-access-grl26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:31:11.839861 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:11.839765    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f9bd3a-302f-4115-88ac-6b542ece9817-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:31:11.839861 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:11.839807    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grl26\" (UniqueName: \"kubernetes.io/projected/94f9bd3a-302f-4115-88ac-6b542ece9817-kube-api-access-grl26\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:31:12.551881 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:12.551843    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zx87s"
Nov 03 18:31:12.552092 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:12.551878    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zx87s" event={"ID":"94f9bd3a-302f-4115-88ac-6b542ece9817","Type":"ContainerDied","Data":"8fd8b94f70a434d8d210c3be5f59a933a36e223cedd2161666a9ddb847ab0ed8"}
Nov 03 18:31:12.552092 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:12.551941    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fd8b94f70a434d8d210c3be5f59a933a36e223cedd2161666a9ddb847ab0ed8"
Nov 03 18:31:18.051603 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:18.051506    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff6d6a8e-v7tvz"]
Nov 03 18:31:18.103274 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:18.103224    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff6d6a8e-v7tvz"]
Nov 03 18:31:19.334773 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:19.334738    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f24b5a6-eb74-4310-8cea-8645fd109b8e" path="/var/lib/kubelet/pods/0f24b5a6-eb74-4310-8cea-8645fd109b8e/volumes"
Nov 03 18:31:19.671210 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:19.671123    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:19.855501 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:19.855469    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:20.663994 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:20.663948    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lkxgq"]
Nov 03 18:31:20.664640 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:20.664617    6059 cpu_manager.go:414] "containerMap: removing container" podUID="94f9bd3a-302f-4115-88ac-6b542ece9817" containerName="revoke-keys"
Nov 03 18:31:20.664730 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:20.664642    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f9bd3a-302f-4115-88ac-6b542ece9817" containerName="revoke-keys"
Nov 03 18:31:20.664792 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:20.664775    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f9bd3a-302f-4115-88ac-6b542ece9817" containerName="revoke-keys"
Nov 03 18:31:20.668401 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:20.668379    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lkxgq"
Nov 03 18:31:20.732794 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:20.732760    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347f02b0-79ec-493b-8e06-bae4d0a7d62d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-lkxgq\" (UID: \"347f02b0-79ec-493b-8e06-bae4d0a7d62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lkxgq"
Nov 03 18:31:20.732996 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:20.732919    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv2ft\" (UniqueName: \"kubernetes.io/projected/347f02b0-79ec-493b-8e06-bae4d0a7d62d-kube-api-access-vv2ft\") pod \"maas-api-revoke-sub-6b19338f-lkxgq\" (UID: \"347f02b0-79ec-493b-8e06-bae4d0a7d62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lkxgq"
Nov 03 18:31:20.732996 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:20.732973    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347f02b0-79ec-493b-8e06-bae4d0a7d62d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-lkxgq\" (UID: \"347f02b0-79ec-493b-8e06-bae4d0a7d62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lkxgq"
Nov 03 18:31:20.777220 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:20.777182    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:31:20.777486 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:20.777288    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:31:20.777486 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:20.777317    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:31:20.777486 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:20.777328    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:31:20.779343 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:20.779318    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lkxgq"]
Nov 03 18:31:20.833756 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:20.833719    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347f02b0-79ec-493b-8e06-bae4d0a7d62d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-lkxgq\" (UID: \"347f02b0-79ec-493b-8e06-bae4d0a7d62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lkxgq"
Nov 03 18:31:20.833979 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:20.833778    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv2ft\" (UniqueName: \"kubernetes.io/projected/347f02b0-79ec-493b-8e06-bae4d0a7d62d-kube-api-access-vv2ft\") pod \"maas-api-revoke-sub-6b19338f-lkxgq\" (UID: \"347f02b0-79ec-493b-8e06-bae4d0a7d62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lkxgq"
Nov 03 18:31:20.833979 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:20.833804    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347f02b0-79ec-493b-8e06-bae4d0a7d62d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-lkxgq\" (UID: \"347f02b0-79ec-493b-8e06-bae4d0a7d62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lkxgq"
Nov 03 18:31:20.834366 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:20.834330    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347f02b0-79ec-493b-8e06-bae4d0a7d62d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-lkxgq\" (UID: \"347f02b0-79ec-493b-8e06-bae4d0a7d62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lkxgq"
Nov 03 18:31:20.836126 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:20.836100    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347f02b0-79ec-493b-8e06-bae4d0a7d62d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-lkxgq\" (UID: \"347f02b0-79ec-493b-8e06-bae4d0a7d62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lkxgq"
Nov 03 18:31:20.897170 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:20.897139    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv2ft\" (UniqueName: \"kubernetes.io/projected/347f02b0-79ec-493b-8e06-bae4d0a7d62d-kube-api-access-vv2ft\") pod \"maas-api-revoke-sub-6b19338f-lkxgq\" (UID: \"347f02b0-79ec-493b-8e06-bae4d0a7d62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lkxgq"
Nov 03 18:31:20.981564 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:20.981518    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lkxgq"
Nov 03 18:31:21.354514 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:31:21.354476    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347f02b0_79ec_493b_8e06_bae4d0a7d62d.slice/crio-1ba465418d001796fa76ec3ab48b2958e2f6e28ab5ca439e10b6212e4bedb47e WatchSource:0}: Error finding container 1ba465418d001796fa76ec3ab48b2958e2f6e28ab5ca439e10b6212e4bedb47e: Status 404 returned error can't find the container with id 1ba465418d001796fa76ec3ab48b2958e2f6e28ab5ca439e10b6212e4bedb47e
Nov 03 18:31:21.356870 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:21.356844    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lkxgq"]
Nov 03 18:31:21.601763 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:21.601669    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lkxgq" event={"ID":"347f02b0-79ec-493b-8e06-bae4d0a7d62d","Type":"ContainerStarted","Data":"1ba465418d001796fa76ec3ab48b2958e2f6e28ab5ca439e10b6212e4bedb47e"}
Nov 03 18:31:22.609305 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:22.609270    6059 generic.go:356] "Generic (PLEG): container finished" podID="347f02b0-79ec-493b-8e06-bae4d0a7d62d" containerID="cb3f721075581674ffa54729b669393143bbbf951e263de5908518c60dca6aec" exitCode=0
Nov 03 18:31:22.609754 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:22.609342    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lkxgq" event={"ID":"347f02b0-79ec-493b-8e06-bae4d0a7d62d","Type":"ContainerDied","Data":"cb3f721075581674ffa54729b669393143bbbf951e263de5908518c60dca6aec"}
Nov 03 18:31:23.759409 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:23.759382    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lkxgq"
Nov 03 18:31:23.865509 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:23.865408    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/347f02b0-79ec-493b-8e06-bae4d0a7d62d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347f02b0-79ec-493b-8e06-bae4d0a7d62d-maas-api-openshift-ca\") pod \"347f02b0-79ec-493b-8e06-bae4d0a7d62d\" (UID: \"347f02b0-79ec-493b-8e06-bae4d0a7d62d\") "
Nov 03 18:31:23.865509 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:23.865496    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/347f02b0-79ec-493b-8e06-bae4d0a7d62d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347f02b0-79ec-493b-8e06-bae4d0a7d62d-maas-api-xks-ca\") pod \"347f02b0-79ec-493b-8e06-bae4d0a7d62d\" (UID: \"347f02b0-79ec-493b-8e06-bae4d0a7d62d\") "
Nov 03 18:31:23.865771 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:23.865605    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/347f02b0-79ec-493b-8e06-bae4d0a7d62d-kube-api-access-vv2ft\" (UniqueName: \"kubernetes.io/projected/347f02b0-79ec-493b-8e06-bae4d0a7d62d-kube-api-access-vv2ft\") pod \"347f02b0-79ec-493b-8e06-bae4d0a7d62d\" (UID: \"347f02b0-79ec-493b-8e06-bae4d0a7d62d\") "
Nov 03 18:31:23.865872 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:23.865847    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/347f02b0-79ec-493b-8e06-bae4d0a7d62d-maas-api-openshift-ca" pod "347f02b0-79ec-493b-8e06-bae4d0a7d62d" (UID: "347f02b0-79ec-493b-8e06-bae4d0a7d62d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:31:23.866021 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:23.866001    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347f02b0-79ec-493b-8e06-bae4d0a7d62d-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:31:23.867669 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:23.867630    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/347f02b0-79ec-493b-8e06-bae4d0a7d62d-maas-api-xks-ca" pod "347f02b0-79ec-493b-8e06-bae4d0a7d62d" (UID: "347f02b0-79ec-493b-8e06-bae4d0a7d62d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:31:23.867779 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:23.867707    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347f02b0-79ec-493b-8e06-bae4d0a7d62d-kube-api-access-vv2ft" pod "347f02b0-79ec-493b-8e06-bae4d0a7d62d" (UID: "347f02b0-79ec-493b-8e06-bae4d0a7d62d"). InnerVolumeSpecName "kube-api-access-vv2ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:31:23.967574 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:23.967529    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347f02b0-79ec-493b-8e06-bae4d0a7d62d-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:31:23.967574 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:23.967568    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv2ft\" (UniqueName: \"kubernetes.io/projected/347f02b0-79ec-493b-8e06-bae4d0a7d62d-kube-api-access-vv2ft\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:31:24.622034 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:24.622005    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lkxgq"
Nov 03 18:31:24.622210 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:24.622046    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lkxgq" event={"ID":"347f02b0-79ec-493b-8e06-bae4d0a7d62d","Type":"ContainerDied","Data":"1ba465418d001796fa76ec3ab48b2958e2f6e28ab5ca439e10b6212e4bedb47e"}
Nov 03 18:31:24.622210 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:24.622096    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ba465418d001796fa76ec3ab48b2958e2f6e28ab5ca439e10b6212e4bedb47e"
Nov 03 18:31:32.058315 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:32.058273    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg"]
Nov 03 18:31:32.109470 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:32.109423    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d2fd950-9bkhg"]
Nov 03 18:31:33.335212 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:33.335176    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a59cf8bf-640b-4a56-8f65-6e1ebf380caf" path="/var/lib/kubelet/pods/a59cf8bf-640b-4a56-8f65-6e1ebf380caf/volumes"
Nov 03 18:31:35.899905 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:35.899860    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"]
Nov 03 18:31:35.905321 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:35.905289    6059 cpu_manager.go:414] "containerMap: removing container" podUID="347f02b0-79ec-493b-8e06-bae4d0a7d62d" containerName="revoke-keys"
Nov 03 18:31:35.905552 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:35.905534    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="347f02b0-79ec-493b-8e06-bae4d0a7d62d" containerName="revoke-keys"
Nov 03 18:31:35.905783 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:35.905768    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="347f02b0-79ec-493b-8e06-bae4d0a7d62d" containerName="revoke-keys"
Nov 03 18:31:35.914394 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:35.913640    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:35.923331 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:35.923297    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wsq7\" (UniqueName: \"kubernetes.io/projected/df5c783c-603a-44df-a44b-3de9f805a79c-kube-api-access-7wsq7\") pod \"e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") " pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:35.923607 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:35.923584    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-model-cache\") pod \"e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") " pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:35.923815 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:35.923797    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-dshm\") pod \"e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") " pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:35.923976 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:35.923960    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-home\") pod \"e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") " pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:35.924129 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:35.924113    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-tmp-dir\") pod \"e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") " pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:35.924239 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:35.924226    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5c783c-603a-44df-a44b-3de9f805a79c-tls-certs\") pod \"e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") " pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:35.997384 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:35.962201    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"]
Nov 03 18:31:36.020860 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:36.020826    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-0c882e6f-kserve-self-signed-certs\""
Nov 03 18:31:36.042517 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:36.042481    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:36.055382 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:36.052004    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-model-cache\") pod \"e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") " pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:36.061582 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:36.061549    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-dshm\") pod \"e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") " pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:36.062329 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:36.062300    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-home\") pod \"e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") " pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:36.062902 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:36.062880    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-tmp-dir\") pod \"e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") " pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:36.063066 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:36.063048    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5c783c-603a-44df-a44b-3de9f805a79c-tls-certs\") pod \"e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") " pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:36.063204 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:36.063184    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wsq7\" (UniqueName: \"kubernetes.io/projected/df5c783c-603a-44df-a44b-3de9f805a79c-kube-api-access-7wsq7\") pod \"e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") " pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:36.063717 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:36.061385    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-model-cache\") pod \"e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") " pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:36.088581 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:36.064087    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-tmp-dir\") pod \"e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") " pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:36.189144 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:36.062774    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-home\") pod \"e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") " pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:36.200258 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:36.100961    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-dshm\") pod \"e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") " pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:36.209127 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:36.209089    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5c783c-603a-44df-a44b-3de9f805a79c-tls-certs\") pod \"e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") " pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:36.218144 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:36.218105    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wsq7\" (UniqueName: \"kubernetes.io/projected/df5c783c-603a-44df-a44b-3de9f805a79c-kube-api-access-7wsq7\") pod \"e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") " pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:36.293203 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:36.293088    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:36.669486 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:36.669429    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:37.012135 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:31:37.012095    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:31:37.147776 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:37.147744    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"]
Nov 03 18:31:37.176588 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:37.176476    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-l2xpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:31:37.176817 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:37.176549    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" podUID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:31:37.273697 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:37.273487    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gxxkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:31:37.273697 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:37.273557    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" podUID="2159e743-a61b-4d76-9f78-9be36e825a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:31:37.773378 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:37.773073    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg" event={"ID":"df5c783c-603a-44df-a44b-3de9f805a79c","Type":"ContainerStarted","Data":"6e07d0249349a401a23ebca116447e27201cbf0af57efa196d7b6c984974244d"}
Nov 03 18:31:37.773378 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:37.773135    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg" event={"ID":"df5c783c-603a-44df-a44b-3de9f805a79c","Type":"ContainerStarted","Data":"9027d5ea7ba5d25a1181330cecf67cc7cf887effb3851b870007d00205c1dd7b"}
Nov 03 18:31:37.899867 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:37.899805    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg" podStartSLOduration=2.89979106 podStartE2EDuration="2.89979106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:31:37.89782597 +0000 UTC m=+4438.695538751" watchObservedRunningTime="2026-11-03 18:31:37.89979106 +0000 UTC m=+4438.697503840"
Nov 03 18:31:38.294051 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:38.294011    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:40.723504 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:40.723463    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:31:43.069636 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:43.069592    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9604928-z2z5r"]
Nov 03 18:31:43.118906 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:43.118861    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9604928-z2z5r"]
Nov 03 18:31:43.292333 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:43.292294    6059 scope.go:122] "RemoveContainer" containerID="6a9a33108b200a2706ab6784ba38c9a6495540457c7c0490983e86bccae267c7"
Nov 03 18:31:43.524529 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:43.524482    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e84c1e6-3809-4ce6-9430-7c7ec1f26a33" path="/var/lib/kubelet/pods/4e84c1e6-3809-4ce6-9430-7c7ec1f26a33/volumes"
Nov 03 18:31:46.293723 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:46.293687    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:46.306418 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:46.306389    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:46.871751 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:46.871715    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:31:55.923996 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:31:55.923951    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:06.862263 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:32:06.862223    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:32:36.862212 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:32:36.862166    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:32:37.402131 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:37.402085    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t"]
Nov 03 18:32:37.405921 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:37.405903    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t"
Nov 03 18:32:37.466005 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:37.465972    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t"]
Nov 03 18:32:37.484655 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:37.484604    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:32:37.484655 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:37.484604    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:32:37.484655 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:37.484604    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:32:37.514704 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:37.514665    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:32:37.520805 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:37.520770    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkwtn\" (UniqueName: \"kubernetes.io/projected/be62999d-979e-481c-8e93-932f8a9415bc-kube-api-access-vkwtn\") pod \"maas-api-revoke-sub-846b092f-46j6t\" (UID: \"be62999d-979e-481c-8e93-932f8a9415bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t"
Nov 03 18:32:37.520967 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:37.520846    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be62999d-979e-481c-8e93-932f8a9415bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-846b092f-46j6t\" (UID: \"be62999d-979e-481c-8e93-932f8a9415bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t"
Nov 03 18:32:37.521010 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:37.520956    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be62999d-979e-481c-8e93-932f8a9415bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-846b092f-46j6t\" (UID: \"be62999d-979e-481c-8e93-932f8a9415bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t"
Nov 03 18:32:37.621746 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:37.621708    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be62999d-979e-481c-8e93-932f8a9415bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-846b092f-46j6t\" (UID: \"be62999d-979e-481c-8e93-932f8a9415bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t"
Nov 03 18:32:37.621938 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:37.621791    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkwtn\" (UniqueName: \"kubernetes.io/projected/be62999d-979e-481c-8e93-932f8a9415bc-kube-api-access-vkwtn\") pod \"maas-api-revoke-sub-846b092f-46j6t\" (UID: \"be62999d-979e-481c-8e93-932f8a9415bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t"
Nov 03 18:32:37.621938 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:37.621877    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be62999d-979e-481c-8e93-932f8a9415bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-846b092f-46j6t\" (UID: \"be62999d-979e-481c-8e93-932f8a9415bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t"
Nov 03 18:32:37.622451 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:37.622421    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be62999d-979e-481c-8e93-932f8a9415bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-846b092f-46j6t\" (UID: \"be62999d-979e-481c-8e93-932f8a9415bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t"
Nov 03 18:32:37.624454 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:37.624424    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be62999d-979e-481c-8e93-932f8a9415bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-846b092f-46j6t\" (UID: \"be62999d-979e-481c-8e93-932f8a9415bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t"
Nov 03 18:32:37.685594 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:37.685521    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkwtn\" (UniqueName: \"kubernetes.io/projected/be62999d-979e-481c-8e93-932f8a9415bc-kube-api-access-vkwtn\") pod \"maas-api-revoke-sub-846b092f-46j6t\" (UID: \"be62999d-979e-481c-8e93-932f8a9415bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t"
Nov 03 18:32:37.719511 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:37.719477    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t"
Nov 03 18:32:38.093176 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:32:38.093134    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe62999d_979e_481c_8e93_932f8a9415bc.slice/crio-efabd0e53c15cd86d817dd09b32d828b14496b420306f84e1434c323d9fe039e WatchSource:0}: Error finding container efabd0e53c15cd86d817dd09b32d828b14496b420306f84e1434c323d9fe039e: Status 404 returned error can't find the container with id efabd0e53c15cd86d817dd09b32d828b14496b420306f84e1434c323d9fe039e
Nov 03 18:32:38.095333 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:38.095308    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t"]
Nov 03 18:32:38.354699 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:38.354661    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t" event={"ID":"be62999d-979e-481c-8e93-932f8a9415bc","Type":"ContainerStarted","Data":"b7c693ae19186ce83a4cbcf026b751d147582125fce50eee083e7f91ddc03cd4"}
Nov 03 18:32:38.354849 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:38.354729    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t" event={"ID":"be62999d-979e-481c-8e93-932f8a9415bc","Type":"ContainerStarted","Data":"efabd0e53c15cd86d817dd09b32d828b14496b420306f84e1434c323d9fe039e"}
Nov 03 18:32:38.483794 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:38.483736    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t" podStartSLOduration=1.483719051 podStartE2EDuration="1.483719051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:32:38.482278079 +0000 UTC m=+4499.279990860" watchObservedRunningTime="2026-11-03 18:32:38.483719051 +0000 UTC m=+4499.281431829"
Nov 03 18:32:39.362945 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:39.362904    6059 generic.go:356] "Generic (PLEG): container finished" podID="be62999d-979e-481c-8e93-932f8a9415bc" containerID="b7c693ae19186ce83a4cbcf026b751d147582125fce50eee083e7f91ddc03cd4" exitCode=0
Nov 03 18:32:39.363449 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:39.362986    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t" event={"ID":"be62999d-979e-481c-8e93-932f8a9415bc","Type":"ContainerDied","Data":"b7c693ae19186ce83a4cbcf026b751d147582125fce50eee083e7f91ddc03cd4"}
Nov 03 18:32:40.025015 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:40.024986    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:32:40.044551 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:40.044519    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:32:40.049112 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:40.049089    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:32:40.066085 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:40.066058    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:32:40.521044 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:40.521018    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t"
Nov 03 18:32:40.656180 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:40.656098    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be62999d-979e-481c-8e93-932f8a9415bc-kube-api-access-vkwtn\" (UniqueName: \"kubernetes.io/projected/be62999d-979e-481c-8e93-932f8a9415bc-kube-api-access-vkwtn\") pod \"be62999d-979e-481c-8e93-932f8a9415bc\" (UID: \"be62999d-979e-481c-8e93-932f8a9415bc\") "
Nov 03 18:32:40.656344 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:40.656251    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be62999d-979e-481c-8e93-932f8a9415bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be62999d-979e-481c-8e93-932f8a9415bc-maas-api-openshift-ca\") pod \"be62999d-979e-481c-8e93-932f8a9415bc\" (UID: \"be62999d-979e-481c-8e93-932f8a9415bc\") "
Nov 03 18:32:40.656344 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:40.656323    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be62999d-979e-481c-8e93-932f8a9415bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be62999d-979e-481c-8e93-932f8a9415bc-maas-api-xks-ca\") pod \"be62999d-979e-481c-8e93-932f8a9415bc\" (UID: \"be62999d-979e-481c-8e93-932f8a9415bc\") "
Nov 03 18:32:40.656742 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:40.656695    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be62999d-979e-481c-8e93-932f8a9415bc-maas-api-openshift-ca" pod "be62999d-979e-481c-8e93-932f8a9415bc" (UID: "be62999d-979e-481c-8e93-932f8a9415bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:32:40.658878 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:40.658850    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be62999d-979e-481c-8e93-932f8a9415bc-maas-api-xks-ca" pod "be62999d-979e-481c-8e93-932f8a9415bc" (UID: "be62999d-979e-481c-8e93-932f8a9415bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:32:40.659491 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:40.659470    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be62999d-979e-481c-8e93-932f8a9415bc-kube-api-access-vkwtn" pod "be62999d-979e-481c-8e93-932f8a9415bc" (UID: "be62999d-979e-481c-8e93-932f8a9415bc"). InnerVolumeSpecName "kube-api-access-vkwtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:32:40.757490 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:40.757443    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be62999d-979e-481c-8e93-932f8a9415bc-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:32:40.757490 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:40.757482    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be62999d-979e-481c-8e93-932f8a9415bc-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:32:40.757490 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:40.757497    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkwtn\" (UniqueName: \"kubernetes.io/projected/be62999d-979e-481c-8e93-932f8a9415bc-kube-api-access-vkwtn\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:32:41.378037 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:41.378004    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t" event={"ID":"be62999d-979e-481c-8e93-932f8a9415bc","Type":"ContainerDied","Data":"efabd0e53c15cd86d817dd09b32d828b14496b420306f84e1434c323d9fe039e"}
Nov 03 18:32:41.378037 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:41.378038    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t"
Nov 03 18:32:41.378252 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:41.378052    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efabd0e53c15cd86d817dd09b32d828b14496b420306f84e1434c323d9fe039e"
Nov 03 18:32:43.593259 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:43.593223    6059 scope.go:122] "RemoveContainer" containerID="e2aa24d2a4f356463721f29eb6e99d7a015fff47e9fc4a5e1fb0fce0b764b3f5"
Nov 03 18:32:43.602926 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:43.602903    6059 scope.go:122] "RemoveContainer" containerID="9c7c5804008f716ca0833f3afd72e0f1f3c525b7672020ffa6fd78870dbd71c7"
Nov 03 18:32:43.611551 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:43.611529    6059 scope.go:122] "RemoveContainer" containerID="759ab4504c17b7357802114eb867f975ed4085f32c345c1f74e0afd25492e9be"
Nov 03 18:32:43.620155 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:43.620131    6059 scope.go:122] "RemoveContainer" containerID="d302e64b466b74d1c421ef23f671e9d4f0b29ca8c5a3d9e0fad74ad64bbab936"
Nov 03 18:32:43.628941 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:43.628920    6059 scope.go:122] "RemoveContainer" containerID="fbe1d07f4538df91be61d3ddb83baca0c51e6abb28823ed8838ecc6a2c1848cf"
Nov 03 18:32:50.479985 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:50.479946    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"]
Nov 03 18:32:50.498848 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:50.498807    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:50.674933 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:50.674891    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:56.087002 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:56.086964    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz"]
Nov 03 18:32:56.087584 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:56.087565    6059 cpu_manager.go:414] "containerMap: removing container" podUID="be62999d-979e-481c-8e93-932f8a9415bc" containerName="revoke-keys"
Nov 03 18:32:56.087584 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:56.087586    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be62999d-979e-481c-8e93-932f8a9415bc" containerName="revoke-keys"
Nov 03 18:32:56.087709 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:56.087704    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="be62999d-979e-481c-8e93-932f8a9415bc" containerName="revoke-keys"
Nov 03 18:32:56.091019 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:56.090991    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz"
Nov 03 18:32:56.148126 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:56.148084    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:32:56.148321 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:56.148270    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:32:56.148429 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:56.148376    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:32:56.150168 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:56.150139    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz"]
Nov 03 18:32:56.198701 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:56.198665    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:32:56.229431 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:56.229390    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37108181-1bdb-48e1-a772-105a6e2d947b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-225ff5a4-x5pvz\" (UID: \"37108181-1bdb-48e1-a772-105a6e2d947b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz"
Nov 03 18:32:56.229647 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:56.229445    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37108181-1bdb-48e1-a772-105a6e2d947b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-225ff5a4-x5pvz\" (UID: \"37108181-1bdb-48e1-a772-105a6e2d947b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz"
Nov 03 18:32:56.229647 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:56.229584    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvwl5\" (UniqueName: \"kubernetes.io/projected/37108181-1bdb-48e1-a772-105a6e2d947b-kube-api-access-pvwl5\") pod \"maas-api-revoke-sub-225ff5a4-x5pvz\" (UID: \"37108181-1bdb-48e1-a772-105a6e2d947b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz"
Nov 03 18:32:56.264880 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:56.264845    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:56.330419 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:56.330349    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37108181-1bdb-48e1-a772-105a6e2d947b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-225ff5a4-x5pvz\" (UID: \"37108181-1bdb-48e1-a772-105a6e2d947b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz"
Nov 03 18:32:56.330616 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:56.330431    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37108181-1bdb-48e1-a772-105a6e2d947b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-225ff5a4-x5pvz\" (UID: \"37108181-1bdb-48e1-a772-105a6e2d947b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz"
Nov 03 18:32:56.330616 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:56.330482    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvwl5\" (UniqueName: \"kubernetes.io/projected/37108181-1bdb-48e1-a772-105a6e2d947b-kube-api-access-pvwl5\") pod \"maas-api-revoke-sub-225ff5a4-x5pvz\" (UID: \"37108181-1bdb-48e1-a772-105a6e2d947b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz"
Nov 03 18:32:56.331136 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:56.331114    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37108181-1bdb-48e1-a772-105a6e2d947b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-225ff5a4-x5pvz\" (UID: \"37108181-1bdb-48e1-a772-105a6e2d947b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz"
Nov 03 18:32:56.332807 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:56.332785    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37108181-1bdb-48e1-a772-105a6e2d947b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-225ff5a4-x5pvz\" (UID: \"37108181-1bdb-48e1-a772-105a6e2d947b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz"
Nov 03 18:32:56.393577 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:56.393499    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvwl5\" (UniqueName: \"kubernetes.io/projected/37108181-1bdb-48e1-a772-105a6e2d947b-kube-api-access-pvwl5\") pod \"maas-api-revoke-sub-225ff5a4-x5pvz\" (UID: \"37108181-1bdb-48e1-a772-105a6e2d947b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz"
Nov 03 18:32:56.404343 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:56.404315    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz"
Nov 03 18:32:56.780082 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:56.779984    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz"]
Nov 03 18:32:57.484778 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:57.483759    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz" event={"ID":"37108181-1bdb-48e1-a772-105a6e2d947b","Type":"ContainerStarted","Data":"c64e231a9fcc61843446270890ba703b438f5cd6706082df61f87847529ade05"}
Nov 03 18:32:57.484778 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:57.483832    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz" event={"ID":"37108181-1bdb-48e1-a772-105a6e2d947b","Type":"ContainerStarted","Data":"12075172c113c6f7c731c5116ee4f7d63da70a68966c870fbd82812ade84ab26"}
Nov 03 18:32:57.606508 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:57.606426    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz" podStartSLOduration=1.606405463 podStartE2EDuration="1.606405463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:32:57.605275112 +0000 UTC m=+4518.402987894" watchObservedRunningTime="2026-11-03 18:32:57.606405463 +0000 UTC m=+4518.404118248"
Nov 03 18:32:58.188438 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:58.188257    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:32:58.522896 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:58.522857    6059 generic.go:356] "Generic (PLEG): container finished" podID="37108181-1bdb-48e1-a772-105a6e2d947b" containerID="c64e231a9fcc61843446270890ba703b438f5cd6706082df61f87847529ade05" exitCode=0
Nov 03 18:32:58.525131 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:58.524889    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz" event={"ID":"37108181-1bdb-48e1-a772-105a6e2d947b","Type":"ContainerDied","Data":"c64e231a9fcc61843446270890ba703b438f5cd6706082df61f87847529ade05"}
Nov 03 18:32:59.228649 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:32:59.228606    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:00.730568 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:00.730458    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz" event={"ID":"37108181-1bdb-48e1-a772-105a6e2d947b","Type":"ContainerDied","Data":"12075172c113c6f7c731c5116ee4f7d63da70a68966c870fbd82812ade84ab26"}
Nov 03 18:33:00.730568 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:00.730531    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12075172c113c6f7c731c5116ee4f7d63da70a68966c870fbd82812ade84ab26"
Nov 03 18:33:00.731595 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:00.731571    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz"
Nov 03 18:33:00.892440 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:00.879937    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37108181-1bdb-48e1-a772-105a6e2d947b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37108181-1bdb-48e1-a772-105a6e2d947b-maas-api-xks-ca\") pod \"37108181-1bdb-48e1-a772-105a6e2d947b\" (UID: \"37108181-1bdb-48e1-a772-105a6e2d947b\") "
Nov 03 18:33:00.892440 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:00.880222    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37108181-1bdb-48e1-a772-105a6e2d947b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37108181-1bdb-48e1-a772-105a6e2d947b-maas-api-openshift-ca\") pod \"37108181-1bdb-48e1-a772-105a6e2d947b\" (UID: \"37108181-1bdb-48e1-a772-105a6e2d947b\") "
Nov 03 18:33:00.892440 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:00.880435    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37108181-1bdb-48e1-a772-105a6e2d947b-kube-api-access-pvwl5\" (UniqueName: \"kubernetes.io/projected/37108181-1bdb-48e1-a772-105a6e2d947b-kube-api-access-pvwl5\") pod \"37108181-1bdb-48e1-a772-105a6e2d947b\" (UID: \"37108181-1bdb-48e1-a772-105a6e2d947b\") "
Nov 03 18:33:00.897300 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:00.896410    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37108181-1bdb-48e1-a772-105a6e2d947b-maas-api-openshift-ca" pod "37108181-1bdb-48e1-a772-105a6e2d947b" (UID: "37108181-1bdb-48e1-a772-105a6e2d947b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:33:00.988485 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:00.988432    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37108181-1bdb-48e1-a772-105a6e2d947b-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:00.993586 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:00.993536    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37108181-1bdb-48e1-a772-105a6e2d947b-kube-api-access-pvwl5" pod "37108181-1bdb-48e1-a772-105a6e2d947b" (UID: "37108181-1bdb-48e1-a772-105a6e2d947b"). InnerVolumeSpecName "kube-api-access-pvwl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:33:01.075380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:01.073499    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37108181-1bdb-48e1-a772-105a6e2d947b-maas-api-xks-ca" pod "37108181-1bdb-48e1-a772-105a6e2d947b" (UID: "37108181-1bdb-48e1-a772-105a6e2d947b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:33:01.092259 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:01.092224    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37108181-1bdb-48e1-a772-105a6e2d947b-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:01.092529 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:01.092507    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvwl5\" (UniqueName: \"kubernetes.io/projected/37108181-1bdb-48e1-a772-105a6e2d947b-kube-api-access-pvwl5\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:01.735929 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:01.735890    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz"
Nov 03 18:33:05.501178 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.501127    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg" podUID="df5c783c-603a-44df-a44b-3de9f805a79c" containerName="main" containerID="cri-o://6e07d0249349a401a23ebca116447e27201cbf0af57efa196d7b6c984974244d" gracePeriod=45
Nov 03 18:33:05.757773 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.757688    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:33:05.759440 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.759402    6059 generic.go:356] "Generic (PLEG): container finished" podID="df5c783c-603a-44df-a44b-3de9f805a79c" containerID="6e07d0249349a401a23ebca116447e27201cbf0af57efa196d7b6c984974244d" exitCode=0
Nov 03 18:33:05.759575 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.759469    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg" event={"ID":"df5c783c-603a-44df-a44b-3de9f805a79c","Type":"ContainerDied","Data":"6e07d0249349a401a23ebca116447e27201cbf0af57efa196d7b6c984974244d"}
Nov 03 18:33:05.759575 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.759485    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"
Nov 03 18:33:05.759575 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.759498    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg" event={"ID":"df5c783c-603a-44df-a44b-3de9f805a79c","Type":"ContainerDied","Data":"9027d5ea7ba5d25a1181330cecf67cc7cf887effb3851b870007d00205c1dd7b"}
Nov 03 18:33:05.759575 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.759514    6059 scope.go:122] "RemoveContainer" containerID="6e07d0249349a401a23ebca116447e27201cbf0af57efa196d7b6c984974244d"
Nov 03 18:33:05.769148 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.769122    6059 scope.go:122] "RemoveContainer" containerID="6e07d0249349a401a23ebca116447e27201cbf0af57efa196d7b6c984974244d"
Nov 03 18:33:05.769518 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:33:05.769491    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e07d0249349a401a23ebca116447e27201cbf0af57efa196d7b6c984974244d\": container with ID starting with 6e07d0249349a401a23ebca116447e27201cbf0af57efa196d7b6c984974244d not found: ID does not exist" containerID="6e07d0249349a401a23ebca116447e27201cbf0af57efa196d7b6c984974244d"
Nov 03 18:33:05.769609 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.769531    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e07d0249349a401a23ebca116447e27201cbf0af57efa196d7b6c984974244d"} err="failed to get container status \"6e07d0249349a401a23ebca116447e27201cbf0af57efa196d7b6c984974244d\": rpc error: code = NotFound desc = could not find container \"6e07d0249349a401a23ebca116447e27201cbf0af57efa196d7b6c984974244d\": container with ID starting with 6e07d0249349a401a23ebca116447e27201cbf0af57efa196d7b6c984974244d not found: ID does not exist"
Nov 03 18:33:05.842593 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.842554    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df5c783c-603a-44df-a44b-3de9f805a79c-kube-api-access-7wsq7\" (UniqueName: \"kubernetes.io/projected/df5c783c-603a-44df-a44b-3de9f805a79c-kube-api-access-7wsq7\") pod \"df5c783c-603a-44df-a44b-3de9f805a79c\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") "
Nov 03 18:33:05.842805 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.842605    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-dshm\") pod \"df5c783c-603a-44df-a44b-3de9f805a79c\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") "
Nov 03 18:33:05.842805 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.842674    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-tmp-dir\") pod \"df5c783c-603a-44df-a44b-3de9f805a79c\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") "
Nov 03 18:33:05.842805 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.842707    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-home\" (UniqueName: \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-home\") pod \"df5c783c-603a-44df-a44b-3de9f805a79c\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") "
Nov 03 18:33:05.842805 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.842794    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-model-cache\") pod \"df5c783c-603a-44df-a44b-3de9f805a79c\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") "
Nov 03 18:33:05.843064 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.842830    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df5c783c-603a-44df-a44b-3de9f805a79c-tls-certs\" (UniqueName: \"kubernetes.io/secret/df5c783c-603a-44df-a44b-3de9f805a79c-tls-certs\") pod \"df5c783c-603a-44df-a44b-3de9f805a79c\" (UID: \"df5c783c-603a-44df-a44b-3de9f805a79c\") "
Nov 03 18:33:05.843064 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.842954    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-tmp-dir" pod "df5c783c-603a-44df-a44b-3de9f805a79c" (UID: "df5c783c-603a-44df-a44b-3de9f805a79c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:33:05.843064 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.843042    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-home" pod "df5c783c-603a-44df-a44b-3de9f805a79c" (UID: "df5c783c-603a-44df-a44b-3de9f805a79c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:33:05.843233 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.843202    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-model-cache" pod "df5c783c-603a-44df-a44b-3de9f805a79c" (UID: "df5c783c-603a-44df-a44b-3de9f805a79c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:33:05.843327 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.843307    6059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-home\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:05.843401 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.843331    6059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-model-cache\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:05.843401 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.843347    6059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-tmp-dir\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:05.845027 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.845000    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df5c783c-603a-44df-a44b-3de9f805a79c-tls-certs" pod "df5c783c-603a-44df-a44b-3de9f805a79c" (UID: "df5c783c-603a-44df-a44b-3de9f805a79c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:33:05.845027 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.845006    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-dshm" pod "df5c783c-603a-44df-a44b-3de9f805a79c" (UID: "df5c783c-603a-44df-a44b-3de9f805a79c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:33:05.845254 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.845002    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5c783c-603a-44df-a44b-3de9f805a79c-kube-api-access-7wsq7" pod "df5c783c-603a-44df-a44b-3de9f805a79c" (UID: "df5c783c-603a-44df-a44b-3de9f805a79c"). InnerVolumeSpecName "kube-api-access-7wsq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:33:05.944347 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.944307    6059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5c783c-603a-44df-a44b-3de9f805a79c-tls-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:05.944347 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.944343    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wsq7\" (UniqueName: \"kubernetes.io/projected/df5c783c-603a-44df-a44b-3de9f805a79c-kube-api-access-7wsq7\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:05.944347 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:05.944387    6059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df5c783c-603a-44df-a44b-3de9f805a79c-dshm\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:06.183838 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:06.183805    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"]
Nov 03 18:33:06.188005 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:06.187977    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-0c882e6f-kserve-59c69c5d9f-8fqfg"]
Nov 03 18:33:06.861723 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:33:06.861685    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:33:07.335549 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:07.335512    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df5c783c-603a-44df-a44b-3de9f805a79c" path="/var/lib/kubelet/pods/df5c783c-603a-44df-a44b-3de9f805a79c/volumes"
Nov 03 18:33:08.999808 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:08.999776    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:09.535158 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.535120    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"]
Nov 03 18:33:09.535590 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.535575    6059 cpu_manager.go:414] "containerMap: removing container" podUID="37108181-1bdb-48e1-a772-105a6e2d947b" containerName="revoke-keys"
Nov 03 18:33:09.535659 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.535591    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37108181-1bdb-48e1-a772-105a6e2d947b" containerName="revoke-keys"
Nov 03 18:33:09.535659 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.535609    6059 cpu_manager.go:414] "containerMap: removing container" podUID="df5c783c-603a-44df-a44b-3de9f805a79c" containerName="main"
Nov 03 18:33:09.535659 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.535615    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df5c783c-603a-44df-a44b-3de9f805a79c" containerName="main"
Nov 03 18:33:09.535783 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.535729    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="37108181-1bdb-48e1-a772-105a6e2d947b" containerName="revoke-keys"
Nov 03 18:33:09.535783 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.535739    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="df5c783c-603a-44df-a44b-3de9f805a79c" containerName="main"
Nov 03 18:33:09.540504 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.540478    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.589798 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.582803    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-workload-socket\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.589798 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.582874    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-workload-certs\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.589798 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.582933    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-credential-socket\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.589798 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.582983    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-data\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.589798 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.583020    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-token\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.589798 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.583075    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-envoy\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.589798 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.583123    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxm7x\" (UniqueName: \"kubernetes.io/projected/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-kube-api-access-vxm7x\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.589798 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.583170    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-podinfo\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.589798 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.583214    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istiod-ca-cert\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.601958 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.601928    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-57d07b-openshift-default-dockercfg-rj8sz\""
Nov 03 18:33:09.656813 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.656780    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"]
Nov 03 18:33:09.659340 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.659309    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:09.684220 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.684170    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-credential-socket\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.684417 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.684255    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-data\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.684417 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.684302    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-token\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.684417 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.684383    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-envoy\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.684642 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.684424    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxm7x\" (UniqueName: \"kubernetes.io/projected/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-kube-api-access-vxm7x\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.684642 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.684475    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-podinfo\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.684642 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.684516    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istiod-ca-cert\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.684642 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.684611    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-workload-socket\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.684895 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.684653    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-credential-socket\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.684895 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.684656    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-workload-certs\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.684895 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.684697    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-data\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.685066 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.684987    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-workload-socket\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.685066 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.685036    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-workload-certs\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.685368 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.685326    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istiod-ca-cert\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.686986 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.686960    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-podinfo\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.687101 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.686961    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-envoy\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.747653 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.747624    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-token\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.747922 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.747895    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxm7x\" (UniqueName: \"kubernetes.io/projected/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-kube-api-access-vxm7x\") pod \"e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") " pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.853797 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.853681    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:09.943442 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:09.943318    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:10.666375 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:10.662492    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"]
Nov 03 18:33:10.708414 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:33:10.708377    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f06ef8_c5b8_4a2b_a24e_35c508ba5b7d.slice/crio-d5ed04a82f315941aeed411a7a860cf8b78c4524a8af2b0700044c883bbfe4ca WatchSource:0}: Error finding container d5ed04a82f315941aeed411a7a860cf8b78c4524a8af2b0700044c883bbfe4ca: Status 404 returned error can't find the container with id d5ed04a82f315941aeed411a7a860cf8b78c4524a8af2b0700044c883bbfe4ca
Nov 03 18:33:10.724693 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:10.723579    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:33:10.724693 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:10.723671    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:33:10.724693 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:10.723731    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:33:10.873535 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:10.871242    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l" event={"ID":"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d","Type":"ContainerStarted","Data":"d5ed04a82f315941aeed411a7a860cf8b78c4524a8af2b0700044c883bbfe4ca"}
Nov 03 18:33:11.918380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:11.918048    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l" event={"ID":"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d","Type":"ContainerStarted","Data":"b8a70d633547bf97da5b635b0c452038cc103abb1286b87a37ea0e67f6b666e3"}
Nov 03 18:33:12.167381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:12.166437    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l" podStartSLOduration=3.166416713 podStartE2EDuration="3.166416713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:33:12.108125818 +0000 UTC m=+4532.905838592" watchObservedRunningTime="2026-11-03 18:33:12.166416713 +0000 UTC m=+4532.964129498"
Nov 03 18:33:12.854328 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:12.854281    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:12.861838 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:12.861801    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:12.923534 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:12.923501    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:12.924695 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:12.924673    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:15.211005 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:15.210964    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:15.391192 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:15.391160    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:15.564763 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:15.564679    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:17.698386 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:17.698320    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:18.428564 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:18.428531    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:18.623461 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:18.623426    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:18.814647 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:18.814609    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp"]
Nov 03 18:33:18.818495 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:18.818476    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp"
Nov 03 18:33:18.877540 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:18.877507    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-57d07b\""
Nov 03 18:33:18.882431 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:18.882396    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp"]
Nov 03 18:33:18.929077 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:18.929046    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-8a437858-dockercfg-wf85p\""
Nov 03 18:33:18.974443 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:18.974395    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d5238f-d75d-496f-92f0-19e4c640c23f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp\" (UID: \"61d5238f-d75d-496f-92f0-19e4c640c23f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp"
Nov 03 18:33:18.974685 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:18.974654    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hhch\" (UniqueName: \"kubernetes.io/projected/61d5238f-d75d-496f-92f0-19e4c640c23f-kube-api-access-9hhch\") pod \"payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp\" (UID: \"61d5238f-d75d-496f-92f0-19e4c640c23f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp"
Nov 03 18:33:18.988530 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:18.988491    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v"]
Nov 03 18:33:18.992451 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:18.992418    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v"
Nov 03 18:33:19.056554 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:19.056517    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v"]
Nov 03 18:33:19.075914 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:19.075808    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc564e5-69e5-468e-9b03-a66d33ad95f3-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v\" (UID: \"5cc564e5-69e5-468e-9b03-a66d33ad95f3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v"
Nov 03 18:33:19.075914 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:19.075862    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hhch\" (UniqueName: \"kubernetes.io/projected/61d5238f-d75d-496f-92f0-19e4c640c23f-kube-api-access-9hhch\") pod \"payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp\" (UID: \"61d5238f-d75d-496f-92f0-19e4c640c23f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp"
Nov 03 18:33:19.075914 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:19.075886    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgghd\" (UniqueName: \"kubernetes.io/projected/5cc564e5-69e5-468e-9b03-a66d33ad95f3-kube-api-access-fgghd\") pod \"payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v\" (UID: \"5cc564e5-69e5-468e-9b03-a66d33ad95f3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v"
Nov 03 18:33:19.076235 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:19.076168    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d5238f-d75d-496f-92f0-19e4c640c23f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp\" (UID: \"61d5238f-d75d-496f-92f0-19e4c640c23f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp"
Nov 03 18:33:19.077155 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:19.077125    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d5238f-d75d-496f-92f0-19e4c640c23f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp\" (UID: \"61d5238f-d75d-496f-92f0-19e4c640c23f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp"
Nov 03 18:33:19.139071 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:19.139030    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hhch\" (UniqueName: \"kubernetes.io/projected/61d5238f-d75d-496f-92f0-19e4c640c23f-kube-api-access-9hhch\") pod \"payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp\" (UID: \"61d5238f-d75d-496f-92f0-19e4c640c23f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp"
Nov 03 18:33:19.177146 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:19.177102    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc564e5-69e5-468e-9b03-a66d33ad95f3-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v\" (UID: \"5cc564e5-69e5-468e-9b03-a66d33ad95f3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v"
Nov 03 18:33:19.177328 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:19.177156    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgghd\" (UniqueName: \"kubernetes.io/projected/5cc564e5-69e5-468e-9b03-a66d33ad95f3-kube-api-access-fgghd\") pod \"payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v\" (UID: \"5cc564e5-69e5-468e-9b03-a66d33ad95f3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v"
Nov 03 18:33:19.177903 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:19.177880    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc564e5-69e5-468e-9b03-a66d33ad95f3-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v\" (UID: \"5cc564e5-69e5-468e-9b03-a66d33ad95f3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v"
Nov 03 18:33:19.240268 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:19.240232    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgghd\" (UniqueName: \"kubernetes.io/projected/5cc564e5-69e5-468e-9b03-a66d33ad95f3-kube-api-access-fgghd\") pod \"payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v\" (UID: \"5cc564e5-69e5-468e-9b03-a66d33ad95f3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v"
Nov 03 18:33:19.308551 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:19.308508    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v"
Nov 03 18:33:19.431734 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:19.431700    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp"
Nov 03 18:33:19.684946 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:33:19.684908    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc564e5_69e5_468e_9b03_a66d33ad95f3.slice/crio-bc6769ec054160cb5712151ddaa80318bf0991a54090d5cf0ae30c7a17a2b22d WatchSource:0}: Error finding container bc6769ec054160cb5712151ddaa80318bf0991a54090d5cf0ae30c7a17a2b22d: Status 404 returned error can't find the container with id bc6769ec054160cb5712151ddaa80318bf0991a54090d5cf0ae30c7a17a2b22d
Nov 03 18:33:19.686382 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:19.686332    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v"]
Nov 03 18:33:19.805009 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:33:19.804974    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d5238f_d75d_496f_92f0_19e4c640c23f.slice/crio-4db027d2c0d1b6b3febca4565fa77443661644ee6724e125a4658a0730689198 WatchSource:0}: Error finding container 4db027d2c0d1b6b3febca4565fa77443661644ee6724e125a4658a0730689198: Status 404 returned error can't find the container with id 4db027d2c0d1b6b3febca4565fa77443661644ee6724e125a4658a0730689198
Nov 03 18:33:19.807180 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:19.807158    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp"]
Nov 03 18:33:19.963422 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:19.963387    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v" event={"ID":"5cc564e5-69e5-468e-9b03-a66d33ad95f3","Type":"ContainerStarted","Data":"84f7038c4d352a4d0c44845527d4e942fd10ec6dd1288d898a8b50e21883a480"}
Nov 03 18:33:19.963912 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:19.963440    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v" event={"ID":"5cc564e5-69e5-468e-9b03-a66d33ad95f3","Type":"ContainerStarted","Data":"bc6769ec054160cb5712151ddaa80318bf0991a54090d5cf0ae30c7a17a2b22d"}
Nov 03 18:33:19.963912 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:19.963545    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v"
Nov 03 18:33:19.965288 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:19.965259    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp" event={"ID":"61d5238f-d75d-496f-92f0-19e4c640c23f","Type":"ContainerStarted","Data":"b4bf9c63615afdd06b229bbea16c042e5c6bde2d7b38df5bc6a70b967edc0f6a"}
Nov 03 18:33:19.965432 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:19.965309    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp" event={"ID":"61d5238f-d75d-496f-92f0-19e4c640c23f","Type":"ContainerStarted","Data":"4db027d2c0d1b6b3febca4565fa77443661644ee6724e125a4658a0730689198"}
Nov 03 18:33:19.965432 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:19.965339    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp"
Nov 03 18:33:20.085623 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:20.085563    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v" podStartSLOduration=2.08554749 podStartE2EDuration="2.08554749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:33:20.083751377 +0000 UTC m=+4540.881464159" watchObservedRunningTime="2026-11-03 18:33:20.08554749 +0000 UTC m=+4540.883260270"
Nov 03 18:33:20.226304 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:20.226192    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp" podStartSLOduration=2.226176153 podStartE2EDuration="2.226176153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:33:20.223712053 +0000 UTC m=+4541.021424835" watchObservedRunningTime="2026-11-03 18:33:20.226176153 +0000 UTC m=+4541.023888933"
Nov 03 18:33:21.702318 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:21.702186    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:21.878586 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:21.878549    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:26.762149 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:26.762114    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:30.974024 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:30.973991    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp"
Nov 03 18:33:30.974549 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:30.974442    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v"
Nov 03 18:33:31.740382 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:31.729604    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:36.687157 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:36.687120    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:36.861759 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:33:36.861663    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:33:36.869197 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:36.869168    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:37.102631 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:37.102598    6059 generic.go:356] "Generic (PLEG): container finished" podID="a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" containerID="b8a70d633547bf97da5b635b0c452038cc103abb1286b87a37ea0e67f6b666e3" exitCode=0
Nov 03 18:33:37.102824 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:37.102662    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l" event={"ID":"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d","Type":"ContainerDied","Data":"b8a70d633547bf97da5b635b0c452038cc103abb1286b87a37ea0e67f6b666e3"}
Nov 03 18:33:37.103106 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:37.103088    6059 scope.go:122] "RemoveContainer" containerID="b8a70d633547bf97da5b635b0c452038cc103abb1286b87a37ea0e67f6b666e3"
Nov 03 18:33:37.104426 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:37.104404    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:33:37.104486 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:37.104467    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:33:37.104525 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:37.104501    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:33:38.111387 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:38.111316    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l" event={"ID":"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d","Type":"ContainerStarted","Data":"0b87d581773bce22e7bae477d15f530b951d6e6d307088d48c3b86910cba631c"}
Nov 03 18:33:38.854624 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:38.854586    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:38.855843 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:38.855815    6059 patch_prober.go:29] interesting pod/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Nov 03 18:33:38.856007 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:38.855862    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l" podUID="a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Nov 03 18:33:39.854865 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:39.854807    6059 patch_prober.go:29] interesting pod/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Nov 03 18:33:39.855265 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:39.854904    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l" podUID="a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Nov 03 18:33:39.974044 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:39.974009    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:40.588874 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:40.588836    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:40.769169 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:40.769130    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:40.854877 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:40.854792    6059 patch_prober.go:29] interesting pod/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Nov 03 18:33:40.855265 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:40.854878    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l" podUID="a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Nov 03 18:33:41.854803 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:41.854762    6059 patch_prober.go:29] interesting pod/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Nov 03 18:33:41.855036 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:41.854835    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l" podUID="a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Nov 03 18:33:42.112049 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:42.111954    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:42.854884 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:42.854836    6059 patch_prober.go:29] interesting pod/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Nov 03 18:33:42.855077 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:42.854929    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l" podUID="a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Nov 03 18:33:42.924217 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:42.924177    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:43.073951 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:43.073906    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:43.858790 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:43.858755    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:43.859869 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:43.859842    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:33:46.982957 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:46.982916    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:51.981920 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:51.981887    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:56.610343 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:56.610299    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:33:56.895135 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:33:56.895045    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:34:01.795262 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:01.795227    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:34:06.686747 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:06.686690    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:34:06.862488 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:34:06.862451    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:34:11.688347 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:11.688311    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:34:16.599642 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:16.599591    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:34:21.749159 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:21.749074    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:34:26.596177 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:26.596142    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:34:27.049828 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:27.049791    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-klgk6"]
Nov 03 18:34:27.101422 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:27.101387    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-klgk6"]
Nov 03 18:34:27.334975 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:27.334888    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="debab382-efb3-462d-a4f5-890c3410f5f4" path="/var/lib/kubelet/pods/debab382-efb3-462d-a4f5-890c3410f5f4/volumes"
Nov 03 18:34:28.949388 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:28.949338    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq"]
Nov 03 18:34:28.953410 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:28.953392    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq"
Nov 03 18:34:29.059601 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:29.059561    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:34:29.059795 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:29.059565    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:34:29.059795 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:29.059611    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:34:29.059795 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:29.059676    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:34:29.060424 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:29.060403    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396d6528-ddd5-4693-b335-8b88324034f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb79a368-m82mq\" (UID: \"396d6528-ddd5-4693-b335-8b88324034f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq"
Nov 03 18:34:29.060535 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:29.060521    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fftf6\" (UniqueName: \"kubernetes.io/projected/396d6528-ddd5-4693-b335-8b88324034f3-kube-api-access-fftf6\") pod \"maas-api-revoke-sub-bb79a368-m82mq\" (UID: \"396d6528-ddd5-4693-b335-8b88324034f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq"
Nov 03 18:34:29.060584 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:29.060552    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396d6528-ddd5-4693-b335-8b88324034f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb79a368-m82mq\" (UID: \"396d6528-ddd5-4693-b335-8b88324034f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq"
Nov 03 18:34:29.062030 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:29.062008    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq"]
Nov 03 18:34:29.161992 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:29.161956    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396d6528-ddd5-4693-b335-8b88324034f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb79a368-m82mq\" (UID: \"396d6528-ddd5-4693-b335-8b88324034f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq"
Nov 03 18:34:29.162187 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:29.162047    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fftf6\" (UniqueName: \"kubernetes.io/projected/396d6528-ddd5-4693-b335-8b88324034f3-kube-api-access-fftf6\") pod \"maas-api-revoke-sub-bb79a368-m82mq\" (UID: \"396d6528-ddd5-4693-b335-8b88324034f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq"
Nov 03 18:34:29.162187 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:29.162084    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396d6528-ddd5-4693-b335-8b88324034f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb79a368-m82mq\" (UID: \"396d6528-ddd5-4693-b335-8b88324034f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq"
Nov 03 18:34:29.162667 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:29.162645    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396d6528-ddd5-4693-b335-8b88324034f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb79a368-m82mq\" (UID: \"396d6528-ddd5-4693-b335-8b88324034f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq"
Nov 03 18:34:29.164613 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:29.164578    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396d6528-ddd5-4693-b335-8b88324034f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb79a368-m82mq\" (UID: \"396d6528-ddd5-4693-b335-8b88324034f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq"
Nov 03 18:34:29.225316 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:29.225280    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fftf6\" (UniqueName: \"kubernetes.io/projected/396d6528-ddd5-4693-b335-8b88324034f3-kube-api-access-fftf6\") pod \"maas-api-revoke-sub-bb79a368-m82mq\" (UID: \"396d6528-ddd5-4693-b335-8b88324034f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq"
Nov 03 18:34:29.266622 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:29.266586    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq"
Nov 03 18:34:29.640480 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:29.640317    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq"]
Nov 03 18:34:30.456749 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:30.456711    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq" event={"ID":"396d6528-ddd5-4693-b335-8b88324034f3","Type":"ContainerStarted","Data":"ffb24cc19d361bfcec7e1ce8d896894f08448a3418769c7a21a1678cbaf3c9f9"}
Nov 03 18:34:30.457194 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:30.456773    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq" event={"ID":"396d6528-ddd5-4693-b335-8b88324034f3","Type":"ContainerStarted","Data":"72c91c0389b29c1ce649126f6906523ee63fdb8222b1f7a833d6e969677ea922"}
Nov 03 18:34:30.577315 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:30.577254    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq" podStartSLOduration=2.577237912 podStartE2EDuration="2.577237912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:34:30.576102535 +0000 UTC m=+4611.373815316" watchObservedRunningTime="2026-11-03 18:34:30.577237912 +0000 UTC m=+4611.374950692"
Nov 03 18:34:31.464785 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:31.464751    6059 generic.go:356] "Generic (PLEG): container finished" podID="396d6528-ddd5-4693-b335-8b88324034f3" containerID="ffb24cc19d361bfcec7e1ce8d896894f08448a3418769c7a21a1678cbaf3c9f9" exitCode=0
Nov 03 18:34:31.465221 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:31.464802    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq" event={"ID":"396d6528-ddd5-4693-b335-8b88324034f3","Type":"ContainerDied","Data":"ffb24cc19d361bfcec7e1ce8d896894f08448a3418769c7a21a1678cbaf3c9f9"}
Nov 03 18:34:31.498799 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:31.498763    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:34:32.611943 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:32.611912    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq"
Nov 03 18:34:32.698069 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:32.698030    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/396d6528-ddd5-4693-b335-8b88324034f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396d6528-ddd5-4693-b335-8b88324034f3-maas-api-xks-ca\") pod \"396d6528-ddd5-4693-b335-8b88324034f3\" (UID: \"396d6528-ddd5-4693-b335-8b88324034f3\") "
Nov 03 18:34:32.698279 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:32.698108    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/396d6528-ddd5-4693-b335-8b88324034f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396d6528-ddd5-4693-b335-8b88324034f3-maas-api-openshift-ca\") pod \"396d6528-ddd5-4693-b335-8b88324034f3\" (UID: \"396d6528-ddd5-4693-b335-8b88324034f3\") "
Nov 03 18:34:32.698279 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:32.698170    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/396d6528-ddd5-4693-b335-8b88324034f3-kube-api-access-fftf6\" (UniqueName: \"kubernetes.io/projected/396d6528-ddd5-4693-b335-8b88324034f3-kube-api-access-fftf6\") pod \"396d6528-ddd5-4693-b335-8b88324034f3\" (UID: \"396d6528-ddd5-4693-b335-8b88324034f3\") "
Nov 03 18:34:32.698523 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:32.698498    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396d6528-ddd5-4693-b335-8b88324034f3-maas-api-openshift-ca" pod "396d6528-ddd5-4693-b335-8b88324034f3" (UID: "396d6528-ddd5-4693-b335-8b88324034f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:34:32.700249 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:32.700213    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/396d6528-ddd5-4693-b335-8b88324034f3-maas-api-xks-ca" pod "396d6528-ddd5-4693-b335-8b88324034f3" (UID: "396d6528-ddd5-4693-b335-8b88324034f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:34:32.700378 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:32.700305    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396d6528-ddd5-4693-b335-8b88324034f3-kube-api-access-fftf6" pod "396d6528-ddd5-4693-b335-8b88324034f3" (UID: "396d6528-ddd5-4693-b335-8b88324034f3"). InnerVolumeSpecName "kube-api-access-fftf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:34:32.799962 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:32.799877    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396d6528-ddd5-4693-b335-8b88324034f3-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:34:32.799962 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:32.799906    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396d6528-ddd5-4693-b335-8b88324034f3-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:34:32.799962 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:32.799918    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fftf6\" (UniqueName: \"kubernetes.io/projected/396d6528-ddd5-4693-b335-8b88324034f3-kube-api-access-fftf6\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:34:33.478977 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:33.478936    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq" event={"ID":"396d6528-ddd5-4693-b335-8b88324034f3","Type":"ContainerDied","Data":"72c91c0389b29c1ce649126f6906523ee63fdb8222b1f7a833d6e969677ea922"}
Nov 03 18:34:33.478977 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:33.478986    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72c91c0389b29c1ce649126f6906523ee63fdb8222b1f7a833d6e969677ea922"
Nov 03 18:34:33.479208 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:33.478952    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq"
Nov 03 18:34:36.387058 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:36.387021    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:34:36.862100 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:34:36.862064    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:34:40.123562 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:40.123521    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:34:40.589859 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:40.589822    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:34:41.290452 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:41.290417    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:34:43.682447 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:43.682414    6059 scope.go:122] "RemoveContainer" containerID="c447370a82eb0c6958b8ee1c2d806bb671b2bd3540bcd17ca48678b0ac94ce61"
Nov 03 18:34:45.155438 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:45.155403    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:34:45.651735 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:45.651686    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v"]
Nov 03 18:34:45.759991 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:45.759953    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp"]
Nov 03 18:34:46.025584 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:46.025550    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:34:46.203707 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:46.203663    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:34:46.380756 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:46.380649    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:34:46.742080 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:46.742044    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:34:50.652593 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:50.652547    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v" podUID="5cc564e5-69e5-468e-9b03-a66d33ad95f3" containerName="payload-processing" containerID="cri-o://84f7038c4d352a4d0c44845527d4e942fd10ec6dd1288d898a8b50e21883a480" gracePeriod=25
Nov 03 18:34:50.761084 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:50.761014    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp" podUID="61d5238f-d75d-496f-92f0-19e4c640c23f" containerName="payload-pre-processing" containerID="cri-o://b4bf9c63615afdd06b229bbea16c042e5c6bde2d7b38df5bc6a70b967edc0f6a" gracePeriod=25
Nov 03 18:34:50.906121 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:50.906036    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:34:50.972495 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:50.972460    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp" podUID="61d5238f-d75d-496f-92f0-19e4c640c23f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:34:50.972495 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:50.972476    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v" podUID="5cc564e5-69e5-468e-9b03-a66d33ad95f3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:34:52.583672 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:52.583631    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:34:52.756501 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:52.756465    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:34:56.958386 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:34:56.958338    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:00.971957 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:00.971916    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v" podUID="5cc564e5-69e5-468e-9b03-a66d33ad95f3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:35:00.972406 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:00.971916    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp" podUID="61d5238f-d75d-496f-92f0-19e4c640c23f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:35:06.861717 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:35:06.861674    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:35:10.921229 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:10.921148    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v"
Nov 03 18:35:11.012411 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.012350    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp"
Nov 03 18:35:11.036092 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.036052    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc564e5-69e5-468e-9b03-a66d33ad95f3-kube-api-access-fgghd\" (UniqueName: \"kubernetes.io/projected/5cc564e5-69e5-468e-9b03-a66d33ad95f3-kube-api-access-fgghd\") pod \"5cc564e5-69e5-468e-9b03-a66d33ad95f3\" (UID: \"5cc564e5-69e5-468e-9b03-a66d33ad95f3\") "
Nov 03 18:35:11.036295 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.036146    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cc564e5-69e5-468e-9b03-a66d33ad95f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc564e5-69e5-468e-9b03-a66d33ad95f3-plugins-config-volume\") pod \"5cc564e5-69e5-468e-9b03-a66d33ad95f3\" (UID: \"5cc564e5-69e5-468e-9b03-a66d33ad95f3\") "
Nov 03 18:35:11.036558 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.036530    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc564e5-69e5-468e-9b03-a66d33ad95f3-plugins-config-volume" pod "5cc564e5-69e5-468e-9b03-a66d33ad95f3" (UID: "5cc564e5-69e5-468e-9b03-a66d33ad95f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:35:11.038241 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.038212    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc564e5-69e5-468e-9b03-a66d33ad95f3-kube-api-access-fgghd" pod "5cc564e5-69e5-468e-9b03-a66d33ad95f3" (UID: "5cc564e5-69e5-468e-9b03-a66d33ad95f3"). InnerVolumeSpecName "kube-api-access-fgghd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:35:11.137522 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.137482    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d5238f-d75d-496f-92f0-19e4c640c23f-kube-api-access-9hhch\" (UniqueName: \"kubernetes.io/projected/61d5238f-d75d-496f-92f0-19e4c640c23f-kube-api-access-9hhch\") pod \"61d5238f-d75d-496f-92f0-19e4c640c23f\" (UID: \"61d5238f-d75d-496f-92f0-19e4c640c23f\") "
Nov 03 18:35:11.137703 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.137542    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d5238f-d75d-496f-92f0-19e4c640c23f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d5238f-d75d-496f-92f0-19e4c640c23f-plugins-config-volume\") pod \"61d5238f-d75d-496f-92f0-19e4c640c23f\" (UID: \"61d5238f-d75d-496f-92f0-19e4c640c23f\") "
Nov 03 18:35:11.137911 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.137884    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc564e5-69e5-468e-9b03-a66d33ad95f3-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:11.137911 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.137901    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgghd\" (UniqueName: \"kubernetes.io/projected/5cc564e5-69e5-468e-9b03-a66d33ad95f3-kube-api-access-fgghd\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:11.138075 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.137942    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d5238f-d75d-496f-92f0-19e4c640c23f-plugins-config-volume" pod "61d5238f-d75d-496f-92f0-19e4c640c23f" (UID: "61d5238f-d75d-496f-92f0-19e4c640c23f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:35:11.139704 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.139680    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d5238f-d75d-496f-92f0-19e4c640c23f-kube-api-access-9hhch" pod "61d5238f-d75d-496f-92f0-19e4c640c23f" (UID: "61d5238f-d75d-496f-92f0-19e4c640c23f"). InnerVolumeSpecName "kube-api-access-9hhch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:35:11.238844 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.238801    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hhch\" (UniqueName: \"kubernetes.io/projected/61d5238f-d75d-496f-92f0-19e4c640c23f-kube-api-access-9hhch\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:11.238844 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.238842    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d5238f-d75d-496f-92f0-19e4c640c23f-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:11.714429 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.714394    6059 generic.go:356] "Generic (PLEG): container finished" podID="5cc564e5-69e5-468e-9b03-a66d33ad95f3" containerID="84f7038c4d352a4d0c44845527d4e942fd10ec6dd1288d898a8b50e21883a480" exitCode=0
Nov 03 18:35:11.714649 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.714453    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v"
Nov 03 18:35:11.714649 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.714487    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v" event={"ID":"5cc564e5-69e5-468e-9b03-a66d33ad95f3","Type":"ContainerDied","Data":"84f7038c4d352a4d0c44845527d4e942fd10ec6dd1288d898a8b50e21883a480"}
Nov 03 18:35:11.714649 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.714541    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v" event={"ID":"5cc564e5-69e5-468e-9b03-a66d33ad95f3","Type":"ContainerDied","Data":"bc6769ec054160cb5712151ddaa80318bf0991a54090d5cf0ae30c7a17a2b22d"}
Nov 03 18:35:11.714649 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.714565    6059 scope.go:122] "RemoveContainer" containerID="84f7038c4d352a4d0c44845527d4e942fd10ec6dd1288d898a8b50e21883a480"
Nov 03 18:35:11.715912 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.715883    6059 generic.go:356] "Generic (PLEG): container finished" podID="61d5238f-d75d-496f-92f0-19e4c640c23f" containerID="b4bf9c63615afdd06b229bbea16c042e5c6bde2d7b38df5bc6a70b967edc0f6a" exitCode=0
Nov 03 18:35:11.716040 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.715924    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp"
Nov 03 18:35:11.716040 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.715973    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp" event={"ID":"61d5238f-d75d-496f-92f0-19e4c640c23f","Type":"ContainerDied","Data":"b4bf9c63615afdd06b229bbea16c042e5c6bde2d7b38df5bc6a70b967edc0f6a"}
Nov 03 18:35:11.716040 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.716032    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp" event={"ID":"61d5238f-d75d-496f-92f0-19e4c640c23f","Type":"ContainerDied","Data":"4db027d2c0d1b6b3febca4565fa77443661644ee6724e125a4658a0730689198"}
Nov 03 18:35:11.724272 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.724249    6059 scope.go:122] "RemoveContainer" containerID="84f7038c4d352a4d0c44845527d4e942fd10ec6dd1288d898a8b50e21883a480"
Nov 03 18:35:11.724591 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:35:11.724571    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84f7038c4d352a4d0c44845527d4e942fd10ec6dd1288d898a8b50e21883a480\": container with ID starting with 84f7038c4d352a4d0c44845527d4e942fd10ec6dd1288d898a8b50e21883a480 not found: ID does not exist" containerID="84f7038c4d352a4d0c44845527d4e942fd10ec6dd1288d898a8b50e21883a480"
Nov 03 18:35:11.724679 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.724607    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f7038c4d352a4d0c44845527d4e942fd10ec6dd1288d898a8b50e21883a480"} err="failed to get container status \"84f7038c4d352a4d0c44845527d4e942fd10ec6dd1288d898a8b50e21883a480\": rpc error: code = NotFound desc = could not find container \"84f7038c4d352a4d0c44845527d4e942fd10ec6dd1288d898a8b50e21883a480\": container with ID starting with 84f7038c4d352a4d0c44845527d4e942fd10ec6dd1288d898a8b50e21883a480 not found: ID does not exist"
Nov 03 18:35:11.724679 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.724634    6059 scope.go:122] "RemoveContainer" containerID="b4bf9c63615afdd06b229bbea16c042e5c6bde2d7b38df5bc6a70b967edc0f6a"
Nov 03 18:35:11.733157 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.733138    6059 scope.go:122] "RemoveContainer" containerID="b4bf9c63615afdd06b229bbea16c042e5c6bde2d7b38df5bc6a70b967edc0f6a"
Nov 03 18:35:11.733497 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:35:11.733469    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4bf9c63615afdd06b229bbea16c042e5c6bde2d7b38df5bc6a70b967edc0f6a\": container with ID starting with b4bf9c63615afdd06b229bbea16c042e5c6bde2d7b38df5bc6a70b967edc0f6a not found: ID does not exist" containerID="b4bf9c63615afdd06b229bbea16c042e5c6bde2d7b38df5bc6a70b967edc0f6a"
Nov 03 18:35:11.733604 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.733501    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4bf9c63615afdd06b229bbea16c042e5c6bde2d7b38df5bc6a70b967edc0f6a"} err="failed to get container status \"b4bf9c63615afdd06b229bbea16c042e5c6bde2d7b38df5bc6a70b967edc0f6a\": rpc error: code = NotFound desc = could not find container \"b4bf9c63615afdd06b229bbea16c042e5c6bde2d7b38df5bc6a70b967edc0f6a\": container with ID starting with b4bf9c63615afdd06b229bbea16c042e5c6bde2d7b38df5bc6a70b967edc0f6a not found: ID does not exist"
Nov 03 18:35:11.885075 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.885038    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp"]
Nov 03 18:35:11.890276 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.890250    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp"]
Nov 03 18:35:11.971869 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:11.971827    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-57d07b-568df95vnp" podUID="61d5238f-d75d-496f-92f0-19e4c640c23f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.27:50052\" within 1s: context deadline exceeded"
Nov 03 18:35:11.972301 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:35:11.971927    6059 logging.go:55] [core] [Channel #8845 SubChannel #8846]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.27:50052", ServerName: "10.133.1.27:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.27:50052: operation was canceled"
Nov 03 18:35:12.056661 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:12.056627    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v"]
Nov 03 18:35:12.060217 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:12.060182    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-57d07b-5dff9d457l8s7v"]
Nov 03 18:35:13.335597 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:13.335566    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cc564e5-69e5-468e-9b03-a66d33ad95f3" path="/var/lib/kubelet/pods/5cc564e5-69e5-468e-9b03-a66d33ad95f3/volumes"
Nov 03 18:35:13.336096 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:13.336080    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d5238f-d75d-496f-92f0-19e4c640c23f" path="/var/lib/kubelet/pods/61d5238f-d75d-496f-92f0-19e4c640c23f/volumes"
Nov 03 18:35:17.903706 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:17.903673    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:18.077975 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:18.077942    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:21.478248 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:21.478208    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:22.489730 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:22.489695    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:22.702005 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:22.701974    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:22.974741 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:22.974706    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:24.306866 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:24.306831    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:24.742653 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:24.742616    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:25.335289 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:25.335255    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"]
Nov 03 18:35:25.335782 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:25.335481    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l" podUID="a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" containerName="istio-proxy" containerID="cri-o://0b87d581773bce22e7bae477d15f530b951d6e6d307088d48c3b86910cba631c" gracePeriod=30
Nov 03 18:35:25.357605 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:25.357563    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:28.327188 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:28.327154    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:28.504691 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:28.504646    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:28.861521 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:28.861487    6059 patch_prober.go:29] interesting pod/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:35:28.861707 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:28.861540    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l" podUID="a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:35:30.603848 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.603775    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:35:30.640061 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.640024    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-kube-api-access-vxm7x\" (UniqueName: \"kubernetes.io/projected/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-kube-api-access-vxm7x\") pod \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") "
Nov 03 18:35:30.640244 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.640116    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istiod-ca-cert\") pod \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") "
Nov 03 18:35:30.640244 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.640139    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-workload-socket\") pod \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") "
Nov 03 18:35:30.640244 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.640170    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-data\") pod \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") "
Nov 03 18:35:30.640244 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.640198    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-envoy\") pod \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") "
Nov 03 18:35:30.640244 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.640242    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-podinfo\") pod \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") "
Nov 03 18:35:30.640580 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.640292    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-token\" (UniqueName: \"kubernetes.io/projected/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-token\") pod \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") "
Nov 03 18:35:30.640580 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.640497    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-data" pod "a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" (UID: "a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:35:30.640580 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.640516    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-credential-socket\") pod \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") "
Nov 03 18:35:30.640580 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.640566    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-workload-certs\") pod \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\" (UID: \"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d\") "
Nov 03 18:35:30.640805 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.640584    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istiod-ca-cert" pod "a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" (UID: "a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:35:30.640805 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.640723    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-credential-socket" pod "a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" (UID: "a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:35:30.640923 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.640906    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-workload-certs" pod "a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" (UID: "a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:35:30.641028 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.641004    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:30.641091 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.641022    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-workload-socket" pod "a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" (UID: "a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:35:30.641091 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.641035    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:30.641091 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.641052    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:30.641091 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.641067    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:30.642418 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.642390    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-envoy" pod "a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" (UID: "a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:35:30.642670 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.642651    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-token" pod "a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" (UID: "a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:35:30.642744 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.642725    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-kube-api-access-vxm7x" pod "a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" (UID: "a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d"). InnerVolumeSpecName "kube-api-access-vxm7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:35:30.642805 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.642721    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-podinfo" pod "a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" (UID: "a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:35:30.742605 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.742571    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:30.742605 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.742601    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:30.742605 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.742612    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:30.742870 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.742621    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:30.742870 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.742631    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxm7x\" (UniqueName: \"kubernetes.io/projected/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d-kube-api-access-vxm7x\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:30.845092 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.845058    6059 generic.go:356] "Generic (PLEG): container finished" podID="a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" containerID="0b87d581773bce22e7bae477d15f530b951d6e6d307088d48c3b86910cba631c" exitCode=0
Nov 03 18:35:30.845262 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.845107    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l" event={"ID":"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d","Type":"ContainerDied","Data":"0b87d581773bce22e7bae477d15f530b951d6e6d307088d48c3b86910cba631c"}
Nov 03 18:35:30.845262 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.845112    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"
Nov 03 18:35:30.845262 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.845149    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l" event={"ID":"a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d","Type":"ContainerDied","Data":"d5ed04a82f315941aeed411a7a860cf8b78c4524a8af2b0700044c883bbfe4ca"}
Nov 03 18:35:30.845262 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.845171    6059 scope.go:122] "RemoveContainer" containerID="0b87d581773bce22e7bae477d15f530b951d6e6d307088d48c3b86910cba631c"
Nov 03 18:35:30.855706 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.855682    6059 scope.go:122] "RemoveContainer" containerID="b8a70d633547bf97da5b635b0c452038cc103abb1286b87a37ea0e67f6b666e3"
Nov 03 18:35:30.864458 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.864439    6059 scope.go:122] "RemoveContainer" containerID="0b87d581773bce22e7bae477d15f530b951d6e6d307088d48c3b86910cba631c"
Nov 03 18:35:30.864748 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:35:30.864726    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b87d581773bce22e7bae477d15f530b951d6e6d307088d48c3b86910cba631c\": container with ID starting with 0b87d581773bce22e7bae477d15f530b951d6e6d307088d48c3b86910cba631c not found: ID does not exist" containerID="0b87d581773bce22e7bae477d15f530b951d6e6d307088d48c3b86910cba631c"
Nov 03 18:35:30.864793 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.864762    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b87d581773bce22e7bae477d15f530b951d6e6d307088d48c3b86910cba631c"} err="failed to get container status \"0b87d581773bce22e7bae477d15f530b951d6e6d307088d48c3b86910cba631c\": rpc error: code = NotFound desc = could not find container \"0b87d581773bce22e7bae477d15f530b951d6e6d307088d48c3b86910cba631c\": container with ID starting with 0b87d581773bce22e7bae477d15f530b951d6e6d307088d48c3b86910cba631c not found: ID does not exist"
Nov 03 18:35:30.864793 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.864781    6059 scope.go:122] "RemoveContainer" containerID="b8a70d633547bf97da5b635b0c452038cc103abb1286b87a37ea0e67f6b666e3"
Nov 03 18:35:30.865027 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:35:30.865009    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8a70d633547bf97da5b635b0c452038cc103abb1286b87a37ea0e67f6b666e3\": container with ID starting with b8a70d633547bf97da5b635b0c452038cc103abb1286b87a37ea0e67f6b666e3 not found: ID does not exist" containerID="b8a70d633547bf97da5b635b0c452038cc103abb1286b87a37ea0e67f6b666e3"
Nov 03 18:35:30.865067 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:30.865034    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a70d633547bf97da5b635b0c452038cc103abb1286b87a37ea0e67f6b666e3"} err="failed to get container status \"b8a70d633547bf97da5b635b0c452038cc103abb1286b87a37ea0e67f6b666e3\": rpc error: code = NotFound desc = could not find container \"b8a70d633547bf97da5b635b0c452038cc103abb1286b87a37ea0e67f6b666e3\": container with ID starting with b8a70d633547bf97da5b635b0c452038cc103abb1286b87a37ea0e67f6b666e3 not found: ID does not exist"
Nov 03 18:35:31.050390 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:31.050339    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"]
Nov 03 18:35:31.082233 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:31.082198    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-57d07b-openshift-default-97467ddd5-4hz4l"]
Nov 03 18:35:31.334979 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:31.334944    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" path="/var/lib/kubelet/pods/a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d/volumes"
Nov 03 18:35:36.861109 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:35:36.861071    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:35:42.271711 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.271669    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"]
Nov 03 18:35:42.272266 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.272121    6059 cpu_manager.go:414] "containerMap: removing container" podUID="a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" containerName="istio-proxy"
Nov 03 18:35:42.272266 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.272136    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" containerName="istio-proxy"
Nov 03 18:35:42.272266 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.272145    6059 cpu_manager.go:414] "containerMap: removing container" podUID="61d5238f-d75d-496f-92f0-19e4c640c23f" containerName="payload-pre-processing"
Nov 03 18:35:42.272266 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.272150    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d5238f-d75d-496f-92f0-19e4c640c23f" containerName="payload-pre-processing"
Nov 03 18:35:42.272266 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.272166    6059 cpu_manager.go:414] "containerMap: removing container" podUID="5cc564e5-69e5-468e-9b03-a66d33ad95f3" containerName="payload-processing"
Nov 03 18:35:42.272266 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.272172    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc564e5-69e5-468e-9b03-a66d33ad95f3" containerName="payload-processing"
Nov 03 18:35:42.272266 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.272196    6059 cpu_manager.go:414] "containerMap: removing container" podUID="396d6528-ddd5-4693-b335-8b88324034f3" containerName="revoke-keys"
Nov 03 18:35:42.272266 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.272201    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="396d6528-ddd5-4693-b335-8b88324034f3" containerName="revoke-keys"
Nov 03 18:35:42.272266 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.272274    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" containerName="istio-proxy"
Nov 03 18:35:42.272821 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.272286    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d5238f-d75d-496f-92f0-19e4c640c23f" containerName="payload-pre-processing"
Nov 03 18:35:42.272821 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.272296    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cc564e5-69e5-468e-9b03-a66d33ad95f3" containerName="payload-processing"
Nov 03 18:35:42.272821 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.272304    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" containerName="istio-proxy"
Nov 03 18:35:42.272821 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.272311    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="396d6528-ddd5-4693-b335-8b88324034f3" containerName="revoke-keys"
Nov 03 18:35:42.277031 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.277009    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:42.335167 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.335135    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"]
Nov 03 18:35:42.342158 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.342127    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-5bcc180a-kserve-self-signed-certs\""
Nov 03 18:35:42.361642 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.361601    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-model-cache\") pod \"e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") " pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:42.361803 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.361659    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-dshm\") pod \"e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") " pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:42.361855 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.361831    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gcz9\" (UniqueName: \"kubernetes.io/projected/697549fe-b2d4-4494-82e5-84c3587e1ef5-kube-api-access-4gcz9\") pod \"e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") " pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:42.361918 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.361894    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-home\") pod \"e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") " pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:42.361962 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.361928    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-tmp-dir\") pod \"e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") " pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:42.362020 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.362000    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697549fe-b2d4-4494-82e5-84c3587e1ef5-tls-certs\") pod \"e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") " pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:42.388489 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.388445    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:35:42.463213 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.463162    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-model-cache\") pod \"e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") " pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:42.463438 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.463236    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-dshm\") pod \"e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") " pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:42.463438 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.463305    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gcz9\" (UniqueName: \"kubernetes.io/projected/697549fe-b2d4-4494-82e5-84c3587e1ef5-kube-api-access-4gcz9\") pod \"e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") " pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:42.463438 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.463335    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-home\") pod \"e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") " pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:42.463438 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.463350    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-tmp-dir\") pod \"e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") " pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:42.463438 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.463389    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697549fe-b2d4-4494-82e5-84c3587e1ef5-tls-certs\") pod \"e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") " pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:42.463772 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.463674    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-model-cache\") pod \"e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") " pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:42.463772 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.463707    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-home\") pod \"e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") " pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:42.463882 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.463814    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-tmp-dir\") pod \"e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") " pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:42.465517 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.465497    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-dshm\") pod \"e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") " pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:42.465858 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.465839    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697549fe-b2d4-4494-82e5-84c3587e1ef5-tls-certs\") pod \"e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") " pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:42.526527 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.526448    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gcz9\" (UniqueName: \"kubernetes.io/projected/697549fe-b2d4-4494-82e5-84c3587e1ef5-kube-api-access-4gcz9\") pod \"e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") " pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:42.588932 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.588894    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:42.978153 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:35:42.978120    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod697549fe_b2d4_4494_82e5_84c3587e1ef5.slice/crio-fe5c713eda0eea5adc8f0c5b19576c92fcd2fca6f0108299ab433bfcf17601b2 WatchSource:0}: Error finding container fe5c713eda0eea5adc8f0c5b19576c92fcd2fca6f0108299ab433bfcf17601b2: Status 404 returned error can't find the container with id fe5c713eda0eea5adc8f0c5b19576c92fcd2fca6f0108299ab433bfcf17601b2
Nov 03 18:35:42.980299 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:42.980277    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"]
Nov 03 18:35:43.707170 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:43.707137    6059 scope.go:122] "RemoveContainer" containerID="fd068410fbc49fd38e09b09fcf13c5f21bcf99a2c1b9c11412dd2bd70dd50bcd"
Nov 03 18:35:43.921811 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:43.921772    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2" event={"ID":"697549fe-b2d4-4494-82e5-84c3587e1ef5","Type":"ContainerStarted","Data":"b72109bee6374abdd3219fc0353fd0d1e87e1e1915b498e5d57fa45a03e8dc24"}
Nov 03 18:35:43.921986 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:43.921819    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2" event={"ID":"697549fe-b2d4-4494-82e5-84c3587e1ef5","Type":"ContainerStarted","Data":"fe5c713eda0eea5adc8f0c5b19576c92fcd2fca6f0108299ab433bfcf17601b2"}
Nov 03 18:35:44.044747 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:44.044640    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2" podStartSLOduration=2.044626814 podStartE2EDuration="2.044626814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:35:44.042820677 +0000 UTC m=+4684.840533458" watchObservedRunningTime="2026-11-03 18:35:44.044626814 +0000 UTC m=+4684.842339594"
Nov 03 18:35:44.589571 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:44.589535    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:52.049817 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:52.049778    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kcmw7"]
Nov 03 18:35:52.101612 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:52.101566    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-kcmw7"]
Nov 03 18:35:52.589130 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:52.589088    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:52.601608 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:52.601584    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:53.085466 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:53.085434    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:35:53.334295 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:53.334262    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05846600-c826-431d-974b-3e2108fa19a2" path="/var/lib/kubelet/pods/05846600-c826-431d-974b-3e2108fa19a2/volumes"
Nov 03 18:35:57.321974 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:35:57.321937    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:36:06.860214 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:36:06.860173    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:36:09.486287 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:09.486251    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:36:12.054725 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:12.054686    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zx87s"]
Nov 03 18:36:12.106365 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:12.106319    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zx87s"]
Nov 03 18:36:13.335929 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:13.335894    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94f9bd3a-302f-4115-88ac-6b542ece9817" path="/var/lib/kubelet/pods/94f9bd3a-302f-4115-88ac-6b542ece9817/volumes"
Nov 03 18:36:17.053380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:17.052466    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zp9n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:36:17.053380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:17.052545    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7" podUID="60b66cb6-2bce-4c02-aa7d-7eb25f29f237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:36:17.090379 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:17.089844    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xpshn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:36:17.090379 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:17.089909    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn" podUID="d9e3abf7-c5e1-4414-858e-5f84f6a88670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:36:24.055134 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:24.055096    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lkxgq"]
Nov 03 18:36:24.106611 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:24.106572    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lkxgq"]
Nov 03 18:36:25.334677 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:25.334641    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="347f02b0-79ec-493b-8e06-bae4d0a7d62d" path="/var/lib/kubelet/pods/347f02b0-79ec-493b-8e06-bae4d0a7d62d/volumes"
Nov 03 18:36:36.859900 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:36:36.859864    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:36:44.289122 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:44.289089    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-229pb"]
Nov 03 18:36:44.289580 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:44.289565    6059 cpu_manager.go:414] "containerMap: removing container" podUID="a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" containerName="istio-proxy"
Nov 03 18:36:44.289631 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:44.289581    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4f06ef8-c5b8-4a2b-a24e-35c508ba5b7d" containerName="istio-proxy"
Nov 03 18:36:44.294303 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:44.294286    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-229pb"
Nov 03 18:36:44.349624 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:44.349597    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:36:44.358269 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:44.358240    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:36:44.401465 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:44.401435    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:36:44.401465 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:44.401464    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:36:44.403660 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:44.403630    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-229pb"]
Nov 03 18:36:44.449197 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:44.449167    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e926ddfa-faa4-41b4-900e-272c2c15145b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-229pb\" (UID: \"e926ddfa-faa4-41b4-900e-272c2c15145b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-229pb"
Nov 03 18:36:44.449348 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:44.449279    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e926ddfa-faa4-41b4-900e-272c2c15145b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-229pb\" (UID: \"e926ddfa-faa4-41b4-900e-272c2c15145b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-229pb"
Nov 03 18:36:44.449420 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:44.449380    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48nm8\" (UniqueName: \"kubernetes.io/projected/e926ddfa-faa4-41b4-900e-272c2c15145b-kube-api-access-48nm8\") pod \"maas-api-revoke-sub-452b6d7b-229pb\" (UID: \"e926ddfa-faa4-41b4-900e-272c2c15145b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-229pb"
Nov 03 18:36:44.550029 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:44.549945    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e926ddfa-faa4-41b4-900e-272c2c15145b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-229pb\" (UID: \"e926ddfa-faa4-41b4-900e-272c2c15145b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-229pb"
Nov 03 18:36:44.550029 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:44.550005    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48nm8\" (UniqueName: \"kubernetes.io/projected/e926ddfa-faa4-41b4-900e-272c2c15145b-kube-api-access-48nm8\") pod \"maas-api-revoke-sub-452b6d7b-229pb\" (UID: \"e926ddfa-faa4-41b4-900e-272c2c15145b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-229pb"
Nov 03 18:36:44.550229 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:44.550054    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e926ddfa-faa4-41b4-900e-272c2c15145b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-229pb\" (UID: \"e926ddfa-faa4-41b4-900e-272c2c15145b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-229pb"
Nov 03 18:36:44.550716 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:44.550687    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e926ddfa-faa4-41b4-900e-272c2c15145b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-229pb\" (UID: \"e926ddfa-faa4-41b4-900e-272c2c15145b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-229pb"
Nov 03 18:36:44.552335 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:44.552313    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e926ddfa-faa4-41b4-900e-272c2c15145b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-229pb\" (UID: \"e926ddfa-faa4-41b4-900e-272c2c15145b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-229pb"
Nov 03 18:36:44.613417 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:44.613381    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48nm8\" (UniqueName: \"kubernetes.io/projected/e926ddfa-faa4-41b4-900e-272c2c15145b-kube-api-access-48nm8\") pod \"maas-api-revoke-sub-452b6d7b-229pb\" (UID: \"e926ddfa-faa4-41b4-900e-272c2c15145b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-229pb"
Nov 03 18:36:44.907058 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:44.906970    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-229pb"
Nov 03 18:36:45.278247 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:36:45.278210    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode926ddfa_faa4_41b4_900e_272c2c15145b.slice/crio-037217f4813fc01d3f86a91cce82edad9ee6ecaeb8ed38a5c648918f44d4fc8a WatchSource:0}: Error finding container 037217f4813fc01d3f86a91cce82edad9ee6ecaeb8ed38a5c648918f44d4fc8a: Status 404 returned error can't find the container with id 037217f4813fc01d3f86a91cce82edad9ee6ecaeb8ed38a5c648918f44d4fc8a
Nov 03 18:36:45.279123 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:45.279067    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-229pb"]
Nov 03 18:36:45.440056 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:45.440018    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-229pb" event={"ID":"e926ddfa-faa4-41b4-900e-272c2c15145b","Type":"ContainerStarted","Data":"037217f4813fc01d3f86a91cce82edad9ee6ecaeb8ed38a5c648918f44d4fc8a"}
Nov 03 18:36:46.451556 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:46.451522    6059 generic.go:356] "Generic (PLEG): container finished" podID="e926ddfa-faa4-41b4-900e-272c2c15145b" containerID="f40527202c3d031de0c49f5202a791baf25ca6c82e11019ab047f5616e450ba0" exitCode=0
Nov 03 18:36:46.452253 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:46.451753    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-229pb" event={"ID":"e926ddfa-faa4-41b4-900e-272c2c15145b","Type":"ContainerDied","Data":"f40527202c3d031de0c49f5202a791baf25ca6c82e11019ab047f5616e450ba0"}
Nov 03 18:36:48.442380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:48.430554    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-229pb"
Nov 03 18:36:48.522731 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:48.522643    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-229pb"
Nov 03 18:36:48.523116 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:48.523023    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-229pb" event={"ID":"e926ddfa-faa4-41b4-900e-272c2c15145b","Type":"ContainerDied","Data":"037217f4813fc01d3f86a91cce82edad9ee6ecaeb8ed38a5c648918f44d4fc8a"}
Nov 03 18:36:48.523116 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:48.523088    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="037217f4813fc01d3f86a91cce82edad9ee6ecaeb8ed38a5c648918f44d4fc8a"
Nov 03 18:36:48.619757 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:48.619713    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e926ddfa-faa4-41b4-900e-272c2c15145b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e926ddfa-faa4-41b4-900e-272c2c15145b-maas-api-openshift-ca\") pod \"e926ddfa-faa4-41b4-900e-272c2c15145b\" (UID: \"e926ddfa-faa4-41b4-900e-272c2c15145b\") "
Nov 03 18:36:48.620066 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:48.620042    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e926ddfa-faa4-41b4-900e-272c2c15145b-kube-api-access-48nm8\" (UniqueName: \"kubernetes.io/projected/e926ddfa-faa4-41b4-900e-272c2c15145b-kube-api-access-48nm8\") pod \"e926ddfa-faa4-41b4-900e-272c2c15145b\" (UID: \"e926ddfa-faa4-41b4-900e-272c2c15145b\") "
Nov 03 18:36:48.620325 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:48.620302    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e926ddfa-faa4-41b4-900e-272c2c15145b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e926ddfa-faa4-41b4-900e-272c2c15145b-maas-api-xks-ca\") pod \"e926ddfa-faa4-41b4-900e-272c2c15145b\" (UID: \"e926ddfa-faa4-41b4-900e-272c2c15145b\") "
Nov 03 18:36:48.627375 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:48.623807    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e926ddfa-faa4-41b4-900e-272c2c15145b-maas-api-openshift-ca" pod "e926ddfa-faa4-41b4-900e-272c2c15145b" (UID: "e926ddfa-faa4-41b4-900e-272c2c15145b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:36:48.627375 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:48.627028    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e926ddfa-faa4-41b4-900e-272c2c15145b-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:36:48.627375 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:48.627226    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e926ddfa-faa4-41b4-900e-272c2c15145b-kube-api-access-48nm8" pod "e926ddfa-faa4-41b4-900e-272c2c15145b" (UID: "e926ddfa-faa4-41b4-900e-272c2c15145b"). InnerVolumeSpecName "kube-api-access-48nm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:36:48.687379 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:48.686576    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e926ddfa-faa4-41b4-900e-272c2c15145b-maas-api-xks-ca" pod "e926ddfa-faa4-41b4-900e-272c2c15145b" (UID: "e926ddfa-faa4-41b4-900e-272c2c15145b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:36:48.729379 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:48.728340    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48nm8\" (UniqueName: \"kubernetes.io/projected/e926ddfa-faa4-41b4-900e-272c2c15145b-kube-api-access-48nm8\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:36:48.729379 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:48.728458    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e926ddfa-faa4-41b4-900e-272c2c15145b-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:36:52.699077 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:52.699044    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:36:57.153787 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:57.153751    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"]
Nov 03 18:36:57.210893 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:57.210857    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:36:57.657544 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:36:57.657509    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:37:06.861141 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:37:06.861097    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:37:07.805102 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:07.805063    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:37:12.024153 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.024111    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh"]
Nov 03 18:37:12.024683 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.024664    6059 cpu_manager.go:414] "containerMap: removing container" podUID="e926ddfa-faa4-41b4-900e-272c2c15145b" containerName="revoke-keys"
Nov 03 18:37:12.024729 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.024684    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e926ddfa-faa4-41b4-900e-272c2c15145b" containerName="revoke-keys"
Nov 03 18:37:12.024794 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.024785    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e926ddfa-faa4-41b4-900e-272c2c15145b" containerName="revoke-keys"
Nov 03 18:37:12.028199 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.028183    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh"
Nov 03 18:37:12.075235 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.075202    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e0f10ff-13d9-4d5f-8420-e03866140ffd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-bjwbh\" (UID: \"8e0f10ff-13d9-4d5f-8420-e03866140ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh"
Nov 03 18:37:12.075441 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.075310    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e0f10ff-13d9-4d5f-8420-e03866140ffd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-bjwbh\" (UID: \"8e0f10ff-13d9-4d5f-8420-e03866140ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh"
Nov 03 18:37:12.075441 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.075339    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw2sg\" (UniqueName: \"kubernetes.io/projected/8e0f10ff-13d9-4d5f-8420-e03866140ffd-kube-api-access-tw2sg\") pod \"maas-api-revoke-sub-3e31a506-bjwbh\" (UID: \"8e0f10ff-13d9-4d5f-8420-e03866140ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh"
Nov 03 18:37:12.088269 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.088234    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh"]
Nov 03 18:37:12.136521 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.136490    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:37:12.136705 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.136546    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:37:12.136705 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.136573    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:37:12.136705 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.136576    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:37:12.177873 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.177834    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e0f10ff-13d9-4d5f-8420-e03866140ffd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-bjwbh\" (UID: \"8e0f10ff-13d9-4d5f-8420-e03866140ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh"
Nov 03 18:37:12.178086 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.177966    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e0f10ff-13d9-4d5f-8420-e03866140ffd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-bjwbh\" (UID: \"8e0f10ff-13d9-4d5f-8420-e03866140ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh"
Nov 03 18:37:12.178086 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.177994    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw2sg\" (UniqueName: \"kubernetes.io/projected/8e0f10ff-13d9-4d5f-8420-e03866140ffd-kube-api-access-tw2sg\") pod \"maas-api-revoke-sub-3e31a506-bjwbh\" (UID: \"8e0f10ff-13d9-4d5f-8420-e03866140ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh"
Nov 03 18:37:12.179092 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.179062    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e0f10ff-13d9-4d5f-8420-e03866140ffd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-bjwbh\" (UID: \"8e0f10ff-13d9-4d5f-8420-e03866140ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh"
Nov 03 18:37:12.192391 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.185954    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2" podUID="697549fe-b2d4-4494-82e5-84c3587e1ef5" containerName="main" containerID="cri-o://b72109bee6374abdd3219fc0353fd0d1e87e1e1915b498e5d57fa45a03e8dc24" gracePeriod=45
Nov 03 18:37:12.192926 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.192537    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e0f10ff-13d9-4d5f-8420-e03866140ffd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-bjwbh\" (UID: \"8e0f10ff-13d9-4d5f-8420-e03866140ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh"
Nov 03 18:37:12.243509 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.243461    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw2sg\" (UniqueName: \"kubernetes.io/projected/8e0f10ff-13d9-4d5f-8420-e03866140ffd-kube-api-access-tw2sg\") pod \"maas-api-revoke-sub-3e31a506-bjwbh\" (UID: \"8e0f10ff-13d9-4d5f-8420-e03866140ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh"
Nov 03 18:37:12.342561 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.342501    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh"
Nov 03 18:37:12.556680 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.554281    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:37:12.582121 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.582027    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-tmp-dir\") pod \"697549fe-b2d4-4494-82e5-84c3587e1ef5\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") "
Nov 03 18:37:12.582121 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.582088    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/697549fe-b2d4-4494-82e5-84c3587e1ef5-kube-api-access-4gcz9\" (UniqueName: \"kubernetes.io/projected/697549fe-b2d4-4494-82e5-84c3587e1ef5-kube-api-access-4gcz9\") pod \"697549fe-b2d4-4494-82e5-84c3587e1ef5\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") "
Nov 03 18:37:12.582392 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.582171    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-home\" (UniqueName: \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-home\") pod \"697549fe-b2d4-4494-82e5-84c3587e1ef5\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") "
Nov 03 18:37:12.582392 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.582224    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-model-cache\") pod \"697549fe-b2d4-4494-82e5-84c3587e1ef5\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") "
Nov 03 18:37:12.582392 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.582279    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-dshm\") pod \"697549fe-b2d4-4494-82e5-84c3587e1ef5\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") "
Nov 03 18:37:12.582392 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.582334    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/697549fe-b2d4-4494-82e5-84c3587e1ef5-tls-certs\" (UniqueName: \"kubernetes.io/secret/697549fe-b2d4-4494-82e5-84c3587e1ef5-tls-certs\") pod \"697549fe-b2d4-4494-82e5-84c3587e1ef5\" (UID: \"697549fe-b2d4-4494-82e5-84c3587e1ef5\") "
Nov 03 18:37:12.582963 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.582935    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-tmp-dir" pod "697549fe-b2d4-4494-82e5-84c3587e1ef5" (UID: "697549fe-b2d4-4494-82e5-84c3587e1ef5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:37:12.583112 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.582953    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-model-cache" pod "697549fe-b2d4-4494-82e5-84c3587e1ef5" (UID: "697549fe-b2d4-4494-82e5-84c3587e1ef5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:37:12.583342 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.583313    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-home" pod "697549fe-b2d4-4494-82e5-84c3587e1ef5" (UID: "697549fe-b2d4-4494-82e5-84c3587e1ef5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:37:12.585841 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.585806    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-dshm" pod "697549fe-b2d4-4494-82e5-84c3587e1ef5" (UID: "697549fe-b2d4-4494-82e5-84c3587e1ef5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:37:12.585958 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.585834    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/697549fe-b2d4-4494-82e5-84c3587e1ef5-kube-api-access-4gcz9" pod "697549fe-b2d4-4494-82e5-84c3587e1ef5" (UID: "697549fe-b2d4-4494-82e5-84c3587e1ef5"). InnerVolumeSpecName "kube-api-access-4gcz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:37:12.587829 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.587799    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/697549fe-b2d4-4494-82e5-84c3587e1ef5-tls-certs" pod "697549fe-b2d4-4494-82e5-84c3587e1ef5" (UID: "697549fe-b2d4-4494-82e5-84c3587e1ef5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:37:12.683479 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.683438    6059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-tmp-dir\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:12.683479 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.683482    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gcz9\" (UniqueName: \"kubernetes.io/projected/697549fe-b2d4-4494-82e5-84c3587e1ef5-kube-api-access-4gcz9\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:12.683759 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.683500    6059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-home\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:12.683759 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.683516    6059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-model-cache\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:12.683759 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.683535    6059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697549fe-b2d4-4494-82e5-84c3587e1ef5-dshm\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:12.683759 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.683551    6059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697549fe-b2d4-4494-82e5-84c3587e1ef5-tls-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:12.756825 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.756786    6059 generic.go:356] "Generic (PLEG): container finished" podID="697549fe-b2d4-4494-82e5-84c3587e1ef5" containerID="b72109bee6374abdd3219fc0353fd0d1e87e1e1915b498e5d57fa45a03e8dc24" exitCode=0
Nov 03 18:37:12.757147 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.756847    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"
Nov 03 18:37:12.757285 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.756870    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2" event={"ID":"697549fe-b2d4-4494-82e5-84c3587e1ef5","Type":"ContainerDied","Data":"b72109bee6374abdd3219fc0353fd0d1e87e1e1915b498e5d57fa45a03e8dc24"}
Nov 03 18:37:12.757446 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.757427    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2" event={"ID":"697549fe-b2d4-4494-82e5-84c3587e1ef5","Type":"ContainerDied","Data":"fe5c713eda0eea5adc8f0c5b19576c92fcd2fca6f0108299ab433bfcf17601b2"}
Nov 03 18:37:12.757581 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.757567    6059 scope.go:122] "RemoveContainer" containerID="b72109bee6374abdd3219fc0353fd0d1e87e1e1915b498e5d57fa45a03e8dc24"
Nov 03 18:37:12.771732 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.771702    6059 scope.go:122] "RemoveContainer" containerID="b72109bee6374abdd3219fc0353fd0d1e87e1e1915b498e5d57fa45a03e8dc24"
Nov 03 18:37:12.772148 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:37:12.772122    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b72109bee6374abdd3219fc0353fd0d1e87e1e1915b498e5d57fa45a03e8dc24\": container with ID starting with b72109bee6374abdd3219fc0353fd0d1e87e1e1915b498e5d57fa45a03e8dc24 not found: ID does not exist" containerID="b72109bee6374abdd3219fc0353fd0d1e87e1e1915b498e5d57fa45a03e8dc24"
Nov 03 18:37:12.772249 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.772157    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b72109bee6374abdd3219fc0353fd0d1e87e1e1915b498e5d57fa45a03e8dc24"} err="failed to get container status \"b72109bee6374abdd3219fc0353fd0d1e87e1e1915b498e5d57fa45a03e8dc24\": rpc error: code = NotFound desc = could not find container \"b72109bee6374abdd3219fc0353fd0d1e87e1e1915b498e5d57fa45a03e8dc24\": container with ID starting with b72109bee6374abdd3219fc0353fd0d1e87e1e1915b498e5d57fa45a03e8dc24 not found: ID does not exist"
Nov 03 18:37:12.787288 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.786748    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh"]
Nov 03 18:37:12.788978 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:37:12.788950    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0f10ff_13d9_4d5f_8420_e03866140ffd.slice/crio-e5b34ad8bc3ec6ab0e715f41e85ca12c74d9a43b2f44cfa5d8fcf64fce4b7dc7 WatchSource:0}: Error finding container e5b34ad8bc3ec6ab0e715f41e85ca12c74d9a43b2f44cfa5d8fcf64fce4b7dc7: Status 404 returned error can't find the container with id e5b34ad8bc3ec6ab0e715f41e85ca12c74d9a43b2f44cfa5d8fcf64fce4b7dc7
Nov 03 18:37:12.930842 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.930748    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"]
Nov 03 18:37:12.934140 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:12.934104    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-5bcc180a-kserve-5bbc8dd87c-bmpx2"]
Nov 03 18:37:13.339620 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:13.339576    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="697549fe-b2d4-4494-82e5-84c3587e1ef5" path="/var/lib/kubelet/pods/697549fe-b2d4-4494-82e5-84c3587e1ef5/volumes"
Nov 03 18:37:13.775379 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:13.774712    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh" event={"ID":"8e0f10ff-13d9-4d5f-8420-e03866140ffd","Type":"ContainerStarted","Data":"daa35c74704ad27a2b02fd9e9088bab373aac39770b953e33d78a18199244124"}
Nov 03 18:37:13.775379 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:13.774779    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh" event={"ID":"8e0f10ff-13d9-4d5f-8420-e03866140ffd","Type":"ContainerStarted","Data":"e5b34ad8bc3ec6ab0e715f41e85ca12c74d9a43b2f44cfa5d8fcf64fce4b7dc7"}
Nov 03 18:37:13.903378 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:13.902819    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh" podStartSLOduration=2.902800309 podStartE2EDuration="2.902800309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:13.901849686 +0000 UTC m=+4774.699562467" watchObservedRunningTime="2026-11-03 18:37:13.902800309 +0000 UTC m=+4774.700513094"
Nov 03 18:37:14.823382 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:14.823318    6059 generic.go:356] "Generic (PLEG): container finished" podID="8e0f10ff-13d9-4d5f-8420-e03866140ffd" containerID="daa35c74704ad27a2b02fd9e9088bab373aac39770b953e33d78a18199244124" exitCode=0
Nov 03 18:37:14.824027 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:14.823459    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh" event={"ID":"8e0f10ff-13d9-4d5f-8420-e03866140ffd","Type":"ContainerDied","Data":"daa35c74704ad27a2b02fd9e9088bab373aac39770b953e33d78a18199244124"}
Nov 03 18:37:16.202812 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:16.202784    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh"
Nov 03 18:37:16.331306 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:16.331270    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e0f10ff-13d9-4d5f-8420-e03866140ffd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e0f10ff-13d9-4d5f-8420-e03866140ffd-maas-api-xks-ca\") pod \"8e0f10ff-13d9-4d5f-8420-e03866140ffd\" (UID: \"8e0f10ff-13d9-4d5f-8420-e03866140ffd\") "
Nov 03 18:37:16.331519 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:16.331385    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e0f10ff-13d9-4d5f-8420-e03866140ffd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e0f10ff-13d9-4d5f-8420-e03866140ffd-maas-api-openshift-ca\") pod \"8e0f10ff-13d9-4d5f-8420-e03866140ffd\" (UID: \"8e0f10ff-13d9-4d5f-8420-e03866140ffd\") "
Nov 03 18:37:16.331519 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:16.331451    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e0f10ff-13d9-4d5f-8420-e03866140ffd-kube-api-access-tw2sg\" (UniqueName: \"kubernetes.io/projected/8e0f10ff-13d9-4d5f-8420-e03866140ffd-kube-api-access-tw2sg\") pod \"8e0f10ff-13d9-4d5f-8420-e03866140ffd\" (UID: \"8e0f10ff-13d9-4d5f-8420-e03866140ffd\") "
Nov 03 18:37:16.332843 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:16.332805    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e0f10ff-13d9-4d5f-8420-e03866140ffd-maas-api-openshift-ca" pod "8e0f10ff-13d9-4d5f-8420-e03866140ffd" (UID: "8e0f10ff-13d9-4d5f-8420-e03866140ffd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:37:16.333878 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:16.333846    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e0f10ff-13d9-4d5f-8420-e03866140ffd-maas-api-xks-ca" pod "8e0f10ff-13d9-4d5f-8420-e03866140ffd" (UID: "8e0f10ff-13d9-4d5f-8420-e03866140ffd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:37:16.344019 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:16.343990    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e0f10ff-13d9-4d5f-8420-e03866140ffd-kube-api-access-tw2sg" pod "8e0f10ff-13d9-4d5f-8420-e03866140ffd" (UID: "8e0f10ff-13d9-4d5f-8420-e03866140ffd"). InnerVolumeSpecName "kube-api-access-tw2sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:37:16.433315 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:16.433209    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e0f10ff-13d9-4d5f-8420-e03866140ffd-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:16.433315 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:16.433244    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw2sg\" (UniqueName: \"kubernetes.io/projected/8e0f10ff-13d9-4d5f-8420-e03866140ffd-kube-api-access-tw2sg\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:16.433315 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:16.433259    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e0f10ff-13d9-4d5f-8420-e03866140ffd-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:16.851534 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:16.851503    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh"
Nov 03 18:37:16.851534 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:16.851518    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh" event={"ID":"8e0f10ff-13d9-4d5f-8420-e03866140ffd","Type":"ContainerDied","Data":"e5b34ad8bc3ec6ab0e715f41e85ca12c74d9a43b2f44cfa5d8fcf64fce4b7dc7"}
Nov 03 18:37:16.851771 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:16.851596    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5b34ad8bc3ec6ab0e715f41e85ca12c74d9a43b2f44cfa5d8fcf64fce4b7dc7"
Nov 03 18:37:36.860938 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:37:36.860857    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:37:38.575385 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.575331    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr"]
Nov 03 18:37:38.576093 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.576071    6059 cpu_manager.go:414] "containerMap: removing container" podUID="8e0f10ff-13d9-4d5f-8420-e03866140ffd" containerName="revoke-keys"
Nov 03 18:37:38.576140 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.576096    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e0f10ff-13d9-4d5f-8420-e03866140ffd" containerName="revoke-keys"
Nov 03 18:37:38.576174 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.576151    6059 cpu_manager.go:414] "containerMap: removing container" podUID="697549fe-b2d4-4494-82e5-84c3587e1ef5" containerName="main"
Nov 03 18:37:38.576174 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.576159    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="697549fe-b2d4-4494-82e5-84c3587e1ef5" containerName="main"
Nov 03 18:37:38.576268 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.576257    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e0f10ff-13d9-4d5f-8420-e03866140ffd" containerName="revoke-keys"
Nov 03 18:37:38.576312 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.576281    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="697549fe-b2d4-4494-82e5-84c3587e1ef5" containerName="main"
Nov 03 18:37:38.580103 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.580081    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr"
Nov 03 18:37:38.635223 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.635192    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:37:38.635458 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.635246    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:37:38.635458 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.635378    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:37:38.664977 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.664945    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60794c2d-01a6-4b9e-928c-911b810ea08d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1169f3cb-p7nqr\" (UID: \"60794c2d-01a6-4b9e-928c-911b810ea08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr"
Nov 03 18:37:38.665200 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.664982    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6h4c\" (UniqueName: \"kubernetes.io/projected/60794c2d-01a6-4b9e-928c-911b810ea08d-kube-api-access-p6h4c\") pod \"maas-api-revoke-sub-1169f3cb-p7nqr\" (UID: \"60794c2d-01a6-4b9e-928c-911b810ea08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr"
Nov 03 18:37:38.665200 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.665016    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60794c2d-01a6-4b9e-928c-911b810ea08d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1169f3cb-p7nqr\" (UID: \"60794c2d-01a6-4b9e-928c-911b810ea08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr"
Nov 03 18:37:38.686931 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.686902    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:37:38.689258 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.689236    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr"]
Nov 03 18:37:38.765684 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.765651    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60794c2d-01a6-4b9e-928c-911b810ea08d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1169f3cb-p7nqr\" (UID: \"60794c2d-01a6-4b9e-928c-911b810ea08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr"
Nov 03 18:37:38.765848 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.765692    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6h4c\" (UniqueName: \"kubernetes.io/projected/60794c2d-01a6-4b9e-928c-911b810ea08d-kube-api-access-p6h4c\") pod \"maas-api-revoke-sub-1169f3cb-p7nqr\" (UID: \"60794c2d-01a6-4b9e-928c-911b810ea08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr"
Nov 03 18:37:38.765848 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.765743    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60794c2d-01a6-4b9e-928c-911b810ea08d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1169f3cb-p7nqr\" (UID: \"60794c2d-01a6-4b9e-928c-911b810ea08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr"
Nov 03 18:37:38.766570 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.766538    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60794c2d-01a6-4b9e-928c-911b810ea08d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1169f3cb-p7nqr\" (UID: \"60794c2d-01a6-4b9e-928c-911b810ea08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr"
Nov 03 18:37:38.768012 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.767989    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60794c2d-01a6-4b9e-928c-911b810ea08d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1169f3cb-p7nqr\" (UID: \"60794c2d-01a6-4b9e-928c-911b810ea08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr"
Nov 03 18:37:38.829086 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.829013    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6h4c\" (UniqueName: \"kubernetes.io/projected/60794c2d-01a6-4b9e-928c-911b810ea08d-kube-api-access-p6h4c\") pod \"maas-api-revoke-sub-1169f3cb-p7nqr\" (UID: \"60794c2d-01a6-4b9e-928c-911b810ea08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr"
Nov 03 18:37:38.893181 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:38.893146    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr"
Nov 03 18:37:39.267085 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:37:39.267052    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60794c2d_01a6_4b9e_928c_911b810ea08d.slice/crio-0d5305bb93281a2e2068154954ef37760aab20525aebdc30dd3285c0598a1b9b WatchSource:0}: Error finding container 0d5305bb93281a2e2068154954ef37760aab20525aebdc30dd3285c0598a1b9b: Status 404 returned error can't find the container with id 0d5305bb93281a2e2068154954ef37760aab20525aebdc30dd3285c0598a1b9b
Nov 03 18:37:39.268905 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:39.268867    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr"]
Nov 03 18:37:40.058119 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:40.058073    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr" event={"ID":"60794c2d-01a6-4b9e-928c-911b810ea08d","Type":"ContainerStarted","Data":"c1e3d13a271dfc1283fb708273c668e84f1a0c0d412caa63dccf4a173dba9fe1"}
Nov 03 18:37:40.058582 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:40.058129    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr" event={"ID":"60794c2d-01a6-4b9e-928c-911b810ea08d","Type":"ContainerStarted","Data":"0d5305bb93281a2e2068154954ef37760aab20525aebdc30dd3285c0598a1b9b"}
Nov 03 18:37:40.067988 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:40.067960    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:37:40.086477 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:40.086452    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:37:40.091012 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:40.090994    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:37:40.107592 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:40.107572    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:37:40.178786 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:40.178734    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr" podStartSLOduration=2.178719161 podStartE2EDuration="2.178719161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:40.177656039 +0000 UTC m=+4800.975368821" watchObservedRunningTime="2026-11-03 18:37:40.178719161 +0000 UTC m=+4800.976431941"
Nov 03 18:37:41.051515 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:41.051476    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t"]
Nov 03 18:37:41.066065 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:41.066032    6059 generic.go:356] "Generic (PLEG): container finished" podID="60794c2d-01a6-4b9e-928c-911b810ea08d" containerID="c1e3d13a271dfc1283fb708273c668e84f1a0c0d412caa63dccf4a173dba9fe1" exitCode=0
Nov 03 18:37:41.066507 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:41.066138    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr" event={"ID":"60794c2d-01a6-4b9e-928c-911b810ea08d","Type":"ContainerDied","Data":"c1e3d13a271dfc1283fb708273c668e84f1a0c0d412caa63dccf4a173dba9fe1"}
Nov 03 18:37:41.103229 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:41.103196    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-846b092f-46j6t"]
Nov 03 18:37:41.335424 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:41.335331    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be62999d-979e-481c-8e93-932f8a9415bc" path="/var/lib/kubelet/pods/be62999d-979e-481c-8e93-932f8a9415bc/volumes"
Nov 03 18:37:42.218793 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:42.218766    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr"
Nov 03 18:37:42.299758 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:42.299724    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60794c2d-01a6-4b9e-928c-911b810ea08d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60794c2d-01a6-4b9e-928c-911b810ea08d-maas-api-openshift-ca\") pod \"60794c2d-01a6-4b9e-928c-911b810ea08d\" (UID: \"60794c2d-01a6-4b9e-928c-911b810ea08d\") "
Nov 03 18:37:42.299909 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:42.299793    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60794c2d-01a6-4b9e-928c-911b810ea08d-kube-api-access-p6h4c\" (UniqueName: \"kubernetes.io/projected/60794c2d-01a6-4b9e-928c-911b810ea08d-kube-api-access-p6h4c\") pod \"60794c2d-01a6-4b9e-928c-911b810ea08d\" (UID: \"60794c2d-01a6-4b9e-928c-911b810ea08d\") "
Nov 03 18:37:42.299983 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:42.299917    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60794c2d-01a6-4b9e-928c-911b810ea08d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60794c2d-01a6-4b9e-928c-911b810ea08d-maas-api-xks-ca\") pod \"60794c2d-01a6-4b9e-928c-911b810ea08d\" (UID: \"60794c2d-01a6-4b9e-928c-911b810ea08d\") "
Nov 03 18:37:42.300093 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:42.300069    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60794c2d-01a6-4b9e-928c-911b810ea08d-maas-api-openshift-ca" pod "60794c2d-01a6-4b9e-928c-911b810ea08d" (UID: "60794c2d-01a6-4b9e-928c-911b810ea08d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:37:42.300264 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:42.300237    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60794c2d-01a6-4b9e-928c-911b810ea08d-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:42.301801 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:42.301776    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60794c2d-01a6-4b9e-928c-911b810ea08d-kube-api-access-p6h4c" pod "60794c2d-01a6-4b9e-928c-911b810ea08d" (UID: "60794c2d-01a6-4b9e-928c-911b810ea08d"). InnerVolumeSpecName "kube-api-access-p6h4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:37:42.301905 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:42.301863    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60794c2d-01a6-4b9e-928c-911b810ea08d-maas-api-xks-ca" pod "60794c2d-01a6-4b9e-928c-911b810ea08d" (UID: "60794c2d-01a6-4b9e-928c-911b810ea08d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:37:42.401158 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:42.401069    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6h4c\" (UniqueName: \"kubernetes.io/projected/60794c2d-01a6-4b9e-928c-911b810ea08d-kube-api-access-p6h4c\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:42.401158 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:42.401106    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60794c2d-01a6-4b9e-928c-911b810ea08d-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:43.079865 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:43.079832    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr"
Nov 03 18:37:43.080062 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:43.079835    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr" event={"ID":"60794c2d-01a6-4b9e-928c-911b810ea08d","Type":"ContainerDied","Data":"0d5305bb93281a2e2068154954ef37760aab20525aebdc30dd3285c0598a1b9b"}
Nov 03 18:37:43.080062 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:43.079974    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d5305bb93281a2e2068154954ef37760aab20525aebdc30dd3285c0598a1b9b"
Nov 03 18:37:43.753372 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:43.753337    6059 scope.go:122] "RemoveContainer" containerID="1293ecd1921187c07ac1ac1174eef220689ad7b53c2a4ce45ed7e540a6badc76"
Nov 03 18:37:43.762438 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:43.762414    6059 scope.go:122] "RemoveContainer" containerID="cb3f721075581674ffa54729b669393143bbbf951e263de5908518c60dca6aec"
Nov 03 18:37:43.770377 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:43.770343    6059 scope.go:122] "RemoveContainer" containerID="9eb42916fccdf7e0bb4fdefb43eadbd95e99139ed116d6249decccd24b06036d"
Nov 03 18:37:54.478943 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:54.478907    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:37:55.175474 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.175439    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"]
Nov 03 18:37:55.175913 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.175899    6059 cpu_manager.go:414] "containerMap: removing container" podUID="60794c2d-01a6-4b9e-928c-911b810ea08d" containerName="revoke-keys"
Nov 03 18:37:55.175957 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.175913    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60794c2d-01a6-4b9e-928c-911b810ea08d" containerName="revoke-keys"
Nov 03 18:37:55.175994 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.175988    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="60794c2d-01a6-4b9e-928c-911b810ea08d" containerName="revoke-keys"
Nov 03 18:37:55.179113 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.179097    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.233457 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.233422    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4ee65c-openshift-default-dockercfg-ww2wc\""
Nov 03 18:37:55.238567 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.238541    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-podinfo\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.238715 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.238577    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l7ch\" (UniqueName: \"kubernetes.io/projected/ccf1c62f-ae8f-4486-8613-189132c5379a-kube-api-access-2l7ch\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.238715 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.238686    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-credential-socket\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.238855 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.238810    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-envoy\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.238916 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.238854    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-workload-socket\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.238916 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.238873    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-workload-certs\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.238916 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.238894    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-data\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.238916 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.238913    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-token\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.239154 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.239053    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf1c62f-ae8f-4486-8613-189132c5379a-istiod-ca-cert\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.283976 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.283946    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:37:55.287756 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.287730    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"]
Nov 03 18:37:55.340109 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.340075    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-podinfo\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.340109 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.340110    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l7ch\" (UniqueName: \"kubernetes.io/projected/ccf1c62f-ae8f-4486-8613-189132c5379a-kube-api-access-2l7ch\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.340326 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.340141    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-credential-socket\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.340326 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.340210    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-envoy\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.340428 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.340333    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-workload-socket\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.340428 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.340394    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-workload-certs\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.340515 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.340430    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-data\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.340515 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.340457    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-token\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.340515 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.340498    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf1c62f-ae8f-4486-8613-189132c5379a-istiod-ca-cert\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.340946 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.340705    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-credential-socket\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.340946 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.340758    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-workload-socket\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.340946 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.340780    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-workload-certs\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.340946 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.340892    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-data\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.341342 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.341319    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf1c62f-ae8f-4486-8613-189132c5379a-istiod-ca-cert\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.342488 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.342468    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-envoy\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.342743 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.342728    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-podinfo\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.403269 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.403230    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-token\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.403522 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.403499    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l7ch\" (UniqueName: \"kubernetes.io/projected/ccf1c62f-ae8f-4486-8613-189132c5379a-kube-api-access-2l7ch\") pod \"e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") " pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.478191 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.478162    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:37:55.491959 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.491935    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:55.916935 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:37:55.916906    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf1c62f_ae8f_4486_8613_189132c5379a.slice/crio-1e863ca3f5f8209440359598182452dd6e6c6be80c5e19ec9f439a6ad5dd1afd WatchSource:0}: Error finding container 1e863ca3f5f8209440359598182452dd6e6c6be80c5e19ec9f439a6ad5dd1afd: Status 404 returned error can't find the container with id 1e863ca3f5f8209440359598182452dd6e6c6be80c5e19ec9f439a6ad5dd1afd
Nov 03 18:37:55.918237 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.918208    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"]
Nov 03 18:37:55.919236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.919210    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:37:55.919296 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.919266    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:37:55.919340 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:55.919294    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:37:56.151418 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:56.151381    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9" event={"ID":"ccf1c62f-ae8f-4486-8613-189132c5379a","Type":"ContainerStarted","Data":"86f79ad9d97f27e2546d6985dd93a6a668d837efa0d1ee2b220eb94368fae315"}
Nov 03 18:37:56.151618 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:56.151432    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9" event={"ID":"ccf1c62f-ae8f-4486-8613-189132c5379a","Type":"ContainerStarted","Data":"1e863ca3f5f8209440359598182452dd6e6c6be80c5e19ec9f439a6ad5dd1afd"}
Nov 03 18:37:56.277299 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:56.277238    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9" podStartSLOduration=1.277223736 podStartE2EDuration="1.277223736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:56.275406588 +0000 UTC m=+4817.073119369" watchObservedRunningTime="2026-11-03 18:37:56.277223736 +0000 UTC m=+4817.074936517"
Nov 03 18:37:56.492224 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:56.492168    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:56.497317 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:56.497289    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:57.157263 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:57.157204    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:57.158261 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:57.158240    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:37:58.015330 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:58.015297    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:37:59.238434 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:37:59.238385    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:01.050069 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:01.050025    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz"]
Nov 03 18:38:01.102110 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:01.102068    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-225ff5a4-x5pvz"]
Nov 03 18:38:01.334379 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:01.334280    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37108181-1bdb-48e1-a772-105a6e2d947b" path="/var/lib/kubelet/pods/37108181-1bdb-48e1-a772-105a6e2d947b/volumes"
Nov 03 18:38:01.382317 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:01.382263    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:02.511829 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:02.511757    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:02.689200 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:02.689167    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:02.851373 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:02.851271    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm"]
Nov 03 18:38:02.855343 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:02.855324    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm"
Nov 03 18:38:02.930385 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:02.929784    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4ee65c\""
Nov 03 18:38:02.981516 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:02.981485    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4ee65c-dockercfg-5hwql\""
Nov 03 18:38:02.983868 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:02.983840    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm"]
Nov 03 18:38:02.985952 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:02.985930    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx"]
Nov 03 18:38:02.989724 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:02.989705    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx"
Nov 03 18:38:03.035648 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:03.035613    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g6c4\" (UniqueName: \"kubernetes.io/projected/07edc78f-50ec-4f74-b2f2-5af45b66c651-kube-api-access-2g6c4\") pod \"payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm\" (UID: \"07edc78f-50ec-4f74-b2f2-5af45b66c651\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm"
Nov 03 18:38:03.035855 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:03.035662    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07edc78f-50ec-4f74-b2f2-5af45b66c651-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm\" (UID: \"07edc78f-50ec-4f74-b2f2-5af45b66c651\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm"
Nov 03 18:38:03.050158 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:03.050130    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx"]
Nov 03 18:38:03.137337 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:03.137240    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h42nm\" (UniqueName: \"kubernetes.io/projected/c4a805ff-f044-4aaa-81ec-69b976619282-kube-api-access-h42nm\") pod \"payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx\" (UID: \"c4a805ff-f044-4aaa-81ec-69b976619282\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx"
Nov 03 18:38:03.137544 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:03.137396    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a805ff-f044-4aaa-81ec-69b976619282-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx\" (UID: \"c4a805ff-f044-4aaa-81ec-69b976619282\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx"
Nov 03 18:38:03.137544 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:03.137499    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g6c4\" (UniqueName: \"kubernetes.io/projected/07edc78f-50ec-4f74-b2f2-5af45b66c651-kube-api-access-2g6c4\") pod \"payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm\" (UID: \"07edc78f-50ec-4f74-b2f2-5af45b66c651\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm"
Nov 03 18:38:03.137639 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:03.137555    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07edc78f-50ec-4f74-b2f2-5af45b66c651-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm\" (UID: \"07edc78f-50ec-4f74-b2f2-5af45b66c651\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm"
Nov 03 18:38:03.138285 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:03.138268    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07edc78f-50ec-4f74-b2f2-5af45b66c651-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm\" (UID: \"07edc78f-50ec-4f74-b2f2-5af45b66c651\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm"
Nov 03 18:38:03.200920 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:03.200882    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g6c4\" (UniqueName: \"kubernetes.io/projected/07edc78f-50ec-4f74-b2f2-5af45b66c651-kube-api-access-2g6c4\") pod \"payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm\" (UID: \"07edc78f-50ec-4f74-b2f2-5af45b66c651\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm"
Nov 03 18:38:03.238629 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:03.238585    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h42nm\" (UniqueName: \"kubernetes.io/projected/c4a805ff-f044-4aaa-81ec-69b976619282-kube-api-access-h42nm\") pod \"payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx\" (UID: \"c4a805ff-f044-4aaa-81ec-69b976619282\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx"
Nov 03 18:38:03.238805 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:03.238675    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a805ff-f044-4aaa-81ec-69b976619282-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx\" (UID: \"c4a805ff-f044-4aaa-81ec-69b976619282\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx"
Nov 03 18:38:03.239315 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:03.239297    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a805ff-f044-4aaa-81ec-69b976619282-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx\" (UID: \"c4a805ff-f044-4aaa-81ec-69b976619282\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx"
Nov 03 18:38:03.301434 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:03.301396    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h42nm\" (UniqueName: \"kubernetes.io/projected/c4a805ff-f044-4aaa-81ec-69b976619282-kube-api-access-h42nm\") pod \"payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx\" (UID: \"c4a805ff-f044-4aaa-81ec-69b976619282\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx"
Nov 03 18:38:03.305018 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:03.304999    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx"
Nov 03 18:38:03.443017 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:03.442940    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx"]
Nov 03 18:38:03.468950 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:03.468915    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm"
Nov 03 18:38:03.543875 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:03.543833    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm"]
Nov 03 18:38:03.676751 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:38:03.676724    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a805ff_f044_4aaa_81ec_69b976619282.slice/crio-0cbed45410c91f068c36d3a982a57fe2ffbb8f3e0c3f96176f61f1e798f0dd05 WatchSource:0}: Error finding container 0cbed45410c91f068c36d3a982a57fe2ffbb8f3e0c3f96176f61f1e798f0dd05: Status 404 returned error can't find the container with id 0cbed45410c91f068c36d3a982a57fe2ffbb8f3e0c3f96176f61f1e798f0dd05
Nov 03 18:38:03.678577 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:03.678558    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx"]
Nov 03 18:38:03.801773 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:03.801737    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:03.841814 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:38:03.841769    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07edc78f_50ec_4f74_b2f2_5af45b66c651.slice/crio-481995773bea255cef9555d4c4e58f0014f89ed45fd190bf2f907c70999006b8 WatchSource:0}: Error finding container 481995773bea255cef9555d4c4e58f0014f89ed45fd190bf2f907c70999006b8: Status 404 returned error can't find the container with id 481995773bea255cef9555d4c4e58f0014f89ed45fd190bf2f907c70999006b8
Nov 03 18:38:03.842632 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:03.842610    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm"]
Nov 03 18:38:03.971607 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:03.971514    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:04.194346 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:04.194303    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx" event={"ID":"c4a805ff-f044-4aaa-81ec-69b976619282","Type":"ContainerStarted","Data":"5c72b0a5d6fc0c3333c3d56af4a13247bdd97db33cd95cdf07d7c3cb1a47cb20"}
Nov 03 18:38:04.194567 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:04.194449    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx"
Nov 03 18:38:04.194567 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:04.194487    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx" event={"ID":"c4a805ff-f044-4aaa-81ec-69b976619282","Type":"ContainerStarted","Data":"0cbed45410c91f068c36d3a982a57fe2ffbb8f3e0c3f96176f61f1e798f0dd05"}
Nov 03 18:38:04.195873 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:04.195850    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm" event={"ID":"07edc78f-50ec-4f74-b2f2-5af45b66c651","Type":"ContainerStarted","Data":"60c7db8ad839a7e275a0225e1d208e348965b7a7352a582e2447abceea60592d"}
Nov 03 18:38:04.195944 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:04.195895    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm"
Nov 03 18:38:04.195944 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:04.195911    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm" event={"ID":"07edc78f-50ec-4f74-b2f2-5af45b66c651","Type":"ContainerStarted","Data":"481995773bea255cef9555d4c4e58f0014f89ed45fd190bf2f907c70999006b8"}
Nov 03 18:38:04.314591 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:04.314476    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx" podStartSLOduration=2.314456692 podStartE2EDuration="2.314456692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:04.313191594 +0000 UTC m=+4825.110904378" watchObservedRunningTime="2026-11-03 18:38:04.314456692 +0000 UTC m=+4825.112169474"
Nov 03 18:38:04.434729 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:04.434660    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm" podStartSLOduration=2.434643189 podStartE2EDuration="2.434643189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:04.432799415 +0000 UTC m=+4825.230512196" watchObservedRunningTime="2026-11-03 18:38:04.434643189 +0000 UTC m=+4825.232355974"
Nov 03 18:38:06.861489 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:38:06.861450    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:38:09.195372 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:09.195286    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx" podUID="c4a805ff-f044-4aaa-81ec-69b976619282" containerName="payload-processing" containerID="cri-o://5c72b0a5d6fc0c3333c3d56af4a13247bdd97db33cd95cdf07d7c3cb1a47cb20" gracePeriod=25
Nov 03 18:38:09.196765 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:09.196738    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm" podUID="07edc78f-50ec-4f74-b2f2-5af45b66c651" containerName="payload-pre-processing" containerID="cri-o://60c7db8ad839a7e275a0225e1d208e348965b7a7352a582e2447abceea60592d" gracePeriod=25
Nov 03 18:38:09.453926 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:09.453863    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx"
Nov 03 18:38:09.474345 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:09.474323    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm"
Nov 03 18:38:09.507879 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:09.507850    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07edc78f-50ec-4f74-b2f2-5af45b66c651-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07edc78f-50ec-4f74-b2f2-5af45b66c651-plugins-config-volume\") pod \"07edc78f-50ec-4f74-b2f2-5af45b66c651\" (UID: \"07edc78f-50ec-4f74-b2f2-5af45b66c651\") "
Nov 03 18:38:09.508025 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:09.507902    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07edc78f-50ec-4f74-b2f2-5af45b66c651-kube-api-access-2g6c4\" (UniqueName: \"kubernetes.io/projected/07edc78f-50ec-4f74-b2f2-5af45b66c651-kube-api-access-2g6c4\") pod \"07edc78f-50ec-4f74-b2f2-5af45b66c651\" (UID: \"07edc78f-50ec-4f74-b2f2-5af45b66c651\") "
Nov 03 18:38:09.508025 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:09.507945    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4a805ff-f044-4aaa-81ec-69b976619282-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a805ff-f044-4aaa-81ec-69b976619282-plugins-config-volume\") pod \"c4a805ff-f044-4aaa-81ec-69b976619282\" (UID: \"c4a805ff-f044-4aaa-81ec-69b976619282\") "
Nov 03 18:38:09.508025 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:09.508005    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a805ff-f044-4aaa-81ec-69b976619282-kube-api-access-h42nm\" (UniqueName: \"kubernetes.io/projected/c4a805ff-f044-4aaa-81ec-69b976619282-kube-api-access-h42nm\") pod \"c4a805ff-f044-4aaa-81ec-69b976619282\" (UID: \"c4a805ff-f044-4aaa-81ec-69b976619282\") "
Nov 03 18:38:09.508334 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:09.508301    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07edc78f-50ec-4f74-b2f2-5af45b66c651-plugins-config-volume" pod "07edc78f-50ec-4f74-b2f2-5af45b66c651" (UID: "07edc78f-50ec-4f74-b2f2-5af45b66c651"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:38:09.508334 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:09.508321    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a805ff-f044-4aaa-81ec-69b976619282-plugins-config-volume" pod "c4a805ff-f044-4aaa-81ec-69b976619282" (UID: "c4a805ff-f044-4aaa-81ec-69b976619282"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:38:09.508572 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:09.508550    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07edc78f-50ec-4f74-b2f2-5af45b66c651-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:38:09.508619 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:09.508578    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a805ff-f044-4aaa-81ec-69b976619282-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:38:09.510090 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:09.510071    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07edc78f-50ec-4f74-b2f2-5af45b66c651-kube-api-access-2g6c4" pod "07edc78f-50ec-4f74-b2f2-5af45b66c651" (UID: "07edc78f-50ec-4f74-b2f2-5af45b66c651"). InnerVolumeSpecName "kube-api-access-2g6c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:38:09.510167 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:09.510092    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a805ff-f044-4aaa-81ec-69b976619282-kube-api-access-h42nm" pod "c4a805ff-f044-4aaa-81ec-69b976619282" (UID: "c4a805ff-f044-4aaa-81ec-69b976619282"). InnerVolumeSpecName "kube-api-access-h42nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:38:09.609696 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:09.609654    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g6c4\" (UniqueName: \"kubernetes.io/projected/07edc78f-50ec-4f74-b2f2-5af45b66c651-kube-api-access-2g6c4\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:38:09.609696 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:09.609683    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h42nm\" (UniqueName: \"kubernetes.io/projected/c4a805ff-f044-4aaa-81ec-69b976619282-kube-api-access-h42nm\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:38:10.235284 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:10.235249    6059 generic.go:356] "Generic (PLEG): container finished" podID="07edc78f-50ec-4f74-b2f2-5af45b66c651" containerID="60c7db8ad839a7e275a0225e1d208e348965b7a7352a582e2447abceea60592d" exitCode=0
Nov 03 18:38:10.235742 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:10.235291    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm"
Nov 03 18:38:10.235742 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:10.235342    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm" event={"ID":"07edc78f-50ec-4f74-b2f2-5af45b66c651","Type":"ContainerDied","Data":"60c7db8ad839a7e275a0225e1d208e348965b7a7352a582e2447abceea60592d"}
Nov 03 18:38:10.235742 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:10.235410    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm" event={"ID":"07edc78f-50ec-4f74-b2f2-5af45b66c651","Type":"ContainerDied","Data":"481995773bea255cef9555d4c4e58f0014f89ed45fd190bf2f907c70999006b8"}
Nov 03 18:38:10.235742 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:10.235433    6059 scope.go:122] "RemoveContainer" containerID="60c7db8ad839a7e275a0225e1d208e348965b7a7352a582e2447abceea60592d"
Nov 03 18:38:10.236956 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:10.236933    6059 generic.go:356] "Generic (PLEG): container finished" podID="c4a805ff-f044-4aaa-81ec-69b976619282" containerID="5c72b0a5d6fc0c3333c3d56af4a13247bdd97db33cd95cdf07d7c3cb1a47cb20" exitCode=0
Nov 03 18:38:10.237066 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:10.236972    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx"
Nov 03 18:38:10.237066 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:10.237035    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx" event={"ID":"c4a805ff-f044-4aaa-81ec-69b976619282","Type":"ContainerDied","Data":"5c72b0a5d6fc0c3333c3d56af4a13247bdd97db33cd95cdf07d7c3cb1a47cb20"}
Nov 03 18:38:10.237181 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:10.237086    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx" event={"ID":"c4a805ff-f044-4aaa-81ec-69b976619282","Type":"ContainerDied","Data":"0cbed45410c91f068c36d3a982a57fe2ffbb8f3e0c3f96176f61f1e798f0dd05"}
Nov 03 18:38:10.246409 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:10.246387    6059 scope.go:122] "RemoveContainer" containerID="60c7db8ad839a7e275a0225e1d208e348965b7a7352a582e2447abceea60592d"
Nov 03 18:38:10.246734 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:38:10.246711    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60c7db8ad839a7e275a0225e1d208e348965b7a7352a582e2447abceea60592d\": container with ID starting with 60c7db8ad839a7e275a0225e1d208e348965b7a7352a582e2447abceea60592d not found: ID does not exist" containerID="60c7db8ad839a7e275a0225e1d208e348965b7a7352a582e2447abceea60592d"
Nov 03 18:38:10.246818 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:10.246743    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c7db8ad839a7e275a0225e1d208e348965b7a7352a582e2447abceea60592d"} err="failed to get container status \"60c7db8ad839a7e275a0225e1d208e348965b7a7352a582e2447abceea60592d\": rpc error: code = NotFound desc = could not find container \"60c7db8ad839a7e275a0225e1d208e348965b7a7352a582e2447abceea60592d\": container with ID starting with 60c7db8ad839a7e275a0225e1d208e348965b7a7352a582e2447abceea60592d not found: ID does not exist"
Nov 03 18:38:10.246818 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:10.246761    6059 scope.go:122] "RemoveContainer" containerID="5c72b0a5d6fc0c3333c3d56af4a13247bdd97db33cd95cdf07d7c3cb1a47cb20"
Nov 03 18:38:10.255004 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:10.254979    6059 scope.go:122] "RemoveContainer" containerID="5c72b0a5d6fc0c3333c3d56af4a13247bdd97db33cd95cdf07d7c3cb1a47cb20"
Nov 03 18:38:10.255244 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:38:10.255225    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c72b0a5d6fc0c3333c3d56af4a13247bdd97db33cd95cdf07d7c3cb1a47cb20\": container with ID starting with 5c72b0a5d6fc0c3333c3d56af4a13247bdd97db33cd95cdf07d7c3cb1a47cb20 not found: ID does not exist" containerID="5c72b0a5d6fc0c3333c3d56af4a13247bdd97db33cd95cdf07d7c3cb1a47cb20"
Nov 03 18:38:10.255315 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:10.255258    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c72b0a5d6fc0c3333c3d56af4a13247bdd97db33cd95cdf07d7c3cb1a47cb20"} err="failed to get container status \"5c72b0a5d6fc0c3333c3d56af4a13247bdd97db33cd95cdf07d7c3cb1a47cb20\": rpc error: code = NotFound desc = could not find container \"5c72b0a5d6fc0c3333c3d56af4a13247bdd97db33cd95cdf07d7c3cb1a47cb20\": container with ID starting with 5c72b0a5d6fc0c3333c3d56af4a13247bdd97db33cd95cdf07d7c3cb1a47cb20 not found: ID does not exist"
Nov 03 18:38:10.404731 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:10.404698    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm"]
Nov 03 18:38:10.456370 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:10.456328    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ee65c-7d6b7d5dff-5w7fm"]
Nov 03 18:38:10.570129 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:10.570043    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx"]
Nov 03 18:38:10.576256 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:10.576231    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ee65c-57f548fb77-q4cbx"]
Nov 03 18:38:11.337627 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:11.337592    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07edc78f-50ec-4f74-b2f2-5af45b66c651" path="/var/lib/kubelet/pods/07edc78f-50ec-4f74-b2f2-5af45b66c651/volumes"
Nov 03 18:38:11.338022 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:11.337958    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4a805ff-f044-4aaa-81ec-69b976619282" path="/var/lib/kubelet/pods/c4a805ff-f044-4aaa-81ec-69b976619282/volumes"
Nov 03 18:38:11.480688 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:11.480658    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:11.650793 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:11.650711    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:11.827369 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:11.827326    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:15.348916 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:15.348876    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:15.862776 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:15.862740    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:18.398012 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:18.397977    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:23.644877 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:23.644835    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:27.158655 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:27.158621    6059 patch_prober.go:29] interesting pod/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body=
Nov 03 18:38:27.159101 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:27.158699    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9" podUID="ccf1c62f-ae8f-4486-8613-189132c5379a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused"
Nov 03 18:38:27.345733 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:27.345703    6059 generic.go:356] "Generic (PLEG): container finished" podID="ccf1c62f-ae8f-4486-8613-189132c5379a" containerID="86f79ad9d97f27e2546d6985dd93a6a668d837efa0d1ee2b220eb94368fae315" exitCode=0
Nov 03 18:38:27.345905 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:27.345775    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9" event={"ID":"ccf1c62f-ae8f-4486-8613-189132c5379a","Type":"ContainerDied","Data":"86f79ad9d97f27e2546d6985dd93a6a668d837efa0d1ee2b220eb94368fae315"}
Nov 03 18:38:27.346107 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:27.346091    6059 scope.go:122] "RemoveContainer" containerID="86f79ad9d97f27e2546d6985dd93a6a668d837efa0d1ee2b220eb94368fae315"
Nov 03 18:38:27.347250 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:27.347222    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:38:27.347383 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:27.347283    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:38:27.347383 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:27.347316    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:38:28.360602 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:28.360558    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9" event={"ID":"ccf1c62f-ae8f-4486-8613-189132c5379a","Type":"ContainerStarted","Data":"0dda9d0a3adb0c27a7045118b5cbd5af6f1ffa5a7415712b1d1f115b1fa7fd68"}
Nov 03 18:38:28.492524 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:28.492490    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:38:28.492709 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:28.492686    6059 patch_prober.go:29] interesting pod/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body=
Nov 03 18:38:28.492763 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:28.492741    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9" podUID="ccf1c62f-ae8f-4486-8613-189132c5379a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused"
Nov 03 18:38:29.492541 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:29.492495    6059 patch_prober.go:29] interesting pod/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body=
Nov 03 18:38:29.493136 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:29.493103    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9" podUID="ccf1c62f-ae8f-4486-8613-189132c5379a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused"
Nov 03 18:38:30.376511 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:30.376479    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:30.492965 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:30.492928    6059 patch_prober.go:29] interesting pod/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body=
Nov 03 18:38:30.493381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:30.493002    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9" podUID="ccf1c62f-ae8f-4486-8613-189132c5379a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused"
Nov 03 18:38:31.492840 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:31.492804    6059 patch_prober.go:29] interesting pod/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body=
Nov 03 18:38:31.493030 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:31.492857    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9" podUID="ccf1c62f-ae8f-4486-8613-189132c5379a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused"
Nov 03 18:38:32.493096 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:32.493060    6059 patch_prober.go:29] interesting pod/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body=
Nov 03 18:38:32.493535 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:32.493141    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9" podUID="ccf1c62f-ae8f-4486-8613-189132c5379a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused"
Nov 03 18:38:33.492526 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:33.492491    6059 patch_prober.go:29] interesting pod/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body=
Nov 03 18:38:33.492714 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:33.492557    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9" podUID="ccf1c62f-ae8f-4486-8613-189132c5379a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused"
Nov 03 18:38:34.496835 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:34.496803    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:38:34.497253 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:34.497223    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:38:34.497981 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:34.497965    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:38:34.650558 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:34.650514    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:35.560167 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:35.560117    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:35.726578 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:35.726545    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:36.069644 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.069614    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"]
Nov 03 18:38:36.070259 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.070237    6059 cpu_manager.go:414] "containerMap: removing container" podUID="c4a805ff-f044-4aaa-81ec-69b976619282" containerName="payload-processing"
Nov 03 18:38:36.070310 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.070263    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4a805ff-f044-4aaa-81ec-69b976619282" containerName="payload-processing"
Nov 03 18:38:36.070310 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.070280    6059 cpu_manager.go:414] "containerMap: removing container" podUID="07edc78f-50ec-4f74-b2f2-5af45b66c651" containerName="payload-pre-processing"
Nov 03 18:38:36.070310 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.070288    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07edc78f-50ec-4f74-b2f2-5af45b66c651" containerName="payload-pre-processing"
Nov 03 18:38:36.070466 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.070458    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4a805ff-f044-4aaa-81ec-69b976619282" containerName="payload-processing"
Nov 03 18:38:36.070555 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.070471    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="07edc78f-50ec-4f74-b2f2-5af45b66c651" containerName="payload-pre-processing"
Nov 03 18:38:36.073864 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.073848    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.128220 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.128192    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-48c17a-openshift-default-dockercfg-c9gbq\""
Nov 03 18:38:36.135910 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.135884    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istiod-ca-cert\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.136072 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.135935    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-credential-socket\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.136072 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.135963    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkxsh\" (UniqueName: \"kubernetes.io/projected/789edf94-9b0b-4fb8-869f-dc6791bf5a92-kube-api-access-lkxsh\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.136168 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.136120    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-workload-certs\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.136220 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.136189    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-envoy\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.136307 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.136292    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-data\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.136370 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.136325    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-token\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.136447 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.136431    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-podinfo\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.136498 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.136485    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-workload-socket\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.183222 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.183187    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"]
Nov 03 18:38:36.199538 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.199510    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:36.261709 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.245024    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-credential-socket\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.261709 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.245082    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkxsh\" (UniqueName: \"kubernetes.io/projected/789edf94-9b0b-4fb8-869f-dc6791bf5a92-kube-api-access-lkxsh\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.261709 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.245129    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-workload-certs\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.261709 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.245183    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-envoy\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.261709 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.245269    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-data\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.261709 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.245301    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-token\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.261709 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.245382    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-podinfo\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.261709 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.245446    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-workload-socket\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.261709 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.245489    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istiod-ca-cert\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.261709 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.246405    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istiod-ca-cert\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.261709 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.246736    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-credential-socket\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.261709 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.247277    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-workload-certs\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.261709 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.253168    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-data\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.261709 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.257274    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-podinfo\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.261709 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.257578    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-workload-socket\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.264163 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.264134    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-envoy\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.309669 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.309642    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkxsh\" (UniqueName: \"kubernetes.io/projected/789edf94-9b0b-4fb8-869f-dc6791bf5a92-kube-api-access-lkxsh\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.314555 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.314531    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-token\") pod \"e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") " pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.367892 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.367805    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:36.386410 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.386380    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:36.816402 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.816340    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"]
Nov 03 18:38:36.847262 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:38:36.847230    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789edf94_9b0b_4fb8_869f_dc6791bf5a92.slice/crio-3b68bd5d02f8328927ae1e59390878a2f2709e62e9dc03c49b0babec639b57d8 WatchSource:0}: Error finding container 3b68bd5d02f8328927ae1e59390878a2f2709e62e9dc03c49b0babec639b57d8: Status 404 returned error can't find the container with id 3b68bd5d02f8328927ae1e59390878a2f2709e62e9dc03c49b0babec639b57d8
Nov 03 18:38:36.849834 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.849800    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:38:36.849925 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.849883    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:38:36.849965 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:36.849932    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:38:36.863488 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:38:36.863456    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:38:37.414409 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:37.414351    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn" event={"ID":"789edf94-9b0b-4fb8-869f-dc6791bf5a92","Type":"ContainerStarted","Data":"75c5bc75240907d6128f7377f58ab171cf93208e87fd175ce59951b4de184e0b"}
Nov 03 18:38:37.414618 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:37.414430    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn" event={"ID":"789edf94-9b0b-4fb8-869f-dc6791bf5a92","Type":"ContainerStarted","Data":"3b68bd5d02f8328927ae1e59390878a2f2709e62e9dc03c49b0babec639b57d8"}
Nov 03 18:38:37.541009 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:37.540949    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn" podStartSLOduration=1.540935403 podStartE2EDuration="1.540935403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:37.538296193 +0000 UTC m=+4858.336008975" watchObservedRunningTime="2026-11-03 18:38:37.540935403 +0000 UTC m=+4858.338648183"
Nov 03 18:38:38.387415 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:38.387384    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:38.392230 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:38.392204    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:38.420702 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:38.420673    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:38.421736 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:38.421714    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:38:40.456550 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:40.456516    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:40.658310 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:40.658273    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:42.681515 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:42.681413    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:43.590955 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:43.590904    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:43.765738 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:43.765704    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:43.803682 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:43.803650    6059 scope.go:122] "RemoveContainer" containerID="b7c693ae19186ce83a4cbcf026b751d147582125fce50eee083e7f91ddc03cd4"
Nov 03 18:38:43.979436 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:43.979397    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c"]
Nov 03 18:38:43.984652 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:43.984632    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c"
Nov 03 18:38:44.056511 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:44.055504    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c"]
Nov 03 18:38:44.058111 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:44.058082    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8cec85-8acc-48b4-b951-53bb664df3cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c\" (UID: \"ce8cec85-8acc-48b4-b951-53bb664df3cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c"
Nov 03 18:38:44.058254 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:44.058140    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n95s\" (UniqueName: \"kubernetes.io/projected/ce8cec85-8acc-48b4-b951-53bb664df3cb-kube-api-access-7n95s\") pod \"payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c\" (UID: \"ce8cec85-8acc-48b4-b951-53bb664df3cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c"
Nov 03 18:38:44.103315 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:44.103224    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-48c17a\""
Nov 03 18:38:44.103315 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:44.103232    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-48c17a-dockercfg-px62t\""
Nov 03 18:38:44.159787 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:44.159749    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8cec85-8acc-48b4-b951-53bb664df3cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c\" (UID: \"ce8cec85-8acc-48b4-b951-53bb664df3cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c"
Nov 03 18:38:44.159972 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:44.159807    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n95s\" (UniqueName: \"kubernetes.io/projected/ce8cec85-8acc-48b4-b951-53bb664df3cb-kube-api-access-7n95s\") pod \"payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c\" (UID: \"ce8cec85-8acc-48b4-b951-53bb664df3cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c"
Nov 03 18:38:44.160570 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:44.160549    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8cec85-8acc-48b4-b951-53bb664df3cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c\" (UID: \"ce8cec85-8acc-48b4-b951-53bb664df3cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c"
Nov 03 18:38:44.224693 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:44.224654    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n95s\" (UniqueName: \"kubernetes.io/projected/ce8cec85-8acc-48b4-b951-53bb664df3cb-kube-api-access-7n95s\") pod \"payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c\" (UID: \"ce8cec85-8acc-48b4-b951-53bb664df3cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c"
Nov 03 18:38:44.298554 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:44.298520    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c"
Nov 03 18:38:44.670322 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:38:44.670288    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8cec85_8acc_48b4_b951_53bb664df3cb.slice/crio-c39fbc75dec887cf88799ba7b9ab067b88826cc2148b55e31c910e5a976ab31c WatchSource:0}: Error finding container c39fbc75dec887cf88799ba7b9ab067b88826cc2148b55e31c910e5a976ab31c: Status 404 returned error can't find the container with id c39fbc75dec887cf88799ba7b9ab067b88826cc2148b55e31c910e5a976ab31c
Nov 03 18:38:44.673190 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:44.673164    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c"]
Nov 03 18:38:44.733170 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:44.733144    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c"]
Nov 03 18:38:45.005328 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:45.005295    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:45.178673 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:45.178635    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:45.460263 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:45.460231    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c" event={"ID":"ce8cec85-8acc-48b4-b951-53bb664df3cb","Type":"ContainerStarted","Data":"e7f2ac08a319513ee43903c37064849b09c318de8b329414032d8719d3abb5dc"}
Nov 03 18:38:45.460470 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:45.460296    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c" event={"ID":"ce8cec85-8acc-48b4-b951-53bb664df3cb","Type":"ContainerStarted","Data":"c39fbc75dec887cf88799ba7b9ab067b88826cc2148b55e31c910e5a976ab31c"}
Nov 03 18:38:45.460470 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:45.460387    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c"
Nov 03 18:38:45.582266 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:45.582197    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c" podStartSLOduration=2.582176182 podStartE2EDuration="2.582176182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:45.57992536 +0000 UTC m=+4866.377638142" watchObservedRunningTime="2026-11-03 18:38:45.582176182 +0000 UTC m=+4866.379888965"
Nov 03 18:38:47.988328 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:47.988290    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:50.460578 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:50.460513    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c" podUID="ce8cec85-8acc-48b4-b951-53bb664df3cb" containerName="payload-pre-processing" containerID="cri-o://e7f2ac08a319513ee43903c37064849b09c318de8b329414032d8719d3abb5dc" gracePeriod=25
Nov 03 18:38:50.743997 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:50.743962    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c"
Nov 03 18:38:50.835984 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:50.835945    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce8cec85-8acc-48b4-b951-53bb664df3cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8cec85-8acc-48b4-b951-53bb664df3cb-plugins-config-volume\") pod \"ce8cec85-8acc-48b4-b951-53bb664df3cb\" (UID: \"ce8cec85-8acc-48b4-b951-53bb664df3cb\") "
Nov 03 18:38:50.836195 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:50.836177    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce8cec85-8acc-48b4-b951-53bb664df3cb-kube-api-access-7n95s\" (UniqueName: \"kubernetes.io/projected/ce8cec85-8acc-48b4-b951-53bb664df3cb-kube-api-access-7n95s\") pod \"ce8cec85-8acc-48b4-b951-53bb664df3cb\" (UID: \"ce8cec85-8acc-48b4-b951-53bb664df3cb\") "
Nov 03 18:38:50.836413 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:50.836395    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce8cec85-8acc-48b4-b951-53bb664df3cb-plugins-config-volume" pod "ce8cec85-8acc-48b4-b951-53bb664df3cb" (UID: "ce8cec85-8acc-48b4-b951-53bb664df3cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:38:50.836588 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:50.836572    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8cec85-8acc-48b4-b951-53bb664df3cb-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:38:50.838223 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:50.838200    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce8cec85-8acc-48b4-b951-53bb664df3cb-kube-api-access-7n95s" pod "ce8cec85-8acc-48b4-b951-53bb664df3cb" (UID: "ce8cec85-8acc-48b4-b951-53bb664df3cb"). InnerVolumeSpecName "kube-api-access-7n95s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:38:50.937810 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:50.937776    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n95s\" (UniqueName: \"kubernetes.io/projected/ce8cec85-8acc-48b4-b951-53bb664df3cb-kube-api-access-7n95s\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:38:51.493224 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:51.493186    6059 generic.go:356] "Generic (PLEG): container finished" podID="ce8cec85-8acc-48b4-b951-53bb664df3cb" containerID="e7f2ac08a319513ee43903c37064849b09c318de8b329414032d8719d3abb5dc" exitCode=0
Nov 03 18:38:51.493736 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:51.493236    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c"
Nov 03 18:38:51.493736 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:51.493290    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c" event={"ID":"ce8cec85-8acc-48b4-b951-53bb664df3cb","Type":"ContainerDied","Data":"e7f2ac08a319513ee43903c37064849b09c318de8b329414032d8719d3abb5dc"}
Nov 03 18:38:51.493736 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:51.493545    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c" event={"ID":"ce8cec85-8acc-48b4-b951-53bb664df3cb","Type":"ContainerDied","Data":"c39fbc75dec887cf88799ba7b9ab067b88826cc2148b55e31c910e5a976ab31c"}
Nov 03 18:38:51.493736 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:51.493570    6059 scope.go:122] "RemoveContainer" containerID="e7f2ac08a319513ee43903c37064849b09c318de8b329414032d8719d3abb5dc"
Nov 03 18:38:51.503462 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:51.503438    6059 scope.go:122] "RemoveContainer" containerID="e7f2ac08a319513ee43903c37064849b09c318de8b329414032d8719d3abb5dc"
Nov 03 18:38:51.503736 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:38:51.503717    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7f2ac08a319513ee43903c37064849b09c318de8b329414032d8719d3abb5dc\": container with ID starting with e7f2ac08a319513ee43903c37064849b09c318de8b329414032d8719d3abb5dc not found: ID does not exist" containerID="e7f2ac08a319513ee43903c37064849b09c318de8b329414032d8719d3abb5dc"
Nov 03 18:38:51.503784 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:51.503746    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f2ac08a319513ee43903c37064849b09c318de8b329414032d8719d3abb5dc"} err="failed to get container status \"e7f2ac08a319513ee43903c37064849b09c318de8b329414032d8719d3abb5dc\": rpc error: code = NotFound desc = could not find container \"e7f2ac08a319513ee43903c37064849b09c318de8b329414032d8719d3abb5dc\": container with ID starting with e7f2ac08a319513ee43903c37064849b09c318de8b329414032d8719d3abb5dc not found: ID does not exist"
Nov 03 18:38:51.747415 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:51.747379    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c"]
Nov 03 18:38:51.751389 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:51.751345    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-774b6684ff-lpn7c"]
Nov 03 18:38:53.334688 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:53.334648    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce8cec85-8acc-48b4-b951-53bb664df3cb" path="/var/lib/kubelet/pods/ce8cec85-8acc-48b4-b951-53bb664df3cb/volumes"
Nov 03 18:38:54.718066 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:54.718036    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:54.884036 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:54.883998    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:58.352473 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:58.352436    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:58.528087 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:58.528037    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:59.018183 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:38:59.018146    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:39:00.883702 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:00.883667    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr"]
Nov 03 18:39:00.884338 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:00.884315    6059 cpu_manager.go:414] "containerMap: removing container" podUID="ce8cec85-8acc-48b4-b951-53bb664df3cb" containerName="payload-pre-processing"
Nov 03 18:39:00.884491 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:00.884338    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce8cec85-8acc-48b4-b951-53bb664df3cb" containerName="payload-pre-processing"
Nov 03 18:39:00.884553 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:00.884538    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce8cec85-8acc-48b4-b951-53bb664df3cb" containerName="payload-pre-processing"
Nov 03 18:39:00.889545 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:00.889525    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr"
Nov 03 18:39:00.943879 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:00.943841    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/350d22d5-8b69-4c89-8457-a63183b327e9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr\" (UID: \"350d22d5-8b69-4c89-8457-a63183b327e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr"
Nov 03 18:39:00.944051 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:00.943896    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbkkq\" (UniqueName: \"kubernetes.io/projected/350d22d5-8b69-4c89-8457-a63183b327e9-kube-api-access-nbkkq\") pod \"payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr\" (UID: \"350d22d5-8b69-4c89-8457-a63183b327e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr"
Nov 03 18:39:00.996832 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:00.996802    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-48c17a\""
Nov 03 18:39:00.997043 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:00.996827    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-48c17a-dockercfg-wnd7t\""
Nov 03 18:39:01.000373 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:01.000334    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr"]
Nov 03 18:39:01.054377 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:01.045021    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/350d22d5-8b69-4c89-8457-a63183b327e9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr\" (UID: \"350d22d5-8b69-4c89-8457-a63183b327e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr"
Nov 03 18:39:01.054377 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:01.045105    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbkkq\" (UniqueName: \"kubernetes.io/projected/350d22d5-8b69-4c89-8457-a63183b327e9-kube-api-access-nbkkq\") pod \"payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr\" (UID: \"350d22d5-8b69-4c89-8457-a63183b327e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr"
Nov 03 18:39:01.054377 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:01.046432    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/350d22d5-8b69-4c89-8457-a63183b327e9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr\" (UID: \"350d22d5-8b69-4c89-8457-a63183b327e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr"
Nov 03 18:39:01.112807 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:01.112778    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbkkq\" (UniqueName: \"kubernetes.io/projected/350d22d5-8b69-4c89-8457-a63183b327e9-kube-api-access-nbkkq\") pod \"payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr\" (UID: \"350d22d5-8b69-4c89-8457-a63183b327e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr"
Nov 03 18:39:01.203577 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:01.203488    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr"
Nov 03 18:39:01.603392 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:39:01.602494    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod350d22d5_8b69_4c89_8457_a63183b327e9.slice/crio-c7d99a709b576219a4c2c91637642b835ca2a28f999dc9ff9207a02a5123bedd WatchSource:0}: Error finding container c7d99a709b576219a4c2c91637642b835ca2a28f999dc9ff9207a02a5123bedd: Status 404 returned error can't find the container with id c7d99a709b576219a4c2c91637642b835ca2a28f999dc9ff9207a02a5123bedd
Nov 03 18:39:01.606873 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:01.606854    6059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:39:01.608118 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:01.608094    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr"]
Nov 03 18:39:01.711706 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:01.711670    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:39:02.566060 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:02.566023    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr" event={"ID":"350d22d5-8b69-4c89-8457-a63183b327e9","Type":"ContainerStarted","Data":"c7d99a709b576219a4c2c91637642b835ca2a28f999dc9ff9207a02a5123bedd"}
Nov 03 18:39:04.583010 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:04.582974    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr" event={"ID":"350d22d5-8b69-4c89-8457-a63183b327e9","Type":"ContainerStarted","Data":"e1cd5dab6f843a4431deb2d297f5350a1282323a73e7fe41e27134ea85910808"}
Nov 03 18:39:04.583512 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:04.583074    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr"
Nov 03 18:39:04.712265 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:04.712207    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr" podStartSLOduration=2.308716822 podStartE2EDuration="4.712191392s" totalImagesPullingTime="2.40347457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:39:04.708717624 +0000 UTC m=+4885.506430406" watchObservedRunningTime="2026-11-03 18:39:04.712191392 +0000 UTC m=+4885.509904167"
Nov 03 18:39:06.860805 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:39:06.860687    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:39:06.919402 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:06.919345    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:39:09.462483 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:09.462450    6059 patch_prober.go:29] interesting pod/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:39:09.464061 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:09.462575    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn" podUID="789edf94-9b0b-4fb8-869f-dc6791bf5a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:39:09.614142 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:09.614111    6059 generic.go:356] "Generic (PLEG): container finished" podID="789edf94-9b0b-4fb8-869f-dc6791bf5a92" containerID="75c5bc75240907d6128f7377f58ab171cf93208e87fd175ce59951b4de184e0b" exitCode=0
Nov 03 18:39:09.614315 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:09.614190    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn" event={"ID":"789edf94-9b0b-4fb8-869f-dc6791bf5a92","Type":"ContainerDied","Data":"75c5bc75240907d6128f7377f58ab171cf93208e87fd175ce59951b4de184e0b"}
Nov 03 18:39:09.614622 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:09.614604    6059 scope.go:122] "RemoveContainer" containerID="75c5bc75240907d6128f7377f58ab171cf93208e87fd175ce59951b4de184e0b"
Nov 03 18:39:09.616035 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:09.616011    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:39:09.616106 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:09.616081    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:39:09.616143 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:09.616114    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:39:10.621835 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:10.621803    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn" event={"ID":"789edf94-9b0b-4fb8-869f-dc6791bf5a92","Type":"ContainerStarted","Data":"0f80c69f06e52678311012dd97009a70fd16e6329ae54361478ee723a8eefb13"}
Nov 03 18:39:11.386723 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:11.386691    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:39:11.387022 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:11.387002    6059 patch_prober.go:29] interesting pod/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Nov 03 18:39:11.387074 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:11.387047    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn" podUID="789edf94-9b0b-4fb8-869f-dc6791bf5a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Nov 03 18:39:12.387180 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:12.387146    6059 patch_prober.go:29] interesting pod/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Nov 03 18:39:12.387610 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:12.387197    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn" podUID="789edf94-9b0b-4fb8-869f-dc6791bf5a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Nov 03 18:39:13.387025 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:13.386987    6059 patch_prober.go:29] interesting pod/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Nov 03 18:39:13.387215 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:13.387044    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn" podUID="789edf94-9b0b-4fb8-869f-dc6791bf5a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Nov 03 18:39:13.718233 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:13.718197    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:39:14.386769 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:14.386739    6059 patch_prober.go:29] interesting pod/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Nov 03 18:39:14.386945 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:14.386799    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn" podUID="789edf94-9b0b-4fb8-869f-dc6791bf5a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Nov 03 18:39:15.387683 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:15.387645    6059 patch_prober.go:29] interesting pod/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Nov 03 18:39:15.388069 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:15.387723    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn" podUID="789edf94-9b0b-4fb8-869f-dc6791bf5a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Nov 03 18:39:15.592231 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:15.592197    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr"
Nov 03 18:39:16.390878 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:16.390846    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:39:16.391390 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:16.391235    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:39:16.392029 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:16.392012    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:39:20.370590 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:20.370558    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:39:21.041591 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:21.041554    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:39:25.699135 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:25.699097    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:39:27.887147 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:27.887116    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-pm2wr_927401f9-153a-476e-8b41-7364a03d3cc8/payload-processing/0.log"
Nov 03 18:39:28.138765 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:28.138691    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-pm2wr_927401f9-153a-476e-8b41-7364a03d3cc8/payload-processing/0.log"
Nov 03 18:39:28.356466 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:28.356424    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr_350d22d5-8b69-4c89-8457-a63183b327e9/payload-processing/0.log"
Nov 03 18:39:31.116566 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:31.116528    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:39:32.156414 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:32.156377    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:39:32.409157 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:32.409055    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:39:33.051254 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:33.051209    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq"]
Nov 03 18:39:33.102829 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:33.102796    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb79a368-m82mq"]
Nov 03 18:39:33.335194 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:33.335101    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="396d6528-ddd5-4693-b335-8b88324034f3" path="/var/lib/kubelet/pods/396d6528-ddd5-4693-b335-8b88324034f3/volumes"
Nov 03 18:39:36.862532 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:39:36.862486    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:39:39.080763 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:39.080728    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:39:43.863255 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:43.863220    6059 scope.go:122] "RemoveContainer" containerID="c64e231a9fcc61843446270890ba703b438f5cd6706082df61f87847529ade05"
Nov 03 18:39:44.195895 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:44.195800    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:39:50.906673 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:50.906642    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:39:57.512377 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:57.512308    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:39:58.722705 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:39:58.722659    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:02.695555 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:40:02.695498    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:06.860935 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:40:06.860894    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:40:09.608508 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:40:09.608456    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:16.192346 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:40:16.192297    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:21.326375 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:40:21.326260    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:28.115722 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:40:28.115670    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:29.907178 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:40:29.907150    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-pm2wr_927401f9-153a-476e-8b41-7364a03d3cc8/payload-processing/0.log"
Nov 03 18:40:30.149076 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:40:30.149047    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-pm2wr_927401f9-153a-476e-8b41-7364a03d3cc8/payload-processing/0.log"
Nov 03 18:40:30.374333 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:40:30.374295    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr_350d22d5-8b69-4c89-8457-a63183b327e9/payload-processing/0.log"
Nov 03 18:40:30.609056 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:40:30.609024    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr_350d22d5-8b69-4c89-8457-a63183b327e9/payload-processing/0.log"
Nov 03 18:40:31.683144 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:40:31.683086    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:34.786336 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:40:34.786303    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:36.704546 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:40:36.704509    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:36.862092 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:40:36.862041    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:40:38.276254 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:40:38.276219    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:39.921931 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:40:39.921902    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:43.894463 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:40:43.894429    6059 scope.go:122] "RemoveContainer" containerID="ffb24cc19d361bfcec7e1ce8d896894f08448a3418769c7a21a1678cbaf3c9f9"
Nov 03 18:40:46.708689 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:40:46.708652    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:53.302558 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:40:53.302521    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:58.491462 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:40:58.491432    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:40:59.054868 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:40:59.054831    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:05.285258 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:05.285217    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:06.861100 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:41:06.861066    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:41:11.915128 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:11.915082    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:13.038046 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:13.038012    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:14.505203 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:14.505165    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:17.309015 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:17.308980    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:18.285853 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:18.285818    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:20.299849 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:20.299814    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:24.000514 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:24.000282    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:26.390536 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:26.390498    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:28.433832 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:28.433789    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:28.951973 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:28.951939    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:29.136246 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:29.136214    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:30.400870 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:30.400836    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:32.387989 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:32.387954    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:32.572592 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:32.572552    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:33.863618 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:33.863586    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:34.403522 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:34.403487    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"]
Nov 03 18:41:34.403790 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:34.403760    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9" podUID="ccf1c62f-ae8f-4486-8613-189132c5379a" containerName="istio-proxy" containerID="cri-o://0dda9d0a3adb0c27a7045118b5cbd5af6f1ffa5a7415712b1d1f115b1fa7fd68" gracePeriod=30
Nov 03 18:41:34.442501 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:34.442474    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:34.500101 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:34.500065    6059 patch_prober.go:29] interesting pod/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:41:34.500233 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:34.500127    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9" podUID="ccf1c62f-ae8f-4486-8613-189132c5379a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:41:36.861264 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:41:36.861222    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:41:37.485530 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:37.485440    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:39.665681 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.665653    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:41:39.762947 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.762911    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-token\" (UniqueName: \"kubernetes.io/projected/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-token\") pod \"ccf1c62f-ae8f-4486-8613-189132c5379a\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") "
Nov 03 18:41:39.763172 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.762986    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf1c62f-ae8f-4486-8613-189132c5379a-kube-api-access-2l7ch\" (UniqueName: \"kubernetes.io/projected/ccf1c62f-ae8f-4486-8613-189132c5379a-kube-api-access-2l7ch\") pod \"ccf1c62f-ae8f-4486-8613-189132c5379a\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") "
Nov 03 18:41:39.763172 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.763035    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccf1c62f-ae8f-4486-8613-189132c5379a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf1c62f-ae8f-4486-8613-189132c5379a-istiod-ca-cert\") pod \"ccf1c62f-ae8f-4486-8613-189132c5379a\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") "
Nov 03 18:41:39.763172 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.763071    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-workload-certs\") pod \"ccf1c62f-ae8f-4486-8613-189132c5379a\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") "
Nov 03 18:41:39.763390 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.763194    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-workload-socket\") pod \"ccf1c62f-ae8f-4486-8613-189132c5379a\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") "
Nov 03 18:41:39.763390 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.763236    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-credential-socket\") pod \"ccf1c62f-ae8f-4486-8613-189132c5379a\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") "
Nov 03 18:41:39.763390 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.763280    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-podinfo\") pod \"ccf1c62f-ae8f-4486-8613-189132c5379a\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") "
Nov 03 18:41:39.763390 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.763323    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-data\") pod \"ccf1c62f-ae8f-4486-8613-189132c5379a\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") "
Nov 03 18:41:39.763390 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.763367    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-envoy\") pod \"ccf1c62f-ae8f-4486-8613-189132c5379a\" (UID: \"ccf1c62f-ae8f-4486-8613-189132c5379a\") "
Nov 03 18:41:39.763707 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.763493    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-workload-socket" pod "ccf1c62f-ae8f-4486-8613-189132c5379a" (UID: "ccf1c62f-ae8f-4486-8613-189132c5379a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:41:39.763707 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.763508    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf1c62f-ae8f-4486-8613-189132c5379a-istiod-ca-cert" pod "ccf1c62f-ae8f-4486-8613-189132c5379a" (UID: "ccf1c62f-ae8f-4486-8613-189132c5379a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:41:39.763822 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.763704    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-workload-certs" pod "ccf1c62f-ae8f-4486-8613-189132c5379a" (UID: "ccf1c62f-ae8f-4486-8613-189132c5379a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:41:39.763822 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.763715    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-data" pod "ccf1c62f-ae8f-4486-8613-189132c5379a" (UID: "ccf1c62f-ae8f-4486-8613-189132c5379a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:41:39.763822 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.763779    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-credential-socket" pod "ccf1c62f-ae8f-4486-8613-189132c5379a" (UID: "ccf1c62f-ae8f-4486-8613-189132c5379a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:41:39.764005 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.763978    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf1c62f-ae8f-4486-8613-189132c5379a-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:41:39.764005 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.764001    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:41:39.764135 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.764012    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:41:39.764135 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.764027    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:41:39.764135 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.764038    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:41:39.765302 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.765266    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-token" pod "ccf1c62f-ae8f-4486-8613-189132c5379a" (UID: "ccf1c62f-ae8f-4486-8613-189132c5379a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:41:39.765526 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.765504    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf1c62f-ae8f-4486-8613-189132c5379a-kube-api-access-2l7ch" pod "ccf1c62f-ae8f-4486-8613-189132c5379a" (UID: "ccf1c62f-ae8f-4486-8613-189132c5379a"). InnerVolumeSpecName "kube-api-access-2l7ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:41:39.765624 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.765566    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-podinfo" pod "ccf1c62f-ae8f-4486-8613-189132c5379a" (UID: "ccf1c62f-ae8f-4486-8613-189132c5379a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:41:39.765692 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.765679    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-envoy" pod "ccf1c62f-ae8f-4486-8613-189132c5379a" (UID: "ccf1c62f-ae8f-4486-8613-189132c5379a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:41:39.865020 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.864982    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:41:39.865020 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.865017    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l7ch\" (UniqueName: \"kubernetes.io/projected/ccf1c62f-ae8f-4486-8613-189132c5379a-kube-api-access-2l7ch\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:41:39.865233 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.865034    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:41:39.865233 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:39.865047    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf1c62f-ae8f-4486-8613-189132c5379a-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:41:40.058833 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:40.058749    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:40.485739 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:40.485702    6059 generic.go:356] "Generic (PLEG): container finished" podID="ccf1c62f-ae8f-4486-8613-189132c5379a" containerID="0dda9d0a3adb0c27a7045118b5cbd5af6f1ffa5a7415712b1d1f115b1fa7fd68" exitCode=0
Nov 03 18:41:40.485952 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:40.485748    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"
Nov 03 18:41:40.485952 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:40.485744    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9" event={"ID":"ccf1c62f-ae8f-4486-8613-189132c5379a","Type":"ContainerDied","Data":"0dda9d0a3adb0c27a7045118b5cbd5af6f1ffa5a7415712b1d1f115b1fa7fd68"}
Nov 03 18:41:40.485952 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:40.485813    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9" event={"ID":"ccf1c62f-ae8f-4486-8613-189132c5379a","Type":"ContainerDied","Data":"1e863ca3f5f8209440359598182452dd6e6c6be80c5e19ec9f439a6ad5dd1afd"}
Nov 03 18:41:40.485952 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:40.485842    6059 scope.go:122] "RemoveContainer" containerID="0dda9d0a3adb0c27a7045118b5cbd5af6f1ffa5a7415712b1d1f115b1fa7fd68"
Nov 03 18:41:40.495920 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:40.495902    6059 scope.go:122] "RemoveContainer" containerID="86f79ad9d97f27e2546d6985dd93a6a668d837efa0d1ee2b220eb94368fae315"
Nov 03 18:41:40.504176 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:40.504159    6059 scope.go:122] "RemoveContainer" containerID="0dda9d0a3adb0c27a7045118b5cbd5af6f1ffa5a7415712b1d1f115b1fa7fd68"
Nov 03 18:41:40.504455 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:41:40.504436    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dda9d0a3adb0c27a7045118b5cbd5af6f1ffa5a7415712b1d1f115b1fa7fd68\": container with ID starting with 0dda9d0a3adb0c27a7045118b5cbd5af6f1ffa5a7415712b1d1f115b1fa7fd68 not found: ID does not exist" containerID="0dda9d0a3adb0c27a7045118b5cbd5af6f1ffa5a7415712b1d1f115b1fa7fd68"
Nov 03 18:41:40.504506 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:40.504463    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dda9d0a3adb0c27a7045118b5cbd5af6f1ffa5a7415712b1d1f115b1fa7fd68"} err="failed to get container status \"0dda9d0a3adb0c27a7045118b5cbd5af6f1ffa5a7415712b1d1f115b1fa7fd68\": rpc error: code = NotFound desc = could not find container \"0dda9d0a3adb0c27a7045118b5cbd5af6f1ffa5a7415712b1d1f115b1fa7fd68\": container with ID starting with 0dda9d0a3adb0c27a7045118b5cbd5af6f1ffa5a7415712b1d1f115b1fa7fd68 not found: ID does not exist"
Nov 03 18:41:40.504506 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:40.504483    6059 scope.go:122] "RemoveContainer" containerID="86f79ad9d97f27e2546d6985dd93a6a668d837efa0d1ee2b220eb94368fae315"
Nov 03 18:41:40.504725 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:41:40.504705    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f79ad9d97f27e2546d6985dd93a6a668d837efa0d1ee2b220eb94368fae315\": container with ID starting with 86f79ad9d97f27e2546d6985dd93a6a668d837efa0d1ee2b220eb94368fae315 not found: ID does not exist" containerID="86f79ad9d97f27e2546d6985dd93a6a668d837efa0d1ee2b220eb94368fae315"
Nov 03 18:41:40.504794 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:40.504739    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f79ad9d97f27e2546d6985dd93a6a668d837efa0d1ee2b220eb94368fae315"} err="failed to get container status \"86f79ad9d97f27e2546d6985dd93a6a668d837efa0d1ee2b220eb94368fae315\": rpc error: code = NotFound desc = could not find container \"86f79ad9d97f27e2546d6985dd93a6a668d837efa0d1ee2b220eb94368fae315\": container with ID starting with 86f79ad9d97f27e2546d6985dd93a6a668d837efa0d1ee2b220eb94368fae315 not found: ID does not exist"
Nov 03 18:41:40.644379 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:40.644324    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"]
Nov 03 18:41:40.691939 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:40.691903    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ee65c-openshift-default-8555d5fb6c-kvcr9"]
Nov 03 18:41:41.335518 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:41.335484    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccf1c62f-ae8f-4486-8613-189132c5379a" path="/var/lib/kubelet/pods/ccf1c62f-ae8f-4486-8613-189132c5379a/volumes"
Nov 03 18:41:42.696343 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:42.696260    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:46.194179 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:46.194139    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:49.047708 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:49.047676    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-229pb"]
Nov 03 18:41:49.095127 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:49.095089    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-229pb"]
Nov 03 18:41:49.335055 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:49.334975    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e926ddfa-faa4-41b4-900e-272c2c15145b" path="/var/lib/kubelet/pods/e926ddfa-faa4-41b4-900e-272c2c15145b/volumes"
Nov 03 18:41:49.841018 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:49.840981    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:54.499653 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:54.499618    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr"]
Nov 03 18:41:54.500106 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:54.499803    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr" podUID="350d22d5-8b69-4c89-8457-a63183b327e9" containerName="payload-processing" containerID="cri-o://e1cd5dab6f843a4431deb2d297f5350a1282323a73e7fe41e27134ea85910808" gracePeriod=30
Nov 03 18:41:54.769029 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:54.769004    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr"
Nov 03 18:41:54.822402 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:54.822350    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/350d22d5-8b69-4c89-8457-a63183b327e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/350d22d5-8b69-4c89-8457-a63183b327e9-plugins-config-volume\") pod \"350d22d5-8b69-4c89-8457-a63183b327e9\" (UID: \"350d22d5-8b69-4c89-8457-a63183b327e9\") "
Nov 03 18:41:54.822566 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:54.822514    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/350d22d5-8b69-4c89-8457-a63183b327e9-kube-api-access-nbkkq\" (UniqueName: \"kubernetes.io/projected/350d22d5-8b69-4c89-8457-a63183b327e9-kube-api-access-nbkkq\") pod \"350d22d5-8b69-4c89-8457-a63183b327e9\" (UID: \"350d22d5-8b69-4c89-8457-a63183b327e9\") "
Nov 03 18:41:54.822690 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:54.822667    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/350d22d5-8b69-4c89-8457-a63183b327e9-plugins-config-volume" pod "350d22d5-8b69-4c89-8457-a63183b327e9" (UID: "350d22d5-8b69-4c89-8457-a63183b327e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:41:54.822862 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:54.822847    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/350d22d5-8b69-4c89-8457-a63183b327e9-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:41:54.824512 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:54.824487    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/350d22d5-8b69-4c89-8457-a63183b327e9-kube-api-access-nbkkq" pod "350d22d5-8b69-4c89-8457-a63183b327e9" (UID: "350d22d5-8b69-4c89-8457-a63183b327e9"). InnerVolumeSpecName "kube-api-access-nbkkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:41:54.923671 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:54.923633    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbkkq\" (UniqueName: \"kubernetes.io/projected/350d22d5-8b69-4c89-8457-a63183b327e9-kube-api-access-nbkkq\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:41:55.565939 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:55.565905    6059 generic.go:356] "Generic (PLEG): container finished" podID="350d22d5-8b69-4c89-8457-a63183b327e9" containerID="e1cd5dab6f843a4431deb2d297f5350a1282323a73e7fe41e27134ea85910808" exitCode=0
Nov 03 18:41:55.566326 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:55.565947    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr"
Nov 03 18:41:55.566326 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:55.565999    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr" event={"ID":"350d22d5-8b69-4c89-8457-a63183b327e9","Type":"ContainerDied","Data":"e1cd5dab6f843a4431deb2d297f5350a1282323a73e7fe41e27134ea85910808"}
Nov 03 18:41:55.566326 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:55.566050    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr" event={"ID":"350d22d5-8b69-4c89-8457-a63183b327e9","Type":"ContainerDied","Data":"c7d99a709b576219a4c2c91637642b835ca2a28f999dc9ff9207a02a5123bedd"}
Nov 03 18:41:55.566326 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:55.566072    6059 scope.go:122] "RemoveContainer" containerID="e1cd5dab6f843a4431deb2d297f5350a1282323a73e7fe41e27134ea85910808"
Nov 03 18:41:55.575300 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:55.575277    6059 scope.go:122] "RemoveContainer" containerID="e1cd5dab6f843a4431deb2d297f5350a1282323a73e7fe41e27134ea85910808"
Nov 03 18:41:55.575560 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:41:55.575537    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1cd5dab6f843a4431deb2d297f5350a1282323a73e7fe41e27134ea85910808\": container with ID starting with e1cd5dab6f843a4431deb2d297f5350a1282323a73e7fe41e27134ea85910808 not found: ID does not exist" containerID="e1cd5dab6f843a4431deb2d297f5350a1282323a73e7fe41e27134ea85910808"
Nov 03 18:41:55.575633 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:55.575570    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1cd5dab6f843a4431deb2d297f5350a1282323a73e7fe41e27134ea85910808"} err="failed to get container status \"e1cd5dab6f843a4431deb2d297f5350a1282323a73e7fe41e27134ea85910808\": rpc error: code = NotFound desc = could not find container \"e1cd5dab6f843a4431deb2d297f5350a1282323a73e7fe41e27134ea85910808\": container with ID starting with e1cd5dab6f843a4431deb2d297f5350a1282323a73e7fe41e27134ea85910808 not found: ID does not exist"
Nov 03 18:41:55.730245 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:55.730212    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr"]
Nov 03 18:41:55.762889 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:55.762857    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48c17a-5cc4fbd9d8-txqhr"]
Nov 03 18:41:56.360869 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:56.360837    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:56.528984 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:56.528952    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:41:57.334617 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:57.334583    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="350d22d5-8b69-4c89-8457-a63183b327e9" path="/var/lib/kubelet/pods/350d22d5-8b69-4c89-8457-a63183b327e9/volumes"
Nov 03 18:41:59.416348 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:41:59.416318    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:00.672755 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:00.672722    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:00.850869 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:00.850834    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:02.318646 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:02.318603    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:02.782759 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:02.782727    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"]
Nov 03 18:42:02.782992 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:02.782953    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn" podUID="789edf94-9b0b-4fb8-869f-dc6791bf5a92" containerName="istio-proxy" containerID="cri-o://0f80c69f06e52678311012dd97009a70fd16e6329ae54361478ee723a8eefb13" gracePeriod=30
Nov 03 18:42:02.819279 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:02.819249    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:04.885468 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:04.885429    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr"]
Nov 03 18:42:04.886062 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:04.886040    6059 cpu_manager.go:414] "containerMap: removing container" podUID="350d22d5-8b69-4c89-8457-a63183b327e9" containerName="payload-processing"
Nov 03 18:42:04.886156 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:04.886065    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="350d22d5-8b69-4c89-8457-a63183b327e9" containerName="payload-processing"
Nov 03 18:42:04.886156 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:04.886082    6059 cpu_manager.go:414] "containerMap: removing container" podUID="ccf1c62f-ae8f-4486-8613-189132c5379a" containerName="istio-proxy"
Nov 03 18:42:04.886156 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:04.886089    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf1c62f-ae8f-4486-8613-189132c5379a" containerName="istio-proxy"
Nov 03 18:42:04.886327 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:04.886245    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf1c62f-ae8f-4486-8613-189132c5379a" containerName="istio-proxy"
Nov 03 18:42:04.886327 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:04.886261    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="350d22d5-8b69-4c89-8457-a63183b327e9" containerName="payload-processing"
Nov 03 18:42:04.886327 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:04.886271    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf1c62f-ae8f-4486-8613-189132c5379a" containerName="istio-proxy"
Nov 03 18:42:04.891342 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:04.891321    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr"
Nov 03 18:42:04.949223 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:04.949190    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr"]
Nov 03 18:42:05.069932 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:05.069896    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9xz2\" (UniqueName: \"kubernetes.io/projected/b4468af4-573a-4d81-8a9c-542768dbfe69-kube-api-access-f9xz2\") pod \"payload-pre-processing-c7cb977cb-gs6zr\" (UID: \"b4468af4-573a-4d81-8a9c-542768dbfe69\") " pod="openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr"
Nov 03 18:42:05.070113 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:05.070043    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4468af4-573a-4d81-8a9c-542768dbfe69-plugins-config-volume\") pod \"payload-pre-processing-c7cb977cb-gs6zr\" (UID: \"b4468af4-573a-4d81-8a9c-542768dbfe69\") " pod="openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr"
Nov 03 18:42:05.170998 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:05.170915    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9xz2\" (UniqueName: \"kubernetes.io/projected/b4468af4-573a-4d81-8a9c-542768dbfe69-kube-api-access-f9xz2\") pod \"payload-pre-processing-c7cb977cb-gs6zr\" (UID: \"b4468af4-573a-4d81-8a9c-542768dbfe69\") " pod="openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr"
Nov 03 18:42:05.171185 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:05.171016    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4468af4-573a-4d81-8a9c-542768dbfe69-plugins-config-volume\") pod \"payload-pre-processing-c7cb977cb-gs6zr\" (UID: \"b4468af4-573a-4d81-8a9c-542768dbfe69\") " pod="openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr"
Nov 03 18:42:05.171785 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:05.171759    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4468af4-573a-4d81-8a9c-542768dbfe69-plugins-config-volume\") pod \"payload-pre-processing-c7cb977cb-gs6zr\" (UID: \"b4468af4-573a-4d81-8a9c-542768dbfe69\") " pod="openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr"
Nov 03 18:42:05.229823 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:05.229794    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9xz2\" (UniqueName: \"kubernetes.io/projected/b4468af4-573a-4d81-8a9c-542768dbfe69-kube-api-access-f9xz2\") pod \"payload-pre-processing-c7cb977cb-gs6zr\" (UID: \"b4468af4-573a-4d81-8a9c-542768dbfe69\") " pod="openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr"
Nov 03 18:42:05.503992 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:05.503960    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr"
Nov 03 18:42:05.878609 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:05.878584    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr"]
Nov 03 18:42:05.881080 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:42:05.881053    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4468af4_573a_4d81_8a9c_542768dbfe69.slice/crio-36c57126445ac25d0a48bae3adec6a58edf4bdcc199d262b4d3474f4c1bdea68 WatchSource:0}: Error finding container 36c57126445ac25d0a48bae3adec6a58edf4bdcc199d262b4d3474f4c1bdea68: Status 404 returned error can't find the container with id 36c57126445ac25d0a48bae3adec6a58edf4bdcc199d262b4d3474f4c1bdea68
Nov 03 18:42:06.632705 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:06.632674    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr" event={"ID":"b4468af4-573a-4d81-8a9c-542768dbfe69","Type":"ContainerStarted","Data":"8efe8e534fc670effc2e97de42e47c61be26535df663c24c647e7da17f5926b4"}
Nov 03 18:42:06.633111 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:06.632722    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr" event={"ID":"b4468af4-573a-4d81-8a9c-542768dbfe69","Type":"ContainerStarted","Data":"36c57126445ac25d0a48bae3adec6a58edf4bdcc199d262b4d3474f4c1bdea68"}
Nov 03 18:42:06.633111 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:06.632792    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr"
Nov 03 18:42:06.745938 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:06.745887    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr" podStartSLOduration=2.7458735990000003 podStartE2EDuration="2.745873599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:42:06.743637635 +0000 UTC m=+5067.541350407" watchObservedRunningTime="2026-11-03 18:42:06.745873599 +0000 UTC m=+5067.543586379"
Nov 03 18:42:06.861169 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:42:06.861135    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:42:08.049796 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.049738    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:42:08.204509 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.204473    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-credential-socket\") pod \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") "
Nov 03 18:42:08.204509 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.204511    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-podinfo\") pod \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") "
Nov 03 18:42:08.204735 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.204564    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-data\") pod \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") "
Nov 03 18:42:08.204735 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.204603    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789edf94-9b0b-4fb8-869f-dc6791bf5a92-kube-api-access-lkxsh\" (UniqueName: \"kubernetes.io/projected/789edf94-9b0b-4fb8-869f-dc6791bf5a92-kube-api-access-lkxsh\") pod \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") "
Nov 03 18:42:08.204735 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.204699    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-workload-socket\") pod \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") "
Nov 03 18:42:08.204919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.204734    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-workload-certs\") pod \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") "
Nov 03 18:42:08.204919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.204785    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-token\" (UniqueName: \"kubernetes.io/projected/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-token\") pod \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") "
Nov 03 18:42:08.204919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.204795    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-credential-socket" pod "789edf94-9b0b-4fb8-869f-dc6791bf5a92" (UID: "789edf94-9b0b-4fb8-869f-dc6791bf5a92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:42:08.204919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.204814    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istiod-ca-cert\") pod \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") "
Nov 03 18:42:08.204919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.204838    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-envoy\") pod \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\" (UID: \"789edf94-9b0b-4fb8-869f-dc6791bf5a92\") "
Nov 03 18:42:08.204919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.204897    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-data" pod "789edf94-9b0b-4fb8-869f-dc6791bf5a92" (UID: "789edf94-9b0b-4fb8-869f-dc6791bf5a92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:42:08.205222 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.204982    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-workload-socket" pod "789edf94-9b0b-4fb8-869f-dc6791bf5a92" (UID: "789edf94-9b0b-4fb8-869f-dc6791bf5a92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:42:08.205222 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.205060    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-workload-certs" pod "789edf94-9b0b-4fb8-869f-dc6791bf5a92" (UID: "789edf94-9b0b-4fb8-869f-dc6791bf5a92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:42:08.205222 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.205133    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istiod-ca-cert" pod "789edf94-9b0b-4fb8-869f-dc6791bf5a92" (UID: "789edf94-9b0b-4fb8-869f-dc6791bf5a92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:42:08.205426 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.205300    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:42:08.205426 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.205320    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:42:08.205426 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.205334    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:42:08.205426 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.205351    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:42:08.205426 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.205386    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:42:08.206952 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.206926    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-podinfo" pod "789edf94-9b0b-4fb8-869f-dc6791bf5a92" (UID: "789edf94-9b0b-4fb8-869f-dc6791bf5a92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:42:08.207059 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.206930    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-envoy" pod "789edf94-9b0b-4fb8-869f-dc6791bf5a92" (UID: "789edf94-9b0b-4fb8-869f-dc6791bf5a92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:42:08.207099 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.207051    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-token" pod "789edf94-9b0b-4fb8-869f-dc6791bf5a92" (UID: "789edf94-9b0b-4fb8-869f-dc6791bf5a92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:42:08.207099 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.207080    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789edf94-9b0b-4fb8-869f-dc6791bf5a92-kube-api-access-lkxsh" pod "789edf94-9b0b-4fb8-869f-dc6791bf5a92" (UID: "789edf94-9b0b-4fb8-869f-dc6791bf5a92"). InnerVolumeSpecName "kube-api-access-lkxsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:42:08.306297 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.306218    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:42:08.306297 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.306247    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:42:08.306297 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.306257    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/789edf94-9b0b-4fb8-869f-dc6791bf5a92-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:42:08.306297 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.306268    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkxsh\" (UniqueName: \"kubernetes.io/projected/789edf94-9b0b-4fb8-869f-dc6791bf5a92-kube-api-access-lkxsh\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:42:08.644090 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.643998    6059 generic.go:356] "Generic (PLEG): container finished" podID="789edf94-9b0b-4fb8-869f-dc6791bf5a92" containerID="0f80c69f06e52678311012dd97009a70fd16e6329ae54361478ee723a8eefb13" exitCode=0
Nov 03 18:42:08.644090 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.644049    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"
Nov 03 18:42:08.644090 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.644076    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn" event={"ID":"789edf94-9b0b-4fb8-869f-dc6791bf5a92","Type":"ContainerDied","Data":"0f80c69f06e52678311012dd97009a70fd16e6329ae54361478ee723a8eefb13"}
Nov 03 18:42:08.644376 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.644138    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn" event={"ID":"789edf94-9b0b-4fb8-869f-dc6791bf5a92","Type":"ContainerDied","Data":"3b68bd5d02f8328927ae1e59390878a2f2709e62e9dc03c49b0babec639b57d8"}
Nov 03 18:42:08.644376 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.644161    6059 scope.go:122] "RemoveContainer" containerID="0f80c69f06e52678311012dd97009a70fd16e6329ae54361478ee723a8eefb13"
Nov 03 18:42:08.654469 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.654425    6059 scope.go:122] "RemoveContainer" containerID="75c5bc75240907d6128f7377f58ab171cf93208e87fd175ce59951b4de184e0b"
Nov 03 18:42:08.662925 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.662908    6059 scope.go:122] "RemoveContainer" containerID="0f80c69f06e52678311012dd97009a70fd16e6329ae54361478ee723a8eefb13"
Nov 03 18:42:08.663166 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:42:08.663150    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f80c69f06e52678311012dd97009a70fd16e6329ae54361478ee723a8eefb13\": container with ID starting with 0f80c69f06e52678311012dd97009a70fd16e6329ae54361478ee723a8eefb13 not found: ID does not exist" containerID="0f80c69f06e52678311012dd97009a70fd16e6329ae54361478ee723a8eefb13"
Nov 03 18:42:08.663209 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.663174    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f80c69f06e52678311012dd97009a70fd16e6329ae54361478ee723a8eefb13"} err="failed to get container status \"0f80c69f06e52678311012dd97009a70fd16e6329ae54361478ee723a8eefb13\": rpc error: code = NotFound desc = could not find container \"0f80c69f06e52678311012dd97009a70fd16e6329ae54361478ee723a8eefb13\": container with ID starting with 0f80c69f06e52678311012dd97009a70fd16e6329ae54361478ee723a8eefb13 not found: ID does not exist"
Nov 03 18:42:08.663209 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.663194    6059 scope.go:122] "RemoveContainer" containerID="75c5bc75240907d6128f7377f58ab171cf93208e87fd175ce59951b4de184e0b"
Nov 03 18:42:08.663420 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:42:08.663403    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c5bc75240907d6128f7377f58ab171cf93208e87fd175ce59951b4de184e0b\": container with ID starting with 75c5bc75240907d6128f7377f58ab171cf93208e87fd175ce59951b4de184e0b not found: ID does not exist" containerID="75c5bc75240907d6128f7377f58ab171cf93208e87fd175ce59951b4de184e0b"
Nov 03 18:42:08.663477 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.663426    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c5bc75240907d6128f7377f58ab171cf93208e87fd175ce59951b4de184e0b"} err="failed to get container status \"75c5bc75240907d6128f7377f58ab171cf93208e87fd175ce59951b4de184e0b\": rpc error: code = NotFound desc = could not find container \"75c5bc75240907d6128f7377f58ab171cf93208e87fd175ce59951b4de184e0b\": container with ID starting with 75c5bc75240907d6128f7377f58ab171cf93208e87fd175ce59951b4de184e0b not found: ID does not exist"
Nov 03 18:42:08.802311 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.802280    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"]
Nov 03 18:42:08.849695 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:08.849667    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-48c17a-openshift-default-864bd8f9d9-4h2qn"]
Nov 03 18:42:09.335047 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:09.335008    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789edf94-9b0b-4fb8-869f-dc6791bf5a92" path="/var/lib/kubelet/pods/789edf94-9b0b-4fb8-869f-dc6791bf5a92/volumes"
Nov 03 18:42:16.047447 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:16.047413    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh"]
Nov 03 18:42:16.094668 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:16.094637    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bjwbh"]
Nov 03 18:42:17.335253 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:17.335212    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e0f10ff-13d9-4d5f-8420-e03866140ffd" path="/var/lib/kubelet/pods/8e0f10ff-13d9-4d5f-8420-e03866140ffd/volumes"
Nov 03 18:42:17.641269 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:17.641187    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr"
Nov 03 18:42:17.801273 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:17.801242    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7"]
Nov 03 18:42:19.870157 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:19.870125    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-6964cbcfd5-f44z9"]
Nov 03 18:42:19.870576 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:19.870563    6059 cpu_manager.go:414] "containerMap: removing container" podUID="789edf94-9b0b-4fb8-869f-dc6791bf5a92" containerName="istio-proxy"
Nov 03 18:42:19.870619 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:19.870577    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789edf94-9b0b-4fb8-869f-dc6791bf5a92" containerName="istio-proxy"
Nov 03 18:42:19.870619 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:19.870588    6059 cpu_manager.go:414] "containerMap: removing container" podUID="ccf1c62f-ae8f-4486-8613-189132c5379a" containerName="istio-proxy"
Nov 03 18:42:19.870619 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:19.870596    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf1c62f-ae8f-4486-8613-189132c5379a" containerName="istio-proxy"
Nov 03 18:42:19.870724 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:19.870702    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="789edf94-9b0b-4fb8-869f-dc6791bf5a92" containerName="istio-proxy"
Nov 03 18:42:19.870724 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:19.870713    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="789edf94-9b0b-4fb8-869f-dc6791bf5a92" containerName="istio-proxy"
Nov 03 18:42:19.875526 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:19.875504    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6964cbcfd5-f44z9"
Nov 03 18:42:19.919749 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:19.919718    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb0a010-326d-4b02-8fcd-d3067abb323a-plugins-config-volume\") pod \"payload-processing-6964cbcfd5-f44z9\" (UID: \"bfb0a010-326d-4b02-8fcd-d3067abb323a\") " pod="openshift-ingress/payload-processing-6964cbcfd5-f44z9"
Nov 03 18:42:19.919897 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:19.919769    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpgp9\" (UniqueName: \"kubernetes.io/projected/bfb0a010-326d-4b02-8fcd-d3067abb323a-kube-api-access-lpgp9\") pod \"payload-processing-6964cbcfd5-f44z9\" (UID: \"bfb0a010-326d-4b02-8fcd-d3067abb323a\") " pod="openshift-ingress/payload-processing-6964cbcfd5-f44z9"
Nov 03 18:42:19.934077 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:19.934048    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6964cbcfd5-f44z9"]
Nov 03 18:42:20.020628 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:20.020585    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb0a010-326d-4b02-8fcd-d3067abb323a-plugins-config-volume\") pod \"payload-processing-6964cbcfd5-f44z9\" (UID: \"bfb0a010-326d-4b02-8fcd-d3067abb323a\") " pod="openshift-ingress/payload-processing-6964cbcfd5-f44z9"
Nov 03 18:42:20.020628 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:20.020642    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpgp9\" (UniqueName: \"kubernetes.io/projected/bfb0a010-326d-4b02-8fcd-d3067abb323a-kube-api-access-lpgp9\") pod \"payload-processing-6964cbcfd5-f44z9\" (UID: \"bfb0a010-326d-4b02-8fcd-d3067abb323a\") " pod="openshift-ingress/payload-processing-6964cbcfd5-f44z9"
Nov 03 18:42:20.021391 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:20.021349    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb0a010-326d-4b02-8fcd-d3067abb323a-plugins-config-volume\") pod \"payload-processing-6964cbcfd5-f44z9\" (UID: \"bfb0a010-326d-4b02-8fcd-d3067abb323a\") " pod="openshift-ingress/payload-processing-6964cbcfd5-f44z9"
Nov 03 18:42:20.079801 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:20.079767    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpgp9\" (UniqueName: \"kubernetes.io/projected/bfb0a010-326d-4b02-8fcd-d3067abb323a-kube-api-access-lpgp9\") pod \"payload-processing-6964cbcfd5-f44z9\" (UID: \"bfb0a010-326d-4b02-8fcd-d3067abb323a\") " pod="openshift-ingress/payload-processing-6964cbcfd5-f44z9"
Nov 03 18:42:20.187804 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:20.187719    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6964cbcfd5-f44z9"
Nov 03 18:42:20.558287 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:20.558259    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6964cbcfd5-f44z9"]
Nov 03 18:42:20.560417 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:42:20.560344    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb0a010_326d_4b02_8fcd_d3067abb323a.slice/crio-bbd4658a23f228ce18a3df015a9751e32c1580e9df345dc35bb67be1f5348fd3 WatchSource:0}: Error finding container bbd4658a23f228ce18a3df015a9751e32c1580e9df345dc35bb67be1f5348fd3: Status 404 returned error can't find the container with id bbd4658a23f228ce18a3df015a9751e32c1580e9df345dc35bb67be1f5348fd3
Nov 03 18:42:20.708236 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:20.708200    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6964cbcfd5-f44z9" event={"ID":"bfb0a010-326d-4b02-8fcd-d3067abb323a","Type":"ContainerStarted","Data":"b87789fcd80769725396e8f06614e8622f3b6af31094e15209e40205d8f1a488"}
Nov 03 18:42:20.708417 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:20.708255    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6964cbcfd5-f44z9" event={"ID":"bfb0a010-326d-4b02-8fcd-d3067abb323a","Type":"ContainerStarted","Data":"bbd4658a23f228ce18a3df015a9751e32c1580e9df345dc35bb67be1f5348fd3"}
Nov 03 18:42:20.708417 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:20.708315    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-6964cbcfd5-f44z9"
Nov 03 18:42:20.820464 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:20.820364    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-6964cbcfd5-f44z9" podStartSLOduration=1.820338693 podStartE2EDuration="1.820338693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:42:20.818965459 +0000 UTC m=+5081.616678242" watchObservedRunningTime="2026-11-03 18:42:20.820338693 +0000 UTC m=+5081.618051474"
Nov 03 18:42:22.802314 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:22.802277    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7" podUID="41eba073-3e2a-4b3e-acba-d68aad192e43" containerName="payload-pre-processing" containerID="cri-o://11a873048266de01db216e89377defabe2bbb57e9ed3793756cdc1a953741184" gracePeriod=25
Nov 03 18:42:23.052879 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:23.052822    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7"
Nov 03 18:42:23.154173 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:23.154138    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41eba073-3e2a-4b3e-acba-d68aad192e43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41eba073-3e2a-4b3e-acba-d68aad192e43-plugins-config-volume\") pod \"41eba073-3e2a-4b3e-acba-d68aad192e43\" (UID: \"41eba073-3e2a-4b3e-acba-d68aad192e43\") "
Nov 03 18:42:23.154346 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:23.154297    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41eba073-3e2a-4b3e-acba-d68aad192e43-kube-api-access-qbcnx\" (UniqueName: \"kubernetes.io/projected/41eba073-3e2a-4b3e-acba-d68aad192e43-kube-api-access-qbcnx\") pod \"41eba073-3e2a-4b3e-acba-d68aad192e43\" (UID: \"41eba073-3e2a-4b3e-acba-d68aad192e43\") "
Nov 03 18:42:23.154552 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:23.154529    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41eba073-3e2a-4b3e-acba-d68aad192e43-plugins-config-volume" pod "41eba073-3e2a-4b3e-acba-d68aad192e43" (UID: "41eba073-3e2a-4b3e-acba-d68aad192e43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:42:23.156340 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:23.156318    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41eba073-3e2a-4b3e-acba-d68aad192e43-kube-api-access-qbcnx" pod "41eba073-3e2a-4b3e-acba-d68aad192e43" (UID: "41eba073-3e2a-4b3e-acba-d68aad192e43"). InnerVolumeSpecName "kube-api-access-qbcnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:42:23.255594 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:23.255561    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbcnx\" (UniqueName: \"kubernetes.io/projected/41eba073-3e2a-4b3e-acba-d68aad192e43-kube-api-access-qbcnx\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:42:23.255594 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:23.255589    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41eba073-3e2a-4b3e-acba-d68aad192e43-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:42:23.724409 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:23.724375    6059 generic.go:356] "Generic (PLEG): container finished" podID="41eba073-3e2a-4b3e-acba-d68aad192e43" containerID="11a873048266de01db216e89377defabe2bbb57e9ed3793756cdc1a953741184" exitCode=0
Nov 03 18:42:23.724556 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:23.724414    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7"
Nov 03 18:42:23.724556 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:23.724414    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7" event={"ID":"41eba073-3e2a-4b3e-acba-d68aad192e43","Type":"ContainerDied","Data":"11a873048266de01db216e89377defabe2bbb57e9ed3793756cdc1a953741184"}
Nov 03 18:42:23.724556 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:23.724546    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7" event={"ID":"41eba073-3e2a-4b3e-acba-d68aad192e43","Type":"ContainerDied","Data":"6ca4a2b39ac1a0361e93c0ca374a0be0bceed779783f95188418d90f1b4ef18d"}
Nov 03 18:42:23.724717 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:23.724576    6059 scope.go:122] "RemoveContainer" containerID="11a873048266de01db216e89377defabe2bbb57e9ed3793756cdc1a953741184"
Nov 03 18:42:23.733823 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:23.733802    6059 scope.go:122] "RemoveContainer" containerID="11a873048266de01db216e89377defabe2bbb57e9ed3793756cdc1a953741184"
Nov 03 18:42:23.734072 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:42:23.734053    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a873048266de01db216e89377defabe2bbb57e9ed3793756cdc1a953741184\": container with ID starting with 11a873048266de01db216e89377defabe2bbb57e9ed3793756cdc1a953741184 not found: ID does not exist" containerID="11a873048266de01db216e89377defabe2bbb57e9ed3793756cdc1a953741184"
Nov 03 18:42:23.734142 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:23.734077    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a873048266de01db216e89377defabe2bbb57e9ed3793756cdc1a953741184"} err="failed to get container status \"11a873048266de01db216e89377defabe2bbb57e9ed3793756cdc1a953741184\": rpc error: code = NotFound desc = could not find container \"11a873048266de01db216e89377defabe2bbb57e9ed3793756cdc1a953741184\": container with ID starting with 11a873048266de01db216e89377defabe2bbb57e9ed3793756cdc1a953741184 not found: ID does not exist"
Nov 03 18:42:23.881403 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:23.881372    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7"]
Nov 03 18:42:23.928706 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:23.928679    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n4xz7"]
Nov 03 18:42:24.126229 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:24.126122    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:25.334728 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:25.334684    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41eba073-3e2a-4b3e-acba-d68aad192e43" path="/var/lib/kubelet/pods/41eba073-3e2a-4b3e-acba-d68aad192e43/volumes"
Nov 03 18:42:30.765442 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:30.765409    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:30.946699 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:30.946632    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:31.715292 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:31.715265    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-6964cbcfd5-f44z9"
Nov 03 18:42:31.861453 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:31.861417    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-pm2wr"]
Nov 03 18:42:34.224370 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.224323    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7b88dbb578-2tqps"]
Nov 03 18:42:34.224795 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.224769    6059 cpu_manager.go:414] "containerMap: removing container" podUID="41eba073-3e2a-4b3e-acba-d68aad192e43" containerName="payload-pre-processing"
Nov 03 18:42:34.224795 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.224781    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41eba073-3e2a-4b3e-acba-d68aad192e43" containerName="payload-pre-processing"
Nov 03 18:42:34.224894 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.224800    6059 cpu_manager.go:414] "containerMap: removing container" podUID="789edf94-9b0b-4fb8-869f-dc6791bf5a92" containerName="istio-proxy"
Nov 03 18:42:34.224894 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.224805    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789edf94-9b0b-4fb8-869f-dc6791bf5a92" containerName="istio-proxy"
Nov 03 18:42:34.224894 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.224866    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="41eba073-3e2a-4b3e-acba-d68aad192e43" containerName="payload-pre-processing"
Nov 03 18:42:34.228895 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.228875    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7b88dbb578-2tqps"
Nov 03 18:42:34.291591 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.291557    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7b88dbb578-2tqps"]
Nov 03 18:42:34.368941 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.368904    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h27l6\" (UniqueName: \"kubernetes.io/projected/42c95c41-1e40-4f67-8f76-6291a182f48c-kube-api-access-h27l6\") pod \"payload-processing-7b88dbb578-2tqps\" (UID: \"42c95c41-1e40-4f67-8f76-6291a182f48c\") " pod="openshift-ingress/payload-processing-7b88dbb578-2tqps"
Nov 03 18:42:34.369136 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.369091    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c95c41-1e40-4f67-8f76-6291a182f48c-plugins-config-volume\") pod \"payload-processing-7b88dbb578-2tqps\" (UID: \"42c95c41-1e40-4f67-8f76-6291a182f48c\") " pod="openshift-ingress/payload-processing-7b88dbb578-2tqps"
Nov 03 18:42:34.393199 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.393158    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7987dbcbcc-d2pl9"]
Nov 03 18:42:34.397132 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.397108    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7987dbcbcc-d2pl9"
Nov 03 18:42:34.458209 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.458176    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7987dbcbcc-d2pl9"]
Nov 03 18:42:34.470602 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.470572    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c95c41-1e40-4f67-8f76-6291a182f48c-plugins-config-volume\") pod \"payload-processing-7b88dbb578-2tqps\" (UID: \"42c95c41-1e40-4f67-8f76-6291a182f48c\") " pod="openshift-ingress/payload-processing-7b88dbb578-2tqps"
Nov 03 18:42:34.470764 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.470623    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h27l6\" (UniqueName: \"kubernetes.io/projected/42c95c41-1e40-4f67-8f76-6291a182f48c-kube-api-access-h27l6\") pod \"payload-processing-7b88dbb578-2tqps\" (UID: \"42c95c41-1e40-4f67-8f76-6291a182f48c\") " pod="openshift-ingress/payload-processing-7b88dbb578-2tqps"
Nov 03 18:42:34.470764 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.470647    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj548\" (UniqueName: \"kubernetes.io/projected/e7bdff75-bc2d-4591-993b-97eba56293ac-kube-api-access-vj548\") pod \"payload-pre-processing-7987dbcbcc-d2pl9\" (UID: \"e7bdff75-bc2d-4591-993b-97eba56293ac\") " pod="openshift-ingress/payload-pre-processing-7987dbcbcc-d2pl9"
Nov 03 18:42:34.470870 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.470803    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7bdff75-bc2d-4591-993b-97eba56293ac-plugins-config-volume\") pod \"payload-pre-processing-7987dbcbcc-d2pl9\" (UID: \"e7bdff75-bc2d-4591-993b-97eba56293ac\") " pod="openshift-ingress/payload-pre-processing-7987dbcbcc-d2pl9"
Nov 03 18:42:34.471581 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.471556    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c95c41-1e40-4f67-8f76-6291a182f48c-plugins-config-volume\") pod \"payload-processing-7b88dbb578-2tqps\" (UID: \"42c95c41-1e40-4f67-8f76-6291a182f48c\") " pod="openshift-ingress/payload-processing-7b88dbb578-2tqps"
Nov 03 18:42:34.529583 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.529518    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h27l6\" (UniqueName: \"kubernetes.io/projected/42c95c41-1e40-4f67-8f76-6291a182f48c-kube-api-access-h27l6\") pod \"payload-processing-7b88dbb578-2tqps\" (UID: \"42c95c41-1e40-4f67-8f76-6291a182f48c\") " pod="openshift-ingress/payload-processing-7b88dbb578-2tqps"
Nov 03 18:42:34.542412 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.542388    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7b88dbb578-2tqps"
Nov 03 18:42:34.571849 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.571820    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj548\" (UniqueName: \"kubernetes.io/projected/e7bdff75-bc2d-4591-993b-97eba56293ac-kube-api-access-vj548\") pod \"payload-pre-processing-7987dbcbcc-d2pl9\" (UID: \"e7bdff75-bc2d-4591-993b-97eba56293ac\") " pod="openshift-ingress/payload-pre-processing-7987dbcbcc-d2pl9"
Nov 03 18:42:34.571991 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.571913    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7bdff75-bc2d-4591-993b-97eba56293ac-plugins-config-volume\") pod \"payload-pre-processing-7987dbcbcc-d2pl9\" (UID: \"e7bdff75-bc2d-4591-993b-97eba56293ac\") " pod="openshift-ingress/payload-pre-processing-7987dbcbcc-d2pl9"
Nov 03 18:42:34.572677 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.572655    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7bdff75-bc2d-4591-993b-97eba56293ac-plugins-config-volume\") pod \"payload-pre-processing-7987dbcbcc-d2pl9\" (UID: \"e7bdff75-bc2d-4591-993b-97eba56293ac\") " pod="openshift-ingress/payload-pre-processing-7987dbcbcc-d2pl9"
Nov 03 18:42:34.630834 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.630807    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj548\" (UniqueName: \"kubernetes.io/projected/e7bdff75-bc2d-4591-993b-97eba56293ac-kube-api-access-vj548\") pod \"payload-pre-processing-7987dbcbcc-d2pl9\" (UID: \"e7bdff75-bc2d-4591-993b-97eba56293ac\") " pod="openshift-ingress/payload-pre-processing-7987dbcbcc-d2pl9"
Nov 03 18:42:34.710389 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.710339    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7987dbcbcc-d2pl9"
Nov 03 18:42:34.911716 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:34.911685    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7b88dbb578-2tqps"]
Nov 03 18:42:34.915287 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:42:34.915222    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c95c41_1e40_4f67_8f76_6291a182f48c.slice/crio-927d3d06160e9e39e74d6635f628e62336abfad5c5247fe684e2797728328b97 WatchSource:0}: Error finding container 927d3d06160e9e39e74d6635f628e62336abfad5c5247fe684e2797728328b97: Status 404 returned error can't find the container with id 927d3d06160e9e39e74d6635f628e62336abfad5c5247fe684e2797728328b97
Nov 03 18:42:35.083570 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:35.083547    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7987dbcbcc-d2pl9"]
Nov 03 18:42:35.085738 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:42:35.085706    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7bdff75_bc2d_4591_993b_97eba56293ac.slice/crio-70c9ae17ef1d53038600291a779d294bbe0c8d5db840a185eea77b4bd2186ac4 WatchSource:0}: Error finding container 70c9ae17ef1d53038600291a779d294bbe0c8d5db840a185eea77b4bd2186ac4: Status 404 returned error can't find the container with id 70c9ae17ef1d53038600291a779d294bbe0c8d5db840a185eea77b4bd2186ac4
Nov 03 18:42:35.302048 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:35.302016    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:35.792498 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:35.792458    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7b88dbb578-2tqps" event={"ID":"42c95c41-1e40-4f67-8f76-6291a182f48c","Type":"ContainerStarted","Data":"c8eab173089949556855e121eb83ee77e490fecf4bb3d0c751d4d4414788587f"}
Nov 03 18:42:35.792719 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:35.792517    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7b88dbb578-2tqps" event={"ID":"42c95c41-1e40-4f67-8f76-6291a182f48c","Type":"ContainerStarted","Data":"927d3d06160e9e39e74d6635f628e62336abfad5c5247fe684e2797728328b97"}
Nov 03 18:42:35.792719 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:35.792633    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7b88dbb578-2tqps"
Nov 03 18:42:35.794020 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:35.793992    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7987dbcbcc-d2pl9" event={"ID":"e7bdff75-bc2d-4591-993b-97eba56293ac","Type":"ContainerStarted","Data":"dfb112bf389b49ab1774d08bd7f02b90d3572b2ae11157b7ac504f4a2aad9236"}
Nov 03 18:42:35.794137 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:35.794043    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7987dbcbcc-d2pl9" event={"ID":"e7bdff75-bc2d-4591-993b-97eba56293ac","Type":"ContainerStarted","Data":"70c9ae17ef1d53038600291a779d294bbe0c8d5db840a185eea77b4bd2186ac4"}
Nov 03 18:42:35.794137 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:35.794075    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7987dbcbcc-d2pl9"
Nov 03 18:42:35.904524 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:35.904474    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7b88dbb578-2tqps" podStartSLOduration=1.904458338 podStartE2EDuration="1.904458338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:42:35.903066538 +0000 UTC m=+5096.700779319" watchObservedRunningTime="2026-11-03 18:42:35.904458338 +0000 UTC m=+5096.702171121"
Nov 03 18:42:36.015792 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:36.015740    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7987dbcbcc-d2pl9" podStartSLOduration=2.015727041 podStartE2EDuration="2.015727041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:42:36.015597304 +0000 UTC m=+5096.813310087" watchObservedRunningTime="2026-11-03 18:42:36.015727041 +0000 UTC m=+5096.813439822"
Nov 03 18:42:36.860000 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:42:36.859956    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:42:36.862170 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:36.862144    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-pm2wr" podUID="927401f9-153a-476e-8b41-7364a03d3cc8" containerName="payload-processing" containerID="cri-o://9e752319effd8d398968c4bff83a645f8b203027373e5f81b89f9eb6dcc2bd19" gracePeriod=25
Nov 03 18:42:37.117496 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:37.117435    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-pm2wr"
Nov 03 18:42:37.202504 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:37.202474    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/927401f9-153a-476e-8b41-7364a03d3cc8-kube-api-access-tqshg\" (UniqueName: \"kubernetes.io/projected/927401f9-153a-476e-8b41-7364a03d3cc8-kube-api-access-tqshg\") pod \"927401f9-153a-476e-8b41-7364a03d3cc8\" (UID: \"927401f9-153a-476e-8b41-7364a03d3cc8\") "
Nov 03 18:42:37.202666 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:37.202535    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/927401f9-153a-476e-8b41-7364a03d3cc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/927401f9-153a-476e-8b41-7364a03d3cc8-plugins-config-volume\") pod \"927401f9-153a-476e-8b41-7364a03d3cc8\" (UID: \"927401f9-153a-476e-8b41-7364a03d3cc8\") "
Nov 03 18:42:37.202930 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:37.202905    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/927401f9-153a-476e-8b41-7364a03d3cc8-plugins-config-volume" pod "927401f9-153a-476e-8b41-7364a03d3cc8" (UID: "927401f9-153a-476e-8b41-7364a03d3cc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:42:37.204467 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:37.204447    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/927401f9-153a-476e-8b41-7364a03d3cc8-kube-api-access-tqshg" pod "927401f9-153a-476e-8b41-7364a03d3cc8" (UID: "927401f9-153a-476e-8b41-7364a03d3cc8"). InnerVolumeSpecName "kube-api-access-tqshg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:42:37.303241 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:37.303210    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqshg\" (UniqueName: \"kubernetes.io/projected/927401f9-153a-476e-8b41-7364a03d3cc8-kube-api-access-tqshg\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:42:37.303241 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:37.303239    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/927401f9-153a-476e-8b41-7364a03d3cc8-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:42:37.813449 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:37.813410    6059 generic.go:356] "Generic (PLEG): container finished" podID="927401f9-153a-476e-8b41-7364a03d3cc8" containerID="9e752319effd8d398968c4bff83a645f8b203027373e5f81b89f9eb6dcc2bd19" exitCode=0
Nov 03 18:42:37.813654 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:37.813475    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-pm2wr"
Nov 03 18:42:37.813654 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:37.813478    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-pm2wr" event={"ID":"927401f9-153a-476e-8b41-7364a03d3cc8","Type":"ContainerDied","Data":"9e752319effd8d398968c4bff83a645f8b203027373e5f81b89f9eb6dcc2bd19"}
Nov 03 18:42:37.813654 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:37.813614    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-pm2wr" event={"ID":"927401f9-153a-476e-8b41-7364a03d3cc8","Type":"ContainerDied","Data":"dd66e90c2435dc75873147c3b7f74812722054be31e088a20ec59db039507149"}
Nov 03 18:42:37.813654 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:37.813637    6059 scope.go:122] "RemoveContainer" containerID="9e752319effd8d398968c4bff83a645f8b203027373e5f81b89f9eb6dcc2bd19"
Nov 03 18:42:37.822629 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:37.822613    6059 scope.go:122] "RemoveContainer" containerID="9e752319effd8d398968c4bff83a645f8b203027373e5f81b89f9eb6dcc2bd19"
Nov 03 18:42:37.822898 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:42:37.822878    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e752319effd8d398968c4bff83a645f8b203027373e5f81b89f9eb6dcc2bd19\": container with ID starting with 9e752319effd8d398968c4bff83a645f8b203027373e5f81b89f9eb6dcc2bd19 not found: ID does not exist" containerID="9e752319effd8d398968c4bff83a645f8b203027373e5f81b89f9eb6dcc2bd19"
Nov 03 18:42:37.822948 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:37.822905    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e752319effd8d398968c4bff83a645f8b203027373e5f81b89f9eb6dcc2bd19"} err="failed to get container status \"9e752319effd8d398968c4bff83a645f8b203027373e5f81b89f9eb6dcc2bd19\": rpc error: code = NotFound desc = could not find container \"9e752319effd8d398968c4bff83a645f8b203027373e5f81b89f9eb6dcc2bd19\": container with ID starting with 9e752319effd8d398968c4bff83a645f8b203027373e5f81b89f9eb6dcc2bd19 not found: ID does not exist"
Nov 03 18:42:37.972791 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:37.972757    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-pm2wr"]
Nov 03 18:42:38.020919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:38.020884    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-pm2wr"]
Nov 03 18:42:39.335125 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:39.335092    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="927401f9-153a-476e-8b41-7364a03d3cc8" path="/var/lib/kubelet/pods/927401f9-153a-476e-8b41-7364a03d3cc8/volumes"
Nov 03 18:42:40.107398 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:40.107371    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:42:40.132717 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:40.132687    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:42:40.138604 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:40.138577    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:42:40.153886 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:40.153865    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:42:43.048491 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:43.048454    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr"]
Nov 03 18:42:43.095980 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:43.095949    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1169f3cb-p7nqr"]
Nov 03 18:42:43.338716 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:43.338633    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60794c2d-01a6-4b9e-928c-911b810ea08d" path="/var/lib/kubelet/pods/60794c2d-01a6-4b9e-928c-911b810ea08d/volumes"
Nov 03 18:42:44.671349 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:44.671315    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:46.809283 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:46.809255    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7987dbcbcc-d2pl9"
Nov 03 18:42:46.809698 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:46.809323    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7b88dbb578-2tqps"
Nov 03 18:42:46.969108 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:46.969075    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr"]
Nov 03 18:42:47.060901 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:47.060814    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-6964cbcfd5-f44z9"]
Nov 03 18:42:49.823977 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:49.823943    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:50.228325 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:50.228291    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:51.969439 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:51.969382    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr" podUID="b4468af4-573a-4d81-8a9c-542768dbfe69" containerName="payload-pre-processing" containerID="cri-o://8efe8e534fc670effc2e97de42e47c61be26535df663c24c647e7da17f5926b4" gracePeriod=25
Nov 03 18:42:52.061649 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.061606    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-6964cbcfd5-f44z9" podUID="bfb0a010-326d-4b02-8fcd-d3067abb323a" containerName="payload-processing" containerID="cri-o://b87789fcd80769725396e8f06614e8622f3b6af31094e15209e40205d8f1a488" gracePeriod=25
Nov 03 18:42:52.323408 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.323374    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr"
Nov 03 18:42:52.353272 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.353244    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4468af4-573a-4d81-8a9c-542768dbfe69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4468af4-573a-4d81-8a9c-542768dbfe69-plugins-config-volume\") pod \"b4468af4-573a-4d81-8a9c-542768dbfe69\" (UID: \"b4468af4-573a-4d81-8a9c-542768dbfe69\") "
Nov 03 18:42:52.353447 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.353405    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4468af4-573a-4d81-8a9c-542768dbfe69-kube-api-access-f9xz2\" (UniqueName: \"kubernetes.io/projected/b4468af4-573a-4d81-8a9c-542768dbfe69-kube-api-access-f9xz2\") pod \"b4468af4-573a-4d81-8a9c-542768dbfe69\" (UID: \"b4468af4-573a-4d81-8a9c-542768dbfe69\") "
Nov 03 18:42:52.353639 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.353614    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4468af4-573a-4d81-8a9c-542768dbfe69-plugins-config-volume" pod "b4468af4-573a-4d81-8a9c-542768dbfe69" (UID: "b4468af4-573a-4d81-8a9c-542768dbfe69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:42:52.353878 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.353842    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4468af4-573a-4d81-8a9c-542768dbfe69-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:42:52.355469 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.355448    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4468af4-573a-4d81-8a9c-542768dbfe69-kube-api-access-f9xz2" pod "b4468af4-573a-4d81-8a9c-542768dbfe69" (UID: "b4468af4-573a-4d81-8a9c-542768dbfe69"). InnerVolumeSpecName "kube-api-access-f9xz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:42:52.356803 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.356785    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6964cbcfd5-f44z9"
Nov 03 18:42:52.454293 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.454263    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfb0a010-326d-4b02-8fcd-d3067abb323a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb0a010-326d-4b02-8fcd-d3067abb323a-plugins-config-volume\") pod \"bfb0a010-326d-4b02-8fcd-d3067abb323a\" (UID: \"bfb0a010-326d-4b02-8fcd-d3067abb323a\") "
Nov 03 18:42:52.454512 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.454341    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfb0a010-326d-4b02-8fcd-d3067abb323a-kube-api-access-lpgp9\" (UniqueName: \"kubernetes.io/projected/bfb0a010-326d-4b02-8fcd-d3067abb323a-kube-api-access-lpgp9\") pod \"bfb0a010-326d-4b02-8fcd-d3067abb323a\" (UID: \"bfb0a010-326d-4b02-8fcd-d3067abb323a\") "
Nov 03 18:42:52.454674 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.454649    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb0a010-326d-4b02-8fcd-d3067abb323a-plugins-config-volume" pod "bfb0a010-326d-4b02-8fcd-d3067abb323a" (UID: "bfb0a010-326d-4b02-8fcd-d3067abb323a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:42:52.454833 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.454818    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb0a010-326d-4b02-8fcd-d3067abb323a-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:42:52.454912 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.454836    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9xz2\" (UniqueName: \"kubernetes.io/projected/b4468af4-573a-4d81-8a9c-542768dbfe69-kube-api-access-f9xz2\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:42:52.456559 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.456535    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb0a010-326d-4b02-8fcd-d3067abb323a-kube-api-access-lpgp9" pod "bfb0a010-326d-4b02-8fcd-d3067abb323a" (UID: "bfb0a010-326d-4b02-8fcd-d3067abb323a"). InnerVolumeSpecName "kube-api-access-lpgp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:42:52.555658 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.555578    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpgp9\" (UniqueName: \"kubernetes.io/projected/bfb0a010-326d-4b02-8fcd-d3067abb323a-kube-api-access-lpgp9\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:42:52.715594 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.715541    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:42:52.897384 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.897291    6059 generic.go:356] "Generic (PLEG): container finished" podID="b4468af4-573a-4d81-8a9c-542768dbfe69" containerID="8efe8e534fc670effc2e97de42e47c61be26535df663c24c647e7da17f5926b4" exitCode=0
Nov 03 18:42:52.897384 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.897332    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr"
Nov 03 18:42:52.897601 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.897403    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr" event={"ID":"b4468af4-573a-4d81-8a9c-542768dbfe69","Type":"ContainerDied","Data":"8efe8e534fc670effc2e97de42e47c61be26535df663c24c647e7da17f5926b4"}
Nov 03 18:42:52.897601 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.897471    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr" event={"ID":"b4468af4-573a-4d81-8a9c-542768dbfe69","Type":"ContainerDied","Data":"36c57126445ac25d0a48bae3adec6a58edf4bdcc199d262b4d3474f4c1bdea68"}
Nov 03 18:42:52.897601 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.897508    6059 scope.go:122] "RemoveContainer" containerID="8efe8e534fc670effc2e97de42e47c61be26535df663c24c647e7da17f5926b4"
Nov 03 18:42:52.898787 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.898763    6059 generic.go:356] "Generic (PLEG): container finished" podID="bfb0a010-326d-4b02-8fcd-d3067abb323a" containerID="b87789fcd80769725396e8f06614e8622f3b6af31094e15209e40205d8f1a488" exitCode=0
Nov 03 18:42:52.898906 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.898813    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6964cbcfd5-f44z9"
Nov 03 18:42:52.898906 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.898853    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6964cbcfd5-f44z9" event={"ID":"bfb0a010-326d-4b02-8fcd-d3067abb323a","Type":"ContainerDied","Data":"b87789fcd80769725396e8f06614e8622f3b6af31094e15209e40205d8f1a488"}
Nov 03 18:42:52.898906 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.898898    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6964cbcfd5-f44z9" event={"ID":"bfb0a010-326d-4b02-8fcd-d3067abb323a","Type":"ContainerDied","Data":"bbd4658a23f228ce18a3df015a9751e32c1580e9df345dc35bb67be1f5348fd3"}
Nov 03 18:42:52.906745 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.906724    6059 scope.go:122] "RemoveContainer" containerID="8efe8e534fc670effc2e97de42e47c61be26535df663c24c647e7da17f5926b4"
Nov 03 18:42:52.907026 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:42:52.907006    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8efe8e534fc670effc2e97de42e47c61be26535df663c24c647e7da17f5926b4\": container with ID starting with 8efe8e534fc670effc2e97de42e47c61be26535df663c24c647e7da17f5926b4 not found: ID does not exist" containerID="8efe8e534fc670effc2e97de42e47c61be26535df663c24c647e7da17f5926b4"
Nov 03 18:42:52.907131 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.907033    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8efe8e534fc670effc2e97de42e47c61be26535df663c24c647e7da17f5926b4"} err="failed to get container status \"8efe8e534fc670effc2e97de42e47c61be26535df663c24c647e7da17f5926b4\": rpc error: code = NotFound desc = could not find container \"8efe8e534fc670effc2e97de42e47c61be26535df663c24c647e7da17f5926b4\": container with ID starting with 8efe8e534fc670effc2e97de42e47c61be26535df663c24c647e7da17f5926b4 not found: ID does not exist"
Nov 03 18:42:52.907131 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.907049    6059 scope.go:122] "RemoveContainer" containerID="b87789fcd80769725396e8f06614e8622f3b6af31094e15209e40205d8f1a488"
Nov 03 18:42:52.915934 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.915916    6059 scope.go:122] "RemoveContainer" containerID="b87789fcd80769725396e8f06614e8622f3b6af31094e15209e40205d8f1a488"
Nov 03 18:42:52.916181 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:42:52.916163    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b87789fcd80769725396e8f06614e8622f3b6af31094e15209e40205d8f1a488\": container with ID starting with b87789fcd80769725396e8f06614e8622f3b6af31094e15209e40205d8f1a488 not found: ID does not exist" containerID="b87789fcd80769725396e8f06614e8622f3b6af31094e15209e40205d8f1a488"
Nov 03 18:42:52.916228 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:52.916191    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b87789fcd80769725396e8f06614e8622f3b6af31094e15209e40205d8f1a488"} err="failed to get container status \"b87789fcd80769725396e8f06614e8622f3b6af31094e15209e40205d8f1a488\": rpc error: code = NotFound desc = could not find container \"b87789fcd80769725396e8f06614e8622f3b6af31094e15209e40205d8f1a488\": container with ID starting with b87789fcd80769725396e8f06614e8622f3b6af31094e15209e40205d8f1a488 not found: ID does not exist"
Nov 03 18:42:53.059902 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.059865    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-6964cbcfd5-f44z9"]
Nov 03 18:42:53.107120 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.107087    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-6964cbcfd5-f44z9"]
Nov 03 18:42:53.170230 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.170153    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr"]
Nov 03 18:42:53.171894 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.171870    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-c7cb977cb-gs6zr"]
Nov 03 18:42:53.182691 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.182669    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"]
Nov 03 18:42:53.183092 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.183079    6059 cpu_manager.go:414] "containerMap: removing container" podUID="b4468af4-573a-4d81-8a9c-542768dbfe69" containerName="payload-pre-processing"
Nov 03 18:42:53.183140 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.183094    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4468af4-573a-4d81-8a9c-542768dbfe69" containerName="payload-pre-processing"
Nov 03 18:42:53.183140 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.183109    6059 cpu_manager.go:414] "containerMap: removing container" podUID="927401f9-153a-476e-8b41-7364a03d3cc8" containerName="payload-processing"
Nov 03 18:42:53.183140 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.183114    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="927401f9-153a-476e-8b41-7364a03d3cc8" containerName="payload-processing"
Nov 03 18:42:53.183140 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.183126    6059 cpu_manager.go:414] "containerMap: removing container" podUID="bfb0a010-326d-4b02-8fcd-d3067abb323a" containerName="payload-processing"
Nov 03 18:42:53.183140 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.183132    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfb0a010-326d-4b02-8fcd-d3067abb323a" containerName="payload-processing"
Nov 03 18:42:53.183301 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.183207    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4468af4-573a-4d81-8a9c-542768dbfe69" containerName="payload-pre-processing"
Nov 03 18:42:53.183301 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.183218    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="927401f9-153a-476e-8b41-7364a03d3cc8" containerName="payload-processing"
Nov 03 18:42:53.183301 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.183225    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfb0a010-326d-4b02-8fcd-d3067abb323a" containerName="payload-processing"
Nov 03 18:42:53.187250 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.187230    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"
Nov 03 18:42:53.239601 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.239576    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 18:42:53.239724 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.239670    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\""
Nov 03 18:42:53.239801 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.239747    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 18:42:53.239847 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.239836    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 18:42:53.239895 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.239870    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 18:42:53.239940 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.239932    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pfj2m\""
Nov 03 18:42:53.246458 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.246440    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"]
Nov 03 18:42:53.263172 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.263143    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-7pspx\" (UID: \"cb966265-b8ab-44a7-91d5-50a37ec2fa9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"
Nov 03 18:42:53.263281 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.263199    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhz6q\" (UniqueName: \"kubernetes.io/projected/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-kube-api-access-xhz6q\") pod \"payload-processing-external-model-5bcd7997-7pspx\" (UID: \"cb966265-b8ab-44a7-91d5-50a37ec2fa9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"
Nov 03 18:42:53.263414 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.263397    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-7pspx\" (UID: \"cb966265-b8ab-44a7-91d5-50a37ec2fa9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"
Nov 03 18:42:53.263476 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.263424    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-7pspx\" (UID: \"cb966265-b8ab-44a7-91d5-50a37ec2fa9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"
Nov 03 18:42:53.336309 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.336277    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4468af4-573a-4d81-8a9c-542768dbfe69" path="/var/lib/kubelet/pods/b4468af4-573a-4d81-8a9c-542768dbfe69/volumes"
Nov 03 18:42:53.336683 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.336669    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfb0a010-326d-4b02-8fcd-d3067abb323a" path="/var/lib/kubelet/pods/bfb0a010-326d-4b02-8fcd-d3067abb323a/volumes"
Nov 03 18:42:53.364838 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.364814    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-7pspx\" (UID: \"cb966265-b8ab-44a7-91d5-50a37ec2fa9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"
Nov 03 18:42:53.364838 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.364842    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-7pspx\" (UID: \"cb966265-b8ab-44a7-91d5-50a37ec2fa9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"
Nov 03 18:42:53.365064 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.365041    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-7pspx\" (UID: \"cb966265-b8ab-44a7-91d5-50a37ec2fa9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"
Nov 03 18:42:53.365134 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.365107    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhz6q\" (UniqueName: \"kubernetes.io/projected/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-kube-api-access-xhz6q\") pod \"payload-processing-external-model-5bcd7997-7pspx\" (UID: \"cb966265-b8ab-44a7-91d5-50a37ec2fa9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"
Nov 03 18:42:53.365719 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.365692    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-7pspx\" (UID: \"cb966265-b8ab-44a7-91d5-50a37ec2fa9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"
Nov 03 18:42:53.365820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.365737    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-7pspx\" (UID: \"cb966265-b8ab-44a7-91d5-50a37ec2fa9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"
Nov 03 18:42:53.367319 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.367301    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-7pspx\" (UID: \"cb966265-b8ab-44a7-91d5-50a37ec2fa9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"
Nov 03 18:42:53.424150 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.424091    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhz6q\" (UniqueName: \"kubernetes.io/projected/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-kube-api-access-xhz6q\") pod \"payload-processing-external-model-5bcd7997-7pspx\" (UID: \"cb966265-b8ab-44a7-91d5-50a37ec2fa9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"
Nov 03 18:42:53.499963 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.499934    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"
Nov 03 18:42:53.919533 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:53.919502    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"]
Nov 03 18:42:53.921112 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:42:53.921087    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb966265_b8ab_44a7_91d5_50a37ec2fa9b.slice/crio-0e3db5c47e86d69016caac4feb2d5c69c24bbdad4781cde0597b698bba21f0c1 WatchSource:0}: Error finding container 0e3db5c47e86d69016caac4feb2d5c69c24bbdad4781cde0597b698bba21f0c1: Status 404 returned error can't find the container with id 0e3db5c47e86d69016caac4feb2d5c69c24bbdad4781cde0597b698bba21f0c1
Nov 03 18:42:54.916975 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:54.916931    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx" event={"ID":"cb966265-b8ab-44a7-91d5-50a37ec2fa9b","Type":"ContainerStarted","Data":"9433b636951ea90721e02bfac765386e864066f2fed07aee90124a6b0cc3273e"}
Nov 03 18:42:54.917517 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:54.917036    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx" event={"ID":"cb966265-b8ab-44a7-91d5-50a37ec2fa9b","Type":"ContainerStarted","Data":"0e3db5c47e86d69016caac4feb2d5c69c24bbdad4781cde0597b698bba21f0c1"}
Nov 03 18:42:54.917981 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:54.917958    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"
Nov 03 18:42:55.033411 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:55.033220    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx" podStartSLOduration=2.033202102 podStartE2EDuration="2.033202102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:42:55.029560908 +0000 UTC m=+5115.827273690" watchObservedRunningTime="2026-11-03 18:42:55.033202102 +0000 UTC m=+5115.830914884"
Nov 03 18:42:59.747680 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:42:59.747648    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:43:05.198245 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:43:05.198211    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:43:06.805578 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:43:06.805545    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:43:06.861489 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:43:06.861450    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:43:06.960967 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:43:06.960937    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"
Nov 03 18:43:20.941781 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:43:20.941696    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:43:36.861134 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:43:36.861099    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:43:43.955611 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:43:43.955579    6059 scope.go:122] "RemoveContainer" containerID="daa35c74704ad27a2b02fd9e9088bab373aac39770b953e33d78a18199244124"
Nov 03 18:43:43.964712 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:43:43.964691    6059 scope.go:122] "RemoveContainer" containerID="f40527202c3d031de0c49f5202a791baf25ca6c82e11019ab047f5616e450ba0"
Nov 03 18:43:43.972558 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:43:43.972538    6059 scope.go:122] "RemoveContainer" containerID="c1e3d13a271dfc1283fb708273c668e84f1a0c0d412caa63dccf4a173dba9fe1"
Nov 03 18:44:00.141505 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:00.141469    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:00.322534 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:00.322504    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:06.860709 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:44:06.860672    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:44:14.518901 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:14.518865    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bsl25"]
Nov 03 18:44:14.522722 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:14.522702    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bsl25"
Nov 03 18:44:14.572793 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:14.572755    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:44:14.620062 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:14.620024    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:44:14.620062 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:14.620037    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:44:14.620251 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:14.620044    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:44:14.621324 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:14.621303    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ddbd0f-a039-4be4-b5e9-b68b36a4d287-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-bsl25\" (UID: \"c7ddbd0f-a039-4be4-b5e9-b68b36a4d287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bsl25"
Nov 03 18:44:14.621447 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:14.621429    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ddbd0f-a039-4be4-b5e9-b68b36a4d287-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-bsl25\" (UID: \"c7ddbd0f-a039-4be4-b5e9-b68b36a4d287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bsl25"
Nov 03 18:44:14.621523 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:14.621496    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zw5w\" (UniqueName: \"kubernetes.io/projected/c7ddbd0f-a039-4be4-b5e9-b68b36a4d287-kube-api-access-4zw5w\") pod \"maas-api-revoke-sub-e2547622-bsl25\" (UID: \"c7ddbd0f-a039-4be4-b5e9-b68b36a4d287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bsl25"
Nov 03 18:44:14.622044 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:14.622025    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bsl25"]
Nov 03 18:44:14.722411 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:14.722374    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ddbd0f-a039-4be4-b5e9-b68b36a4d287-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-bsl25\" (UID: \"c7ddbd0f-a039-4be4-b5e9-b68b36a4d287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bsl25"
Nov 03 18:44:14.722584 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:14.722432    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zw5w\" (UniqueName: \"kubernetes.io/projected/c7ddbd0f-a039-4be4-b5e9-b68b36a4d287-kube-api-access-4zw5w\") pod \"maas-api-revoke-sub-e2547622-bsl25\" (UID: \"c7ddbd0f-a039-4be4-b5e9-b68b36a4d287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bsl25"
Nov 03 18:44:14.722584 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:14.722503    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ddbd0f-a039-4be4-b5e9-b68b36a4d287-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-bsl25\" (UID: \"c7ddbd0f-a039-4be4-b5e9-b68b36a4d287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bsl25"
Nov 03 18:44:14.723074 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:14.723052    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ddbd0f-a039-4be4-b5e9-b68b36a4d287-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-bsl25\" (UID: \"c7ddbd0f-a039-4be4-b5e9-b68b36a4d287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bsl25"
Nov 03 18:44:14.724779 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:14.724759    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ddbd0f-a039-4be4-b5e9-b68b36a4d287-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-bsl25\" (UID: \"c7ddbd0f-a039-4be4-b5e9-b68b36a4d287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bsl25"
Nov 03 18:44:14.781231 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:14.781164    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zw5w\" (UniqueName: \"kubernetes.io/projected/c7ddbd0f-a039-4be4-b5e9-b68b36a4d287-kube-api-access-4zw5w\") pod \"maas-api-revoke-sub-e2547622-bsl25\" (UID: \"c7ddbd0f-a039-4be4-b5e9-b68b36a4d287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bsl25"
Nov 03 18:44:14.835483 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:14.835456    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bsl25"
Nov 03 18:44:15.199919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:15.199893    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bsl25"]
Nov 03 18:44:15.202606 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:44:15.202559    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ddbd0f_a039_4be4_b5e9_b68b36a4d287.slice/crio-45c29f0bc5718f5fb5eac876b53107ee5acbade5f844ab2a6fe118a106a43474 WatchSource:0}: Error finding container 45c29f0bc5718f5fb5eac876b53107ee5acbade5f844ab2a6fe118a106a43474: Status 404 returned error can't find the container with id 45c29f0bc5718f5fb5eac876b53107ee5acbade5f844ab2a6fe118a106a43474
Nov 03 18:44:15.389583 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:15.389544    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bsl25" event={"ID":"c7ddbd0f-a039-4be4-b5e9-b68b36a4d287","Type":"ContainerStarted","Data":"45c29f0bc5718f5fb5eac876b53107ee5acbade5f844ab2a6fe118a106a43474"}
Nov 03 18:44:16.396314 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:16.396219    6059 generic.go:356] "Generic (PLEG): container finished" podID="c7ddbd0f-a039-4be4-b5e9-b68b36a4d287" containerID="0e9d2612e04d42ecc0054486534a56149b0f5155ff947bb178a18087cbbead21" exitCode=0
Nov 03 18:44:16.396770 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:16.396308    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bsl25" event={"ID":"c7ddbd0f-a039-4be4-b5e9-b68b36a4d287","Type":"ContainerDied","Data":"0e9d2612e04d42ecc0054486534a56149b0f5155ff947bb178a18087cbbead21"}
Nov 03 18:44:17.540799 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:17.540774    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bsl25"
Nov 03 18:44:17.651706 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:17.651596    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ddbd0f-a039-4be4-b5e9-b68b36a4d287-kube-api-access-4zw5w\" (UniqueName: \"kubernetes.io/projected/c7ddbd0f-a039-4be4-b5e9-b68b36a4d287-kube-api-access-4zw5w\") pod \"c7ddbd0f-a039-4be4-b5e9-b68b36a4d287\" (UID: \"c7ddbd0f-a039-4be4-b5e9-b68b36a4d287\") "
Nov 03 18:44:17.651706 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:17.651654    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7ddbd0f-a039-4be4-b5e9-b68b36a4d287-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ddbd0f-a039-4be4-b5e9-b68b36a4d287-maas-api-xks-ca\") pod \"c7ddbd0f-a039-4be4-b5e9-b68b36a4d287\" (UID: \"c7ddbd0f-a039-4be4-b5e9-b68b36a4d287\") "
Nov 03 18:44:17.651951 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:17.651771    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7ddbd0f-a039-4be4-b5e9-b68b36a4d287-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ddbd0f-a039-4be4-b5e9-b68b36a4d287-maas-api-openshift-ca\") pod \"c7ddbd0f-a039-4be4-b5e9-b68b36a4d287\" (UID: \"c7ddbd0f-a039-4be4-b5e9-b68b36a4d287\") "
Nov 03 18:44:17.652188 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:17.652165    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ddbd0f-a039-4be4-b5e9-b68b36a4d287-maas-api-openshift-ca" pod "c7ddbd0f-a039-4be4-b5e9-b68b36a4d287" (UID: "c7ddbd0f-a039-4be4-b5e9-b68b36a4d287"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:44:17.653750 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:17.653724    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7ddbd0f-a039-4be4-b5e9-b68b36a4d287-maas-api-xks-ca" pod "c7ddbd0f-a039-4be4-b5e9-b68b36a4d287" (UID: "c7ddbd0f-a039-4be4-b5e9-b68b36a4d287"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:44:17.654171 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:17.654148    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ddbd0f-a039-4be4-b5e9-b68b36a4d287-kube-api-access-4zw5w" pod "c7ddbd0f-a039-4be4-b5e9-b68b36a4d287" (UID: "c7ddbd0f-a039-4be4-b5e9-b68b36a4d287"). InnerVolumeSpecName "kube-api-access-4zw5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:44:17.753752 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:17.753718    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zw5w\" (UniqueName: \"kubernetes.io/projected/c7ddbd0f-a039-4be4-b5e9-b68b36a4d287-kube-api-access-4zw5w\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:44:17.753752 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:17.753746    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ddbd0f-a039-4be4-b5e9-b68b36a4d287-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:44:17.753961 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:17.753761    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ddbd0f-a039-4be4-b5e9-b68b36a4d287-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:44:18.408060 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:18.408032    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bsl25"
Nov 03 18:44:18.408258 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:18.408022    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bsl25" event={"ID":"c7ddbd0f-a039-4be4-b5e9-b68b36a4d287","Type":"ContainerDied","Data":"45c29f0bc5718f5fb5eac876b53107ee5acbade5f844ab2a6fe118a106a43474"}
Nov 03 18:44:18.408258 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:18.408162    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45c29f0bc5718f5fb5eac876b53107ee5acbade5f844ab2a6fe118a106a43474"
Nov 03 18:44:26.087800 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:26.087766    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"]
Nov 03 18:44:26.377944 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:26.377860    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:26.551817 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:26.551784    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:26.763213 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:26.763176    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:31.092471 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:31.092420    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx" podUID="cb966265-b8ab-44a7-91d5-50a37ec2fa9b" containerName="payload-processing" containerID="cri-o://9433b636951ea90721e02bfac765386e864066f2fed07aee90124a6b0cc3273e" gracePeriod=25
Nov 03 18:44:31.614208 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:31.614170    6059 generic.go:356] "Generic (PLEG): container finished" podID="cb966265-b8ab-44a7-91d5-50a37ec2fa9b" containerID="9433b636951ea90721e02bfac765386e864066f2fed07aee90124a6b0cc3273e" exitCode=0
Nov 03 18:44:31.614620 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:31.614464    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx" event={"ID":"cb966265-b8ab-44a7-91d5-50a37ec2fa9b","Type":"ContainerDied","Data":"9433b636951ea90721e02bfac765386e864066f2fed07aee90124a6b0cc3273e"}
Nov 03 18:44:31.810122 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:31.809782    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"
Nov 03 18:44:31.909567 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:31.909530    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-kube-api-access-xhz6q\" (UniqueName: \"kubernetes.io/projected/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-kube-api-access-xhz6q\") pod \"cb966265-b8ab-44a7-91d5-50a37ec2fa9b\" (UID: \"cb966265-b8ab-44a7-91d5-50a37ec2fa9b\") "
Nov 03 18:44:31.909755 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:31.909694    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-routing-overlay-volume\") pod \"cb966265-b8ab-44a7-91d5-50a37ec2fa9b\" (UID: \"cb966265-b8ab-44a7-91d5-50a37ec2fa9b\") "
Nov 03 18:44:31.909822 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:31.909805    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-provider-credentials\") pod \"cb966265-b8ab-44a7-91d5-50a37ec2fa9b\" (UID: \"cb966265-b8ab-44a7-91d5-50a37ec2fa9b\") "
Nov 03 18:44:31.909891 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:31.909846    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-plugins-config-volume\") pod \"cb966265-b8ab-44a7-91d5-50a37ec2fa9b\" (UID: \"cb966265-b8ab-44a7-91d5-50a37ec2fa9b\") "
Nov 03 18:44:31.910160 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:31.910068    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-routing-overlay-volume" pod "cb966265-b8ab-44a7-91d5-50a37ec2fa9b" (UID: "cb966265-b8ab-44a7-91d5-50a37ec2fa9b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:44:31.910390 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:31.910344    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-plugins-config-volume" pod "cb966265-b8ab-44a7-91d5-50a37ec2fa9b" (UID: "cb966265-b8ab-44a7-91d5-50a37ec2fa9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:44:31.911889 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:31.911782    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-provider-credentials" pod "cb966265-b8ab-44a7-91d5-50a37ec2fa9b" (UID: "cb966265-b8ab-44a7-91d5-50a37ec2fa9b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:44:31.911889 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:31.911859    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-kube-api-access-xhz6q" pod "cb966265-b8ab-44a7-91d5-50a37ec2fa9b" (UID: "cb966265-b8ab-44a7-91d5-50a37ec2fa9b"). InnerVolumeSpecName "kube-api-access-xhz6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:44:32.011006 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:32.010969    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhz6q\" (UniqueName: \"kubernetes.io/projected/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-kube-api-access-xhz6q\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:44:32.011006 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:32.011003    6059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-routing-overlay-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:44:32.011006 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:32.011013    6059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-provider-credentials\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:44:32.011268 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:32.011023    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb966265-b8ab-44a7-91d5-50a37ec2fa9b-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:44:32.621585 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:32.621546    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx" event={"ID":"cb966265-b8ab-44a7-91d5-50a37ec2fa9b","Type":"ContainerDied","Data":"0e3db5c47e86d69016caac4feb2d5c69c24bbdad4781cde0597b698bba21f0c1"}
Nov 03 18:44:32.621585 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:32.621587    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"
Nov 03 18:44:32.622126 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:32.621618    6059 scope.go:122] "RemoveContainer" containerID="9433b636951ea90721e02bfac765386e864066f2fed07aee90124a6b0cc3273e"
Nov 03 18:44:32.796377 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:32.795734    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"]
Nov 03 18:44:32.849380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:32.846015    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-7pspx"]
Nov 03 18:44:33.352770 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:33.352726    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb966265-b8ab-44a7-91d5-50a37ec2fa9b" path="/var/lib/kubelet/pods/cb966265-b8ab-44a7-91d5-50a37ec2fa9b/volumes"
Nov 03 18:44:36.861408 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:44:36.861344    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:44:49.828238 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:49.828198    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc3b6eae-dngd5"]
Nov 03 18:44:49.828708 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:49.828689    6059 cpu_manager.go:414] "containerMap: removing container" podUID="cb966265-b8ab-44a7-91d5-50a37ec2fa9b" containerName="payload-processing"
Nov 03 18:44:49.828708 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:49.828702    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb966265-b8ab-44a7-91d5-50a37ec2fa9b" containerName="payload-processing"
Nov 03 18:44:49.828790 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:49.828723    6059 cpu_manager.go:414] "containerMap: removing container" podUID="c7ddbd0f-a039-4be4-b5e9-b68b36a4d287" containerName="revoke-keys"
Nov 03 18:44:49.828790 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:49.828729    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ddbd0f-a039-4be4-b5e9-b68b36a4d287" containerName="revoke-keys"
Nov 03 18:44:49.828863 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:49.828815    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb966265-b8ab-44a7-91d5-50a37ec2fa9b" containerName="payload-processing"
Nov 03 18:44:49.828863 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:49.828824    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ddbd0f-a039-4be4-b5e9-b68b36a4d287" containerName="revoke-keys"
Nov 03 18:44:49.833525 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:49.833502    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3b6eae-dngd5"
Nov 03 18:44:49.883239 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:49.883202    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:44:49.930665 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:49.930627    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:44:49.930665 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:49.930627    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:44:49.930665 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:49.930661    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:44:49.932661 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:49.932642    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc3b6eae-dngd5"]
Nov 03 18:44:49.962882 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:49.962852    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4tlv\" (UniqueName: \"kubernetes.io/projected/9c3ccc35-75e7-4132-b9ee-c861e695778c-kube-api-access-h4tlv\") pod \"maas-api-revoke-sub-fc3b6eae-dngd5\" (UID: \"9c3ccc35-75e7-4132-b9ee-c861e695778c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3b6eae-dngd5"
Nov 03 18:44:49.963036 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:49.962906    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c3ccc35-75e7-4132-b9ee-c861e695778c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc3b6eae-dngd5\" (UID: \"9c3ccc35-75e7-4132-b9ee-c861e695778c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3b6eae-dngd5"
Nov 03 18:44:49.963088 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:49.963066    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c3ccc35-75e7-4132-b9ee-c861e695778c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc3b6eae-dngd5\" (UID: \"9c3ccc35-75e7-4132-b9ee-c861e695778c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3b6eae-dngd5"
Nov 03 18:44:50.064126 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:50.064093    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c3ccc35-75e7-4132-b9ee-c861e695778c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc3b6eae-dngd5\" (UID: \"9c3ccc35-75e7-4132-b9ee-c861e695778c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3b6eae-dngd5"
Nov 03 18:44:50.064306 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:50.064145    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4tlv\" (UniqueName: \"kubernetes.io/projected/9c3ccc35-75e7-4132-b9ee-c861e695778c-kube-api-access-h4tlv\") pod \"maas-api-revoke-sub-fc3b6eae-dngd5\" (UID: \"9c3ccc35-75e7-4132-b9ee-c861e695778c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3b6eae-dngd5"
Nov 03 18:44:50.064306 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:50.064183    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c3ccc35-75e7-4132-b9ee-c861e695778c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc3b6eae-dngd5\" (UID: \"9c3ccc35-75e7-4132-b9ee-c861e695778c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3b6eae-dngd5"
Nov 03 18:44:50.064816 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:50.064796    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c3ccc35-75e7-4132-b9ee-c861e695778c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc3b6eae-dngd5\" (UID: \"9c3ccc35-75e7-4132-b9ee-c861e695778c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3b6eae-dngd5"
Nov 03 18:44:50.066494 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:50.066470    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c3ccc35-75e7-4132-b9ee-c861e695778c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc3b6eae-dngd5\" (UID: \"9c3ccc35-75e7-4132-b9ee-c861e695778c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3b6eae-dngd5"
Nov 03 18:44:50.122997 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:50.122927    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4tlv\" (UniqueName: \"kubernetes.io/projected/9c3ccc35-75e7-4132-b9ee-c861e695778c-kube-api-access-h4tlv\") pod \"maas-api-revoke-sub-fc3b6eae-dngd5\" (UID: \"9c3ccc35-75e7-4132-b9ee-c861e695778c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3b6eae-dngd5"
Nov 03 18:44:50.148886 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:50.148861    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3b6eae-dngd5"
Nov 03 18:44:50.519164 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:50.519136    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc3b6eae-dngd5"]
Nov 03 18:44:50.521622 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:44:50.521588    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3ccc35_75e7_4132_b9ee_c861e695778c.slice/crio-e6aa9eaa9a1db9ce2adc97c6d5694d385c68760ece22fda9f70eedcb37c2f4f4 WatchSource:0}: Error finding container e6aa9eaa9a1db9ce2adc97c6d5694d385c68760ece22fda9f70eedcb37c2f4f4: Status 404 returned error can't find the container with id e6aa9eaa9a1db9ce2adc97c6d5694d385c68760ece22fda9f70eedcb37c2f4f4
Nov 03 18:44:50.814714 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:50.814628    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3b6eae-dngd5" event={"ID":"9c3ccc35-75e7-4132-b9ee-c861e695778c","Type":"ContainerStarted","Data":"e6aa9eaa9a1db9ce2adc97c6d5694d385c68760ece22fda9f70eedcb37c2f4f4"}
Nov 03 18:44:51.820754 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:51.820717    6059 generic.go:356] "Generic (PLEG): container finished" podID="9c3ccc35-75e7-4132-b9ee-c861e695778c" containerID="095698a424fcc4205b888ede777529dedf6b9db05fedbb37eda4e0b28ca45683" exitCode=0
Nov 03 18:44:51.821176 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:51.820796    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3b6eae-dngd5" event={"ID":"9c3ccc35-75e7-4132-b9ee-c861e695778c","Type":"ContainerDied","Data":"095698a424fcc4205b888ede777529dedf6b9db05fedbb37eda4e0b28ca45683"}
Nov 03 18:44:52.962997 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:52.962971    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3b6eae-dngd5"
Nov 03 18:44:53.095474 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:53.095400    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c3ccc35-75e7-4132-b9ee-c861e695778c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c3ccc35-75e7-4132-b9ee-c861e695778c-maas-api-xks-ca\") pod \"9c3ccc35-75e7-4132-b9ee-c861e695778c\" (UID: \"9c3ccc35-75e7-4132-b9ee-c861e695778c\") "
Nov 03 18:44:53.095624 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:53.095595    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c3ccc35-75e7-4132-b9ee-c861e695778c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c3ccc35-75e7-4132-b9ee-c861e695778c-maas-api-openshift-ca\") pod \"9c3ccc35-75e7-4132-b9ee-c861e695778c\" (UID: \"9c3ccc35-75e7-4132-b9ee-c861e695778c\") "
Nov 03 18:44:53.095667 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:53.095649    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c3ccc35-75e7-4132-b9ee-c861e695778c-kube-api-access-h4tlv\" (UniqueName: \"kubernetes.io/projected/9c3ccc35-75e7-4132-b9ee-c861e695778c-kube-api-access-h4tlv\") pod \"9c3ccc35-75e7-4132-b9ee-c861e695778c\" (UID: \"9c3ccc35-75e7-4132-b9ee-c861e695778c\") "
Nov 03 18:44:53.095951 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:53.095921    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c3ccc35-75e7-4132-b9ee-c861e695778c-maas-api-openshift-ca" pod "9c3ccc35-75e7-4132-b9ee-c861e695778c" (UID: "9c3ccc35-75e7-4132-b9ee-c861e695778c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:44:53.097597 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:53.097570    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c3ccc35-75e7-4132-b9ee-c861e695778c-maas-api-xks-ca" pod "9c3ccc35-75e7-4132-b9ee-c861e695778c" (UID: "9c3ccc35-75e7-4132-b9ee-c861e695778c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:44:53.097834 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:53.097808    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3ccc35-75e7-4132-b9ee-c861e695778c-kube-api-access-h4tlv" pod "9c3ccc35-75e7-4132-b9ee-c861e695778c" (UID: "9c3ccc35-75e7-4132-b9ee-c861e695778c"). InnerVolumeSpecName "kube-api-access-h4tlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:44:53.196949 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:53.196916    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4tlv\" (UniqueName: \"kubernetes.io/projected/9c3ccc35-75e7-4132-b9ee-c861e695778c-kube-api-access-h4tlv\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:44:53.196949 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:53.196945    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c3ccc35-75e7-4132-b9ee-c861e695778c-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:44:53.196949 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:53.196954    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c3ccc35-75e7-4132-b9ee-c861e695778c-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:44:53.831474 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:53.831444    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3b6eae-dngd5"
Nov 03 18:44:53.831648 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:53.831489    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3b6eae-dngd5" event={"ID":"9c3ccc35-75e7-4132-b9ee-c861e695778c","Type":"ContainerDied","Data":"e6aa9eaa9a1db9ce2adc97c6d5694d385c68760ece22fda9f70eedcb37c2f4f4"}
Nov 03 18:44:53.831648 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:44:53.831562    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6aa9eaa9a1db9ce2adc97c6d5694d385c68760ece22fda9f70eedcb37c2f4f4"
Nov 03 18:45:00.154379 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:00.154328    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254"]
Nov 03 18:45:00.154782 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:00.154769    6059 cpu_manager.go:414] "containerMap: removing container" podUID="9c3ccc35-75e7-4132-b9ee-c861e695778c" containerName="revoke-keys"
Nov 03 18:45:00.154850 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:00.154784    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c3ccc35-75e7-4132-b9ee-c861e695778c" containerName="revoke-keys"
Nov 03 18:45:00.154888 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:00.154873    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c3ccc35-75e7-4132-b9ee-c861e695778c" containerName="revoke-keys"
Nov 03 18:45:00.158280 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:00.158255    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254"
Nov 03 18:45:00.164641 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:00.164615    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6a716b8-443a-4d2e-884f-d311b2b73ba6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895525-wk254\" (UID: \"c6a716b8-443a-4d2e-884f-d311b2b73ba6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254"
Nov 03 18:45:00.164786 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:00.164761    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhpzp\" (UniqueName: \"kubernetes.io/projected/c6a716b8-443a-4d2e-884f-d311b2b73ba6-kube-api-access-qhpzp\") pod \"maas-api-key-cleanup-29895525-wk254\" (UID: \"c6a716b8-443a-4d2e-884f-d311b2b73ba6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254"
Nov 03 18:45:00.208399 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:00.208370    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:45:00.255644 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:00.255610    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:45:00.255813 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:00.255646    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:45:00.264580 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:00.259928    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254"]
Nov 03 18:45:00.266103 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:00.266075    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhpzp\" (UniqueName: \"kubernetes.io/projected/c6a716b8-443a-4d2e-884f-d311b2b73ba6-kube-api-access-qhpzp\") pod \"maas-api-key-cleanup-29895525-wk254\" (UID: \"c6a716b8-443a-4d2e-884f-d311b2b73ba6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254"
Nov 03 18:45:00.266228 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:00.266148    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6a716b8-443a-4d2e-884f-d311b2b73ba6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895525-wk254\" (UID: \"c6a716b8-443a-4d2e-884f-d311b2b73ba6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254"
Nov 03 18:45:00.267156 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:00.267133    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6a716b8-443a-4d2e-884f-d311b2b73ba6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895525-wk254\" (UID: \"c6a716b8-443a-4d2e-884f-d311b2b73ba6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254"
Nov 03 18:45:00.324941 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:00.324914    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhpzp\" (UniqueName: \"kubernetes.io/projected/c6a716b8-443a-4d2e-884f-d311b2b73ba6-kube-api-access-qhpzp\") pod \"maas-api-key-cleanup-29895525-wk254\" (UID: \"c6a716b8-443a-4d2e-884f-d311b2b73ba6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254"
Nov 03 18:45:00.436881 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:00.436788    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:00.472082 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:00.472050    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254"
Nov 03 18:45:00.839815 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:00.839731    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254"]
Nov 03 18:45:00.842432 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:45:00.842400    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a716b8_443a_4d2e_884f_d311b2b73ba6.slice/crio-e853c2f0944da7bd595d58605718aeaccf6c627e3df404d2bbd19193f2086ece WatchSource:0}: Error finding container e853c2f0944da7bd595d58605718aeaccf6c627e3df404d2bbd19193f2086ece: Status 404 returned error can't find the container with id e853c2f0944da7bd595d58605718aeaccf6c627e3df404d2bbd19193f2086ece
Nov 03 18:45:00.868325 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:00.868297    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254" event={"ID":"c6a716b8-443a-4d2e-884f-d311b2b73ba6","Type":"ContainerStarted","Data":"e853c2f0944da7bd595d58605718aeaccf6c627e3df404d2bbd19193f2086ece"}
Nov 03 18:45:01.874894 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:01.874858    6059 generic.go:356] "Generic (PLEG): container finished" podID="c6a716b8-443a-4d2e-884f-d311b2b73ba6" containerID="53a293b61aa75717f11b3ad40d97753b479577f43b16bde6ba59301c6658d17f" exitCode=60
Nov 03 18:45:01.875401 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:01.874939    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254" event={"ID":"c6a716b8-443a-4d2e-884f-d311b2b73ba6","Type":"ContainerDied","Data":"53a293b61aa75717f11b3ad40d97753b479577f43b16bde6ba59301c6658d17f"}
Nov 03 18:45:01.875401 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:01.875207    6059 scope.go:122] "RemoveContainer" containerID="53a293b61aa75717f11b3ad40d97753b479577f43b16bde6ba59301c6658d17f"
Nov 03 18:45:02.881623 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:02.881589    6059 generic.go:356] "Generic (PLEG): container finished" podID="c6a716b8-443a-4d2e-884f-d311b2b73ba6" containerID="12caec3d25c849b53da692fd94b4a034183c40b072ae9a134c688b35a2caad5e" exitCode=60
Nov 03 18:45:02.882056 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:02.881645    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254" event={"ID":"c6a716b8-443a-4d2e-884f-d311b2b73ba6","Type":"ContainerDied","Data":"12caec3d25c849b53da692fd94b4a034183c40b072ae9a134c688b35a2caad5e"}
Nov 03 18:45:02.882056 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:02.881693    6059 scope.go:122] "RemoveContainer" containerID="53a293b61aa75717f11b3ad40d97753b479577f43b16bde6ba59301c6658d17f"
Nov 03 18:45:02.882056 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:02.881934    6059 scope.go:122] "RemoveContainer" containerID="12caec3d25c849b53da692fd94b4a034183c40b072ae9a134c688b35a2caad5e"
Nov 03 18:45:02.882225 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:45:02.882201    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895525-wk254_odh-ai-gateway-infra(c6a716b8-443a-4d2e-884f-d311b2b73ba6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254" podUID="c6a716b8-443a-4d2e-884f-d311b2b73ba6"
Nov 03 18:45:03.889209 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:03.889181    6059 scope.go:122] "RemoveContainer" containerID="12caec3d25c849b53da692fd94b4a034183c40b072ae9a134c688b35a2caad5e"
Nov 03 18:45:03.889634 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:45:03.889474    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895525-wk254_odh-ai-gateway-infra(c6a716b8-443a-4d2e-884f-d311b2b73ba6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254" podUID="c6a716b8-443a-4d2e-884f-d311b2b73ba6"
Nov 03 18:45:06.860746 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:45:06.860708    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:45:10.227108 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:10.227040    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:14.329419 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:14.329383    6059 scope.go:122] "RemoveContainer" containerID="12caec3d25c849b53da692fd94b4a034183c40b072ae9a134c688b35a2caad5e"
Nov 03 18:45:14.994617 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:14.994575    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254" event={"ID":"c6a716b8-443a-4d2e-884f-d311b2b73ba6","Type":"ContainerStarted","Data":"78a3dbbbfeae9200671d5e5bea6688cce6626f31b440b1794a49f4d66fb2daa9"}
Nov 03 18:45:15.105597 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:15.105544    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254" podStartSLOduration=15.105531196 podStartE2EDuration="15.105531196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:15.105032068 +0000 UTC m=+5255.902744849" watchObservedRunningTime="2026-11-03 18:45:15.105531196 +0000 UTC m=+5255.903243977"
Nov 03 18:45:16.001302 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:16.001271    6059 generic.go:356] "Generic (PLEG): container finished" podID="c6a716b8-443a-4d2e-884f-d311b2b73ba6" containerID="78a3dbbbfeae9200671d5e5bea6688cce6626f31b440b1794a49f4d66fb2daa9" exitCode=60
Nov 03 18:45:16.001792 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:16.001349    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254" event={"ID":"c6a716b8-443a-4d2e-884f-d311b2b73ba6","Type":"ContainerDied","Data":"78a3dbbbfeae9200671d5e5bea6688cce6626f31b440b1794a49f4d66fb2daa9"}
Nov 03 18:45:16.001792 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:16.001442    6059 scope.go:122] "RemoveContainer" containerID="12caec3d25c849b53da692fd94b4a034183c40b072ae9a134c688b35a2caad5e"
Nov 03 18:45:16.001920 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:16.001802    6059 scope.go:122] "RemoveContainer" containerID="78a3dbbbfeae9200671d5e5bea6688cce6626f31b440b1794a49f4d66fb2daa9"
Nov 03 18:45:16.002098 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:45:16.002073    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895525-wk254_odh-ai-gateway-infra(c6a716b8-443a-4d2e-884f-d311b2b73ba6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254" podUID="c6a716b8-443a-4d2e-884f-d311b2b73ba6"
Nov 03 18:45:16.121773 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:16.121738    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254"]
Nov 03 18:45:17.152795 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:17.152765    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254"
Nov 03 18:45:17.200201 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:17.200162    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6a716b8-443a-4d2e-884f-d311b2b73ba6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6a716b8-443a-4d2e-884f-d311b2b73ba6-maas-api-service-ca\") pod \"c6a716b8-443a-4d2e-884f-d311b2b73ba6\" (UID: \"c6a716b8-443a-4d2e-884f-d311b2b73ba6\") "
Nov 03 18:45:17.200427 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:17.200242    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6a716b8-443a-4d2e-884f-d311b2b73ba6-kube-api-access-qhpzp\" (UniqueName: \"kubernetes.io/projected/c6a716b8-443a-4d2e-884f-d311b2b73ba6-kube-api-access-qhpzp\") pod \"c6a716b8-443a-4d2e-884f-d311b2b73ba6\" (UID: \"c6a716b8-443a-4d2e-884f-d311b2b73ba6\") "
Nov 03 18:45:17.200565 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:17.200540    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a716b8-443a-4d2e-884f-d311b2b73ba6-maas-api-service-ca" pod "c6a716b8-443a-4d2e-884f-d311b2b73ba6" (UID: "c6a716b8-443a-4d2e-884f-d311b2b73ba6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:45:17.200659 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:17.200647    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6a716b8-443a-4d2e-884f-d311b2b73ba6-maas-api-service-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:17.202257 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:17.202239    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a716b8-443a-4d2e-884f-d311b2b73ba6-kube-api-access-qhpzp" pod "c6a716b8-443a-4d2e-884f-d311b2b73ba6" (UID: "c6a716b8-443a-4d2e-884f-d311b2b73ba6"). InnerVolumeSpecName "kube-api-access-qhpzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:45:17.301622 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:17.301528    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhpzp\" (UniqueName: \"kubernetes.io/projected/c6a716b8-443a-4d2e-884f-d311b2b73ba6-kube-api-access-qhpzp\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:18.016520 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:18.016486    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254"
Nov 03 18:45:18.016520 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:18.016495    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254" event={"ID":"c6a716b8-443a-4d2e-884f-d311b2b73ba6","Type":"ContainerDied","Data":"e853c2f0944da7bd595d58605718aeaccf6c627e3df404d2bbd19193f2086ece"}
Nov 03 18:45:18.016795 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:18.016592    6059 scope.go:122] "RemoveContainer" containerID="78a3dbbbfeae9200671d5e5bea6688cce6626f31b440b1794a49f4d66fb2daa9"
Nov 03 18:45:18.173887 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:18.173848    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254"]
Nov 03 18:45:18.219622 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:18.219589    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895525-wk254"]
Nov 03 18:45:19.334808 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:19.334775    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6a716b8-443a-4d2e-884f-d311b2b73ba6" path="/var/lib/kubelet/pods/c6a716b8-443a-4d2e-884f-d311b2b73ba6/volumes"
Nov 03 18:45:26.948788 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:26.948749    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g"]
Nov 03 18:45:26.949255 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:26.949225    6059 cpu_manager.go:414] "containerMap: removing container" podUID="c6a716b8-443a-4d2e-884f-d311b2b73ba6" containerName="cleanup"
Nov 03 18:45:26.949255 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:26.949238    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6a716b8-443a-4d2e-884f-d311b2b73ba6" containerName="cleanup"
Nov 03 18:45:26.949350 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:26.949262    6059 cpu_manager.go:414] "containerMap: removing container" podUID="c6a716b8-443a-4d2e-884f-d311b2b73ba6" containerName="cleanup"
Nov 03 18:45:26.949350 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:26.949268    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6a716b8-443a-4d2e-884f-d311b2b73ba6" containerName="cleanup"
Nov 03 18:45:26.949350 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:26.949347    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6a716b8-443a-4d2e-884f-d311b2b73ba6" containerName="cleanup"
Nov 03 18:45:26.949488 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:26.949376    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6a716b8-443a-4d2e-884f-d311b2b73ba6" containerName="cleanup"
Nov 03 18:45:26.952624 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:26.952599    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g"
Nov 03 18:45:27.002435 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:27.002398    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:45:27.049479 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:27.049448    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:45:27.049479 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:27.049463    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:45:27.049700 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:27.049514    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:45:27.051625 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:27.051605    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g"]
Nov 03 18:45:27.104800 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:27.104769    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d17028ef-cc8f-4851-8bb3-9f223966c36c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-phs9g\" (UID: \"d17028ef-cc8f-4851-8bb3-9f223966c36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g"
Nov 03 18:45:27.104976 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:27.104815    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc6l8\" (UniqueName: \"kubernetes.io/projected/d17028ef-cc8f-4851-8bb3-9f223966c36c-kube-api-access-cc6l8\") pod \"maas-api-revoke-sub-ec843018-phs9g\" (UID: \"d17028ef-cc8f-4851-8bb3-9f223966c36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g"
Nov 03 18:45:27.104976 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:27.104953    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d17028ef-cc8f-4851-8bb3-9f223966c36c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-phs9g\" (UID: \"d17028ef-cc8f-4851-8bb3-9f223966c36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g"
Nov 03 18:45:27.206139 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:27.206107    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d17028ef-cc8f-4851-8bb3-9f223966c36c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-phs9g\" (UID: \"d17028ef-cc8f-4851-8bb3-9f223966c36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g"
Nov 03 18:45:27.206292 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:27.206182    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d17028ef-cc8f-4851-8bb3-9f223966c36c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-phs9g\" (UID: \"d17028ef-cc8f-4851-8bb3-9f223966c36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g"
Nov 03 18:45:27.206292 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:27.206214    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc6l8\" (UniqueName: \"kubernetes.io/projected/d17028ef-cc8f-4851-8bb3-9f223966c36c-kube-api-access-cc6l8\") pod \"maas-api-revoke-sub-ec843018-phs9g\" (UID: \"d17028ef-cc8f-4851-8bb3-9f223966c36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g"
Nov 03 18:45:27.206929 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:27.206901    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d17028ef-cc8f-4851-8bb3-9f223966c36c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-phs9g\" (UID: \"d17028ef-cc8f-4851-8bb3-9f223966c36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g"
Nov 03 18:45:27.208541 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:27.208521    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d17028ef-cc8f-4851-8bb3-9f223966c36c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-phs9g\" (UID: \"d17028ef-cc8f-4851-8bb3-9f223966c36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g"
Nov 03 18:45:27.264605 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:27.264578    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc6l8\" (UniqueName: \"kubernetes.io/projected/d17028ef-cc8f-4851-8bb3-9f223966c36c-kube-api-access-cc6l8\") pod \"maas-api-revoke-sub-ec843018-phs9g\" (UID: \"d17028ef-cc8f-4851-8bb3-9f223966c36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g"
Nov 03 18:45:27.265334 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:27.265308    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g"
Nov 03 18:45:27.637790 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:27.637760    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g"]
Nov 03 18:45:27.643451 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:45:27.643349    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17028ef_cc8f_4851_8bb3_9f223966c36c.slice/crio-ceae7e769458ca1e825aaafcdd92f0ce9356435a6a53b813d5e6d88e5f367293 WatchSource:0}: Error finding container ceae7e769458ca1e825aaafcdd92f0ce9356435a6a53b813d5e6d88e5f367293: Status 404 returned error can't find the container with id ceae7e769458ca1e825aaafcdd92f0ce9356435a6a53b813d5e6d88e5f367293
Nov 03 18:45:28.074350 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:28.074309    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g" event={"ID":"d17028ef-cc8f-4851-8bb3-9f223966c36c","Type":"ContainerStarted","Data":"178f9e12cc1cff5a7426c2fbc3d1bc6d49463752c06d28a7d7e37571710592e0"}
Nov 03 18:45:28.074786 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:28.074386    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g" event={"ID":"d17028ef-cc8f-4851-8bb3-9f223966c36c","Type":"ContainerStarted","Data":"ceae7e769458ca1e825aaafcdd92f0ce9356435a6a53b813d5e6d88e5f367293"}
Nov 03 18:45:28.186559 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:28.186505    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g" podStartSLOduration=2.186489504 podStartE2EDuration="2.186489504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:28.184983744 +0000 UTC m=+5268.982696527" watchObservedRunningTime="2026-11-03 18:45:28.186489504 +0000 UTC m=+5268.984202284"
Nov 03 18:45:29.081490 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:29.081448    6059 generic.go:356] "Generic (PLEG): container finished" podID="d17028ef-cc8f-4851-8bb3-9f223966c36c" containerID="178f9e12cc1cff5a7426c2fbc3d1bc6d49463752c06d28a7d7e37571710592e0" exitCode=0
Nov 03 18:45:29.081880 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:29.081534    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g" event={"ID":"d17028ef-cc8f-4851-8bb3-9f223966c36c","Type":"ContainerDied","Data":"178f9e12cc1cff5a7426c2fbc3d1bc6d49463752c06d28a7d7e37571710592e0"}
Nov 03 18:45:30.224581 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:30.224553    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g"
Nov 03 18:45:30.339948 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:30.339871    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d17028ef-cc8f-4851-8bb3-9f223966c36c-kube-api-access-cc6l8\" (UniqueName: \"kubernetes.io/projected/d17028ef-cc8f-4851-8bb3-9f223966c36c-kube-api-access-cc6l8\") pod \"d17028ef-cc8f-4851-8bb3-9f223966c36c\" (UID: \"d17028ef-cc8f-4851-8bb3-9f223966c36c\") "
Nov 03 18:45:30.340106 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:30.339986    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d17028ef-cc8f-4851-8bb3-9f223966c36c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d17028ef-cc8f-4851-8bb3-9f223966c36c-maas-api-openshift-ca\") pod \"d17028ef-cc8f-4851-8bb3-9f223966c36c\" (UID: \"d17028ef-cc8f-4851-8bb3-9f223966c36c\") "
Nov 03 18:45:30.340106 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:30.340079    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d17028ef-cc8f-4851-8bb3-9f223966c36c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d17028ef-cc8f-4851-8bb3-9f223966c36c-maas-api-xks-ca\") pod \"d17028ef-cc8f-4851-8bb3-9f223966c36c\" (UID: \"d17028ef-cc8f-4851-8bb3-9f223966c36c\") "
Nov 03 18:45:30.340599 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:30.340570    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d17028ef-cc8f-4851-8bb3-9f223966c36c-maas-api-openshift-ca" pod "d17028ef-cc8f-4851-8bb3-9f223966c36c" (UID: "d17028ef-cc8f-4851-8bb3-9f223966c36c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:45:30.343391 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:30.342448    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17028ef-cc8f-4851-8bb3-9f223966c36c-kube-api-access-cc6l8" pod "d17028ef-cc8f-4851-8bb3-9f223966c36c" (UID: "d17028ef-cc8f-4851-8bb3-9f223966c36c"). InnerVolumeSpecName "kube-api-access-cc6l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:45:30.343391 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:30.342535    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d17028ef-cc8f-4851-8bb3-9f223966c36c-maas-api-xks-ca" pod "d17028ef-cc8f-4851-8bb3-9f223966c36c" (UID: "d17028ef-cc8f-4851-8bb3-9f223966c36c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:45:30.441549 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:30.441508    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc6l8\" (UniqueName: \"kubernetes.io/projected/d17028ef-cc8f-4851-8bb3-9f223966c36c-kube-api-access-cc6l8\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:30.441549 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:30.441546    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d17028ef-cc8f-4851-8bb3-9f223966c36c-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:30.441549 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:30.441557    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d17028ef-cc8f-4851-8bb3-9f223966c36c-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:31.092636 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:31.092608    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g"
Nov 03 18:45:31.092826 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:31.092611    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g" event={"ID":"d17028ef-cc8f-4851-8bb3-9f223966c36c","Type":"ContainerDied","Data":"ceae7e769458ca1e825aaafcdd92f0ce9356435a6a53b813d5e6d88e5f367293"}
Nov 03 18:45:31.092826 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:31.092731    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ceae7e769458ca1e825aaafcdd92f0ce9356435a6a53b813d5e6d88e5f367293"
Nov 03 18:45:36.861217 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:45:36.861176    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:45:47.055380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:47.054449    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zp9n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:45:47.055380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:47.054517    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7" podUID="60b66cb6-2bce-4c02-aa7d-7eb25f29f237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:45:47.073479 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:47.073439    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xpshn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:45:47.073756 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:45:47.073723    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn" podUID="d9e3abf7-c5e1-4414-858e-5f84f6a88670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:46:00.794563 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:00.794521    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:46:03.498065 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.498029    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7msdc"]
Nov 03 18:46:03.498505 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.498480    6059 cpu_manager.go:414] "containerMap: removing container" podUID="c6a716b8-443a-4d2e-884f-d311b2b73ba6" containerName="cleanup"
Nov 03 18:46:03.498505 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.498493    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6a716b8-443a-4d2e-884f-d311b2b73ba6" containerName="cleanup"
Nov 03 18:46:03.498585 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.498510    6059 cpu_manager.go:414] "containerMap: removing container" podUID="d17028ef-cc8f-4851-8bb3-9f223966c36c" containerName="revoke-keys"
Nov 03 18:46:03.498585 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.498516    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d17028ef-cc8f-4851-8bb3-9f223966c36c" containerName="revoke-keys"
Nov 03 18:46:03.498585 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.498581    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d17028ef-cc8f-4851-8bb3-9f223966c36c" containerName="revoke-keys"
Nov 03 18:46:03.498692 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.498598    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6a716b8-443a-4d2e-884f-d311b2b73ba6" containerName="cleanup"
Nov 03 18:46:03.502057 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.502030    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7msdc"
Nov 03 18:46:03.556842 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.556802    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:46:03.593008 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.592968    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68896ba-8099-4ae5-a76d-262bd32fc504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7msdc\" (UID: \"f68896ba-8099-4ae5-a76d-262bd32fc504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7msdc"
Nov 03 18:46:03.593194 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.593024    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68896ba-8099-4ae5-a76d-262bd32fc504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7msdc\" (UID: \"f68896ba-8099-4ae5-a76d-262bd32fc504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7msdc"
Nov 03 18:46:03.593245 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.593218    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxzkr\" (UniqueName: \"kubernetes.io/projected/f68896ba-8099-4ae5-a76d-262bd32fc504-kube-api-access-gxzkr\") pod \"maas-api-revoke-sub-046e1343-7msdc\" (UID: \"f68896ba-8099-4ae5-a76d-262bd32fc504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7msdc"
Nov 03 18:46:03.604085 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.604054    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:46:03.604085 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.604054    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:46:03.604429 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.604137    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:46:03.606462 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.606433    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7msdc"]
Nov 03 18:46:03.694419 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.694378    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxzkr\" (UniqueName: \"kubernetes.io/projected/f68896ba-8099-4ae5-a76d-262bd32fc504-kube-api-access-gxzkr\") pod \"maas-api-revoke-sub-046e1343-7msdc\" (UID: \"f68896ba-8099-4ae5-a76d-262bd32fc504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7msdc"
Nov 03 18:46:03.694590 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.694460    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68896ba-8099-4ae5-a76d-262bd32fc504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7msdc\" (UID: \"f68896ba-8099-4ae5-a76d-262bd32fc504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7msdc"
Nov 03 18:46:03.694590 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.694494    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68896ba-8099-4ae5-a76d-262bd32fc504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7msdc\" (UID: \"f68896ba-8099-4ae5-a76d-262bd32fc504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7msdc"
Nov 03 18:46:03.695275 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.695251    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68896ba-8099-4ae5-a76d-262bd32fc504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7msdc\" (UID: \"f68896ba-8099-4ae5-a76d-262bd32fc504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7msdc"
Nov 03 18:46:03.696891 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.696872    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68896ba-8099-4ae5-a76d-262bd32fc504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7msdc\" (UID: \"f68896ba-8099-4ae5-a76d-262bd32fc504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7msdc"
Nov 03 18:46:03.752987 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.752914    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxzkr\" (UniqueName: \"kubernetes.io/projected/f68896ba-8099-4ae5-a76d-262bd32fc504-kube-api-access-gxzkr\") pod \"maas-api-revoke-sub-046e1343-7msdc\" (UID: \"f68896ba-8099-4ae5-a76d-262bd32fc504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7msdc"
Nov 03 18:46:03.814968 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:03.814931    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7msdc"
Nov 03 18:46:04.182983 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:04.182931    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7msdc"]
Nov 03 18:46:04.186437 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:46:04.186411    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68896ba_8099_4ae5_a76d_262bd32fc504.slice/crio-c06f28f6a15e2d3d71cd83eacc01df05465f2013df33e984465b393f714eb42b WatchSource:0}: Error finding container c06f28f6a15e2d3d71cd83eacc01df05465f2013df33e984465b393f714eb42b: Status 404 returned error can't find the container with id c06f28f6a15e2d3d71cd83eacc01df05465f2013df33e984465b393f714eb42b
Nov 03 18:46:04.410979 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:04.410945    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7msdc" event={"ID":"f68896ba-8099-4ae5-a76d-262bd32fc504","Type":"ContainerStarted","Data":"c06f28f6a15e2d3d71cd83eacc01df05465f2013df33e984465b393f714eb42b"}
Nov 03 18:46:05.417980 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:05.417890    6059 generic.go:356] "Generic (PLEG): container finished" podID="f68896ba-8099-4ae5-a76d-262bd32fc504" containerID="e829fc3fb23c8bbddfecb93803ad1af2b2fbab78b6b1c7376ad1b739df0e488c" exitCode=0
Nov 03 18:46:05.418378 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:05.417971    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7msdc" event={"ID":"f68896ba-8099-4ae5-a76d-262bd32fc504","Type":"ContainerDied","Data":"e829fc3fb23c8bbddfecb93803ad1af2b2fbab78b6b1c7376ad1b739df0e488c"}
Nov 03 18:46:06.561970 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:06.561942    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7msdc"
Nov 03 18:46:06.626327 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:06.626296    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f68896ba-8099-4ae5-a76d-262bd32fc504-kube-api-access-gxzkr\" (UniqueName: \"kubernetes.io/projected/f68896ba-8099-4ae5-a76d-262bd32fc504-kube-api-access-gxzkr\") pod \"f68896ba-8099-4ae5-a76d-262bd32fc504\" (UID: \"f68896ba-8099-4ae5-a76d-262bd32fc504\") "
Nov 03 18:46:06.626515 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:06.626475    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f68896ba-8099-4ae5-a76d-262bd32fc504-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68896ba-8099-4ae5-a76d-262bd32fc504-maas-api-xks-ca\") pod \"f68896ba-8099-4ae5-a76d-262bd32fc504\" (UID: \"f68896ba-8099-4ae5-a76d-262bd32fc504\") "
Nov 03 18:46:06.626515 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:06.626510    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f68896ba-8099-4ae5-a76d-262bd32fc504-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68896ba-8099-4ae5-a76d-262bd32fc504-maas-api-openshift-ca\") pod \"f68896ba-8099-4ae5-a76d-262bd32fc504\" (UID: \"f68896ba-8099-4ae5-a76d-262bd32fc504\") "
Nov 03 18:46:06.626899 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:06.626873    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f68896ba-8099-4ae5-a76d-262bd32fc504-maas-api-openshift-ca" pod "f68896ba-8099-4ae5-a76d-262bd32fc504" (UID: "f68896ba-8099-4ae5-a76d-262bd32fc504"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:46:06.628433 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:06.628406    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f68896ba-8099-4ae5-a76d-262bd32fc504-maas-api-xks-ca" pod "f68896ba-8099-4ae5-a76d-262bd32fc504" (UID: "f68896ba-8099-4ae5-a76d-262bd32fc504"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:46:06.628558 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:06.628538    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f68896ba-8099-4ae5-a76d-262bd32fc504-kube-api-access-gxzkr" pod "f68896ba-8099-4ae5-a76d-262bd32fc504" (UID: "f68896ba-8099-4ae5-a76d-262bd32fc504"). InnerVolumeSpecName "kube-api-access-gxzkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:46:06.727927 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:06.727892    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxzkr\" (UniqueName: \"kubernetes.io/projected/f68896ba-8099-4ae5-a76d-262bd32fc504-kube-api-access-gxzkr\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:46:06.727927 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:06.727924    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68896ba-8099-4ae5-a76d-262bd32fc504-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:46:06.727927 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:06.727934    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68896ba-8099-4ae5-a76d-262bd32fc504-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:46:06.860901 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:46:06.860863    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:46:07.430177 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:07.430138    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7msdc"
Nov 03 18:46:07.430349 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:07.430176    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7msdc" event={"ID":"f68896ba-8099-4ae5-a76d-262bd32fc504","Type":"ContainerDied","Data":"c06f28f6a15e2d3d71cd83eacc01df05465f2013df33e984465b393f714eb42b"}
Nov 03 18:46:07.430349 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:07.430237    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c06f28f6a15e2d3d71cd83eacc01df05465f2013df33e984465b393f714eb42b"
Nov 03 18:46:14.260242 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:14.260205    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq"]
Nov 03 18:46:14.260663 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:14.260651    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f68896ba-8099-4ae5-a76d-262bd32fc504" containerName="revoke-keys"
Nov 03 18:46:14.260707 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:14.260664    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f68896ba-8099-4ae5-a76d-262bd32fc504" containerName="revoke-keys"
Nov 03 18:46:14.260766 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:14.260755    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f68896ba-8099-4ae5-a76d-262bd32fc504" containerName="revoke-keys"
Nov 03 18:46:14.264198 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:14.264177    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq"
Nov 03 18:46:14.314399 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:14.314375    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:46:14.361802 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:14.361773    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:46:14.361953 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:14.361802    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:46:14.361993 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:14.361982    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:46:14.365491 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:14.365470    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq"]
Nov 03 18:46:14.404320 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:14.404294    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7a8ac1-5141-4f40-b054-59dac8b00a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-bjvmq\" (UID: \"6a7a8ac1-5141-4f40-b054-59dac8b00a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq"
Nov 03 18:46:14.404493 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:14.404340    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7a8ac1-5141-4f40-b054-59dac8b00a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-bjvmq\" (UID: \"6a7a8ac1-5141-4f40-b054-59dac8b00a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq"
Nov 03 18:46:14.404596 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:14.404575    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5ggz\" (UniqueName: \"kubernetes.io/projected/6a7a8ac1-5141-4f40-b054-59dac8b00a28-kube-api-access-c5ggz\") pod \"maas-api-revoke-sub-d3fba056-bjvmq\" (UID: \"6a7a8ac1-5141-4f40-b054-59dac8b00a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq"
Nov 03 18:46:14.506205 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:14.506173    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5ggz\" (UniqueName: \"kubernetes.io/projected/6a7a8ac1-5141-4f40-b054-59dac8b00a28-kube-api-access-c5ggz\") pod \"maas-api-revoke-sub-d3fba056-bjvmq\" (UID: \"6a7a8ac1-5141-4f40-b054-59dac8b00a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq"
Nov 03 18:46:14.506205 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:14.506208    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7a8ac1-5141-4f40-b054-59dac8b00a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-bjvmq\" (UID: \"6a7a8ac1-5141-4f40-b054-59dac8b00a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq"
Nov 03 18:46:14.506510 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:14.506337    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7a8ac1-5141-4f40-b054-59dac8b00a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-bjvmq\" (UID: \"6a7a8ac1-5141-4f40-b054-59dac8b00a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq"
Nov 03 18:46:14.507004 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:14.506979    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7a8ac1-5141-4f40-b054-59dac8b00a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-bjvmq\" (UID: \"6a7a8ac1-5141-4f40-b054-59dac8b00a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq"
Nov 03 18:46:14.508546 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:14.508527    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7a8ac1-5141-4f40-b054-59dac8b00a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-bjvmq\" (UID: \"6a7a8ac1-5141-4f40-b054-59dac8b00a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq"
Nov 03 18:46:14.564919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:14.564853    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5ggz\" (UniqueName: \"kubernetes.io/projected/6a7a8ac1-5141-4f40-b054-59dac8b00a28-kube-api-access-c5ggz\") pod \"maas-api-revoke-sub-d3fba056-bjvmq\" (UID: \"6a7a8ac1-5141-4f40-b054-59dac8b00a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq"
Nov 03 18:46:14.579723 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:14.579695    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq"
Nov 03 18:46:14.949980 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:14.949953    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq"]
Nov 03 18:46:14.952082 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:46:14.952049    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7a8ac1_5141_4f40_b054_59dac8b00a28.slice/crio-2f7c7f1544a42e3036449d04865df21869ef88c62514eb17374bf2aab0c2cce0 WatchSource:0}: Error finding container 2f7c7f1544a42e3036449d04865df21869ef88c62514eb17374bf2aab0c2cce0: Status 404 returned error can't find the container with id 2f7c7f1544a42e3036449d04865df21869ef88c62514eb17374bf2aab0c2cce0
Nov 03 18:46:15.485421 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:15.485388    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq" event={"ID":"6a7a8ac1-5141-4f40-b054-59dac8b00a28","Type":"ContainerStarted","Data":"ded52d7ca40a16d90612a554a5b5ce8e15f218940e72cc64b52485d1fb1698da"}
Nov 03 18:46:15.485906 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:15.485451    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq" event={"ID":"6a7a8ac1-5141-4f40-b054-59dac8b00a28","Type":"ContainerStarted","Data":"2f7c7f1544a42e3036449d04865df21869ef88c62514eb17374bf2aab0c2cce0"}
Nov 03 18:46:15.597121 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:15.597016    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq" podStartSLOduration=1.5970016569999999 podStartE2EDuration="1.597001657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:46:15.595546014 +0000 UTC m=+5316.393258808" watchObservedRunningTime="2026-11-03 18:46:15.597001657 +0000 UTC m=+5316.394714489"
Nov 03 18:46:16.501093 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:16.501060    6059 generic.go:356] "Generic (PLEG): container finished" podID="6a7a8ac1-5141-4f40-b054-59dac8b00a28" containerID="ded52d7ca40a16d90612a554a5b5ce8e15f218940e72cc64b52485d1fb1698da" exitCode=0
Nov 03 18:46:16.501496 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:16.501157    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq" event={"ID":"6a7a8ac1-5141-4f40-b054-59dac8b00a28","Type":"ContainerDied","Data":"ded52d7ca40a16d90612a554a5b5ce8e15f218940e72cc64b52485d1fb1698da"}
Nov 03 18:46:17.732751 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:17.732709    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq"
Nov 03 18:46:17.843244 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:17.843137    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a7a8ac1-5141-4f40-b054-59dac8b00a28-kube-api-access-c5ggz\" (UniqueName: \"kubernetes.io/projected/6a7a8ac1-5141-4f40-b054-59dac8b00a28-kube-api-access-c5ggz\") pod \"6a7a8ac1-5141-4f40-b054-59dac8b00a28\" (UID: \"6a7a8ac1-5141-4f40-b054-59dac8b00a28\") "
Nov 03 18:46:17.843502 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:17.843243    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a7a8ac1-5141-4f40-b054-59dac8b00a28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7a8ac1-5141-4f40-b054-59dac8b00a28-maas-api-xks-ca\") pod \"6a7a8ac1-5141-4f40-b054-59dac8b00a28\" (UID: \"6a7a8ac1-5141-4f40-b054-59dac8b00a28\") "
Nov 03 18:46:17.843502 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:17.843457    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a7a8ac1-5141-4f40-b054-59dac8b00a28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7a8ac1-5141-4f40-b054-59dac8b00a28-maas-api-openshift-ca\") pod \"6a7a8ac1-5141-4f40-b054-59dac8b00a28\" (UID: \"6a7a8ac1-5141-4f40-b054-59dac8b00a28\") "
Nov 03 18:46:17.844381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:17.844334    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7a8ac1-5141-4f40-b054-59dac8b00a28-maas-api-openshift-ca" pod "6a7a8ac1-5141-4f40-b054-59dac8b00a28" (UID: "6a7a8ac1-5141-4f40-b054-59dac8b00a28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:46:17.846966 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:17.846920    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7a8ac1-5141-4f40-b054-59dac8b00a28-kube-api-access-c5ggz" pod "6a7a8ac1-5141-4f40-b054-59dac8b00a28" (UID: "6a7a8ac1-5141-4f40-b054-59dac8b00a28"). InnerVolumeSpecName "kube-api-access-c5ggz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:46:17.874066 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:17.862451    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7a8ac1-5141-4f40-b054-59dac8b00a28-maas-api-xks-ca" pod "6a7a8ac1-5141-4f40-b054-59dac8b00a28" (UID: "6a7a8ac1-5141-4f40-b054-59dac8b00a28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:46:17.945466 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:17.945337    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a7a8ac1-5141-4f40-b054-59dac8b00a28-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:46:17.945466 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:17.945391    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5ggz\" (UniqueName: \"kubernetes.io/projected/6a7a8ac1-5141-4f40-b054-59dac8b00a28-kube-api-access-c5ggz\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:46:17.945466 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:17.945408    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a7a8ac1-5141-4f40-b054-59dac8b00a28-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:46:18.529380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:18.528832    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq" event={"ID":"6a7a8ac1-5141-4f40-b054-59dac8b00a28","Type":"ContainerDied","Data":"2f7c7f1544a42e3036449d04865df21869ef88c62514eb17374bf2aab0c2cce0"}
Nov 03 18:46:18.529380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:18.528898    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f7c7f1544a42e3036449d04865df21869ef88c62514eb17374bf2aab0c2cce0"
Nov 03 18:46:18.529380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:18.528965    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq"
Nov 03 18:46:36.994921 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:46:36.994877    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:46:37.155522 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:37.155476    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-l2xpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:46:37.155851 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:37.155817    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" podUID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:46:37.270693 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:37.270495    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gxxkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:46:37.270693 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:37.270578    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" podUID="2159e743-a61b-4d76-9f78-9be36e825a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:46:49.891457 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:49.891412    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-b98zt"]
Nov 03 18:46:49.891954 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:49.891874    6059 cpu_manager.go:414] "containerMap: removing container" podUID="6a7a8ac1-5141-4f40-b054-59dac8b00a28" containerName="revoke-keys"
Nov 03 18:46:49.891954 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:49.891888    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7a8ac1-5141-4f40-b054-59dac8b00a28" containerName="revoke-keys"
Nov 03 18:46:49.892031 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:49.891969    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a7a8ac1-5141-4f40-b054-59dac8b00a28" containerName="revoke-keys"
Nov 03 18:46:49.895819 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:49.895795    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-b98zt"
Nov 03 18:46:49.954260 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:49.954221    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:46:49.954453 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:49.954298    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:46:49.980833 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:49.980796    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52g4x\" (UniqueName: \"kubernetes.io/projected/75c24353-2065-48bc-95c5-e6e590de013d-kube-api-access-52g4x\") pod \"maas-api-revoke-sub-ae594979-b98zt\" (UID: \"75c24353-2065-48bc-95c5-e6e590de013d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-b98zt"
Nov 03 18:46:49.981005 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:49.980851    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c24353-2065-48bc-95c5-e6e590de013d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-b98zt\" (UID: \"75c24353-2065-48bc-95c5-e6e590de013d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-b98zt"
Nov 03 18:46:49.981005 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:49.980874    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c24353-2065-48bc-95c5-e6e590de013d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-b98zt\" (UID: \"75c24353-2065-48bc-95c5-e6e590de013d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-b98zt"
Nov 03 18:46:50.001714 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:50.001682    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:46:50.001955 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:50.001771    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:46:50.005212 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:50.005183    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-b98zt"]
Nov 03 18:46:50.082318 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:50.082270    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52g4x\" (UniqueName: \"kubernetes.io/projected/75c24353-2065-48bc-95c5-e6e590de013d-kube-api-access-52g4x\") pod \"maas-api-revoke-sub-ae594979-b98zt\" (UID: \"75c24353-2065-48bc-95c5-e6e590de013d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-b98zt"
Nov 03 18:46:50.082586 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:50.082336    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c24353-2065-48bc-95c5-e6e590de013d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-b98zt\" (UID: \"75c24353-2065-48bc-95c5-e6e590de013d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-b98zt"
Nov 03 18:46:50.082586 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:50.082396    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c24353-2065-48bc-95c5-e6e590de013d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-b98zt\" (UID: \"75c24353-2065-48bc-95c5-e6e590de013d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-b98zt"
Nov 03 18:46:50.083072 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:50.083036    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c24353-2065-48bc-95c5-e6e590de013d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-b98zt\" (UID: \"75c24353-2065-48bc-95c5-e6e590de013d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-b98zt"
Nov 03 18:46:50.084766 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:50.084747    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c24353-2065-48bc-95c5-e6e590de013d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-b98zt\" (UID: \"75c24353-2065-48bc-95c5-e6e590de013d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-b98zt"
Nov 03 18:46:50.141261 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:50.141229    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52g4x\" (UniqueName: \"kubernetes.io/projected/75c24353-2065-48bc-95c5-e6e590de013d-kube-api-access-52g4x\") pod \"maas-api-revoke-sub-ae594979-b98zt\" (UID: \"75c24353-2065-48bc-95c5-e6e590de013d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-b98zt"
Nov 03 18:46:50.208691 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:50.208655    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-b98zt"
Nov 03 18:46:50.583294 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:50.583257    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-b98zt"]
Nov 03 18:46:50.585862 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:46:50.585830    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c24353_2065_48bc_95c5_e6e590de013d.slice/crio-2b6e4fd2ccd1122f52ac1eae7e88be7460b40e81798c4af958d9cddf7310876f WatchSource:0}: Error finding container 2b6e4fd2ccd1122f52ac1eae7e88be7460b40e81798c4af958d9cddf7310876f: Status 404 returned error can't find the container with id 2b6e4fd2ccd1122f52ac1eae7e88be7460b40e81798c4af958d9cddf7310876f
Nov 03 18:46:50.770062 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:50.770026    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-b98zt" event={"ID":"75c24353-2065-48bc-95c5-e6e590de013d","Type":"ContainerStarted","Data":"2b6e4fd2ccd1122f52ac1eae7e88be7460b40e81798c4af958d9cddf7310876f"}
Nov 03 18:46:51.777307 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:51.777272    6059 generic.go:356] "Generic (PLEG): container finished" podID="75c24353-2065-48bc-95c5-e6e590de013d" containerID="93ab09bfc57910b7046d4f8d1652a6f70909b88f5f73a02e013b69cc09c7d791" exitCode=0
Nov 03 18:46:51.777782 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:51.777377    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-b98zt" event={"ID":"75c24353-2065-48bc-95c5-e6e590de013d","Type":"ContainerDied","Data":"93ab09bfc57910b7046d4f8d1652a6f70909b88f5f73a02e013b69cc09c7d791"}
Nov 03 18:46:52.933164 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:52.933137    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-b98zt"
Nov 03 18:46:53.112926 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:53.112837    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75c24353-2065-48bc-95c5-e6e590de013d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c24353-2065-48bc-95c5-e6e590de013d-maas-api-openshift-ca\") pod \"75c24353-2065-48bc-95c5-e6e590de013d\" (UID: \"75c24353-2065-48bc-95c5-e6e590de013d\") "
Nov 03 18:46:53.113091 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:53.112945    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75c24353-2065-48bc-95c5-e6e590de013d-kube-api-access-52g4x\" (UniqueName: \"kubernetes.io/projected/75c24353-2065-48bc-95c5-e6e590de013d-kube-api-access-52g4x\") pod \"75c24353-2065-48bc-95c5-e6e590de013d\" (UID: \"75c24353-2065-48bc-95c5-e6e590de013d\") "
Nov 03 18:46:53.113091 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:53.112987    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75c24353-2065-48bc-95c5-e6e590de013d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c24353-2065-48bc-95c5-e6e590de013d-maas-api-xks-ca\") pod \"75c24353-2065-48bc-95c5-e6e590de013d\" (UID: \"75c24353-2065-48bc-95c5-e6e590de013d\") "
Nov 03 18:46:53.113321 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:53.113296    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75c24353-2065-48bc-95c5-e6e590de013d-maas-api-openshift-ca" pod "75c24353-2065-48bc-95c5-e6e590de013d" (UID: "75c24353-2065-48bc-95c5-e6e590de013d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:46:53.115198 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:53.115172    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75c24353-2065-48bc-95c5-e6e590de013d-maas-api-xks-ca" pod "75c24353-2065-48bc-95c5-e6e590de013d" (UID: "75c24353-2065-48bc-95c5-e6e590de013d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:46:53.115316 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:53.115201    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c24353-2065-48bc-95c5-e6e590de013d-kube-api-access-52g4x" pod "75c24353-2065-48bc-95c5-e6e590de013d" (UID: "75c24353-2065-48bc-95c5-e6e590de013d"). InnerVolumeSpecName "kube-api-access-52g4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:46:53.214209 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:53.214161    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c24353-2065-48bc-95c5-e6e590de013d-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:46:53.214209 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:53.214201    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c24353-2065-48bc-95c5-e6e590de013d-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:46:53.214209 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:53.214211    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52g4x\" (UniqueName: \"kubernetes.io/projected/75c24353-2065-48bc-95c5-e6e590de013d-kube-api-access-52g4x\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:46:53.789426 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:53.789395    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-b98zt"
Nov 03 18:46:53.789620 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:53.789389    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-b98zt" event={"ID":"75c24353-2065-48bc-95c5-e6e590de013d","Type":"ContainerDied","Data":"2b6e4fd2ccd1122f52ac1eae7e88be7460b40e81798c4af958d9cddf7310876f"}
Nov 03 18:46:53.789620 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:46:53.789545    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b6e4fd2ccd1122f52ac1eae7e88be7460b40e81798c4af958d9cddf7310876f"
Nov 03 18:47:01.146043 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:01.145995    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:01.334388 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:01.334338    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:06.861897 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:47:06.861850    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:47:08.654089 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:08.654033    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb"]
Nov 03 18:47:08.654801 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:08.654774    6059 cpu_manager.go:414] "containerMap: removing container" podUID="75c24353-2065-48bc-95c5-e6e590de013d" containerName="revoke-keys"
Nov 03 18:47:08.654898 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:08.654804    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75c24353-2065-48bc-95c5-e6e590de013d" containerName="revoke-keys"
Nov 03 18:47:08.654967 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:08.654949    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="75c24353-2065-48bc-95c5-e6e590de013d" containerName="revoke-keys"
Nov 03 18:47:08.659492 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:08.659460    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb"
Nov 03 18:47:08.709413 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:08.709369    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:47:08.709631 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:08.709446    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:47:08.758987 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:08.758918    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:47:08.760776 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:08.759469    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:47:08.763291 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:08.763261    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb"]
Nov 03 18:47:08.783278 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:08.783216    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fce4902-178e-4c23-88d0-1d7190733721-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-glfvb\" (UID: \"1fce4902-178e-4c23-88d0-1d7190733721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb"
Nov 03 18:47:08.783522 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:08.783397    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fce4902-178e-4c23-88d0-1d7190733721-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-glfvb\" (UID: \"1fce4902-178e-4c23-88d0-1d7190733721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb"
Nov 03 18:47:08.783914 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:08.783528    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tfx9\" (UniqueName: \"kubernetes.io/projected/1fce4902-178e-4c23-88d0-1d7190733721-kube-api-access-9tfx9\") pod \"maas-api-revoke-sub-a4aef7d7-glfvb\" (UID: \"1fce4902-178e-4c23-88d0-1d7190733721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb"
Nov 03 18:47:08.887100 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:08.887004    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fce4902-178e-4c23-88d0-1d7190733721-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-glfvb\" (UID: \"1fce4902-178e-4c23-88d0-1d7190733721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb"
Nov 03 18:47:08.888030 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:08.888005    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fce4902-178e-4c23-88d0-1d7190733721-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-glfvb\" (UID: \"1fce4902-178e-4c23-88d0-1d7190733721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb"
Nov 03 18:47:08.889060 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:08.888945    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fce4902-178e-4c23-88d0-1d7190733721-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-glfvb\" (UID: \"1fce4902-178e-4c23-88d0-1d7190733721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb"
Nov 03 18:47:08.889322 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:08.889302    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tfx9\" (UniqueName: \"kubernetes.io/projected/1fce4902-178e-4c23-88d0-1d7190733721-kube-api-access-9tfx9\") pod \"maas-api-revoke-sub-a4aef7d7-glfvb\" (UID: \"1fce4902-178e-4c23-88d0-1d7190733721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb"
Nov 03 18:47:08.905158 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:08.905079    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fce4902-178e-4c23-88d0-1d7190733721-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-glfvb\" (UID: \"1fce4902-178e-4c23-88d0-1d7190733721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb"
Nov 03 18:47:08.975389 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:08.968396    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tfx9\" (UniqueName: \"kubernetes.io/projected/1fce4902-178e-4c23-88d0-1d7190733721-kube-api-access-9tfx9\") pod \"maas-api-revoke-sub-a4aef7d7-glfvb\" (UID: \"1fce4902-178e-4c23-88d0-1d7190733721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb"
Nov 03 18:47:08.984392 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:08.982447    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb"
Nov 03 18:47:09.773382 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:09.748942    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb"]
Nov 03 18:47:09.837803 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:47:09.827413    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fce4902_178e_4c23_88d0_1d7190733721.slice/crio-9fd27e8972645f399d74f65e2125921a4dcdeb0b4254c4e54e68fc5a58679755 WatchSource:0}: Error finding container 9fd27e8972645f399d74f65e2125921a4dcdeb0b4254c4e54e68fc5a58679755: Status 404 returned error can't find the container with id 9fd27e8972645f399d74f65e2125921a4dcdeb0b4254c4e54e68fc5a58679755
Nov 03 18:47:09.924693 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:09.924649    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb" event={"ID":"1fce4902-178e-4c23-88d0-1d7190733721","Type":"ContainerStarted","Data":"9fd27e8972645f399d74f65e2125921a4dcdeb0b4254c4e54e68fc5a58679755"}
Nov 03 18:47:10.966382 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:10.947384    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb" event={"ID":"1fce4902-178e-4c23-88d0-1d7190733721","Type":"ContainerStarted","Data":"d4e5fff7f08bde94cc56a4ba0fcb88ba06c6e80271f78e4b77e0f6f9a3ab1bc6"}
Nov 03 18:47:11.071595 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:11.070782    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb" podStartSLOduration=3.070759646 podStartE2EDuration="3.070759646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:47:11.065954472 +0000 UTC m=+5371.863667264" watchObservedRunningTime="2026-11-03 18:47:11.070759646 +0000 UTC m=+5371.868472428"
Nov 03 18:47:11.960727 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:11.960677    6059 generic.go:356] "Generic (PLEG): container finished" podID="1fce4902-178e-4c23-88d0-1d7190733721" containerID="d4e5fff7f08bde94cc56a4ba0fcb88ba06c6e80271f78e4b77e0f6f9a3ab1bc6" exitCode=0
Nov 03 18:47:11.960964 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:11.960802    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb" event={"ID":"1fce4902-178e-4c23-88d0-1d7190733721","Type":"ContainerDied","Data":"d4e5fff7f08bde94cc56a4ba0fcb88ba06c6e80271f78e4b77e0f6f9a3ab1bc6"}
Nov 03 18:47:13.108683 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:13.108655    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb"
Nov 03 18:47:13.168767 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:13.168733    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fce4902-178e-4c23-88d0-1d7190733721-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fce4902-178e-4c23-88d0-1d7190733721-maas-api-xks-ca\") pod \"1fce4902-178e-4c23-88d0-1d7190733721\" (UID: \"1fce4902-178e-4c23-88d0-1d7190733721\") "
Nov 03 18:47:13.168983 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:13.168872    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fce4902-178e-4c23-88d0-1d7190733721-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fce4902-178e-4c23-88d0-1d7190733721-maas-api-openshift-ca\") pod \"1fce4902-178e-4c23-88d0-1d7190733721\" (UID: \"1fce4902-178e-4c23-88d0-1d7190733721\") "
Nov 03 18:47:13.168983 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:13.168909    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fce4902-178e-4c23-88d0-1d7190733721-kube-api-access-9tfx9\" (UniqueName: \"kubernetes.io/projected/1fce4902-178e-4c23-88d0-1d7190733721-kube-api-access-9tfx9\") pod \"1fce4902-178e-4c23-88d0-1d7190733721\" (UID: \"1fce4902-178e-4c23-88d0-1d7190733721\") "
Nov 03 18:47:13.169273 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:13.169247    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fce4902-178e-4c23-88d0-1d7190733721-maas-api-openshift-ca" pod "1fce4902-178e-4c23-88d0-1d7190733721" (UID: "1fce4902-178e-4c23-88d0-1d7190733721"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:47:13.170891 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:13.170871    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fce4902-178e-4c23-88d0-1d7190733721-maas-api-xks-ca" pod "1fce4902-178e-4c23-88d0-1d7190733721" (UID: "1fce4902-178e-4c23-88d0-1d7190733721"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:47:13.171082 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:13.171054    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fce4902-178e-4c23-88d0-1d7190733721-kube-api-access-9tfx9" pod "1fce4902-178e-4c23-88d0-1d7190733721" (UID: "1fce4902-178e-4c23-88d0-1d7190733721"). InnerVolumeSpecName "kube-api-access-9tfx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:47:13.269918 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:13.269826    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fce4902-178e-4c23-88d0-1d7190733721-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:47:13.269918 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:13.269855    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tfx9\" (UniqueName: \"kubernetes.io/projected/1fce4902-178e-4c23-88d0-1d7190733721-kube-api-access-9tfx9\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:47:13.269918 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:13.269866    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fce4902-178e-4c23-88d0-1d7190733721-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:47:13.972885 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:13.972855    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb"
Nov 03 18:47:13.973075 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:13.972863    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb" event={"ID":"1fce4902-178e-4c23-88d0-1d7190733721","Type":"ContainerDied","Data":"9fd27e8972645f399d74f65e2125921a4dcdeb0b4254c4e54e68fc5a58679755"}
Nov 03 18:47:13.973075 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:13.972968    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fd27e8972645f399d74f65e2125921a4dcdeb0b4254c4e54e68fc5a58679755"
Nov 03 18:47:29.932899 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:29.932856    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9jbg9"]
Nov 03 18:47:29.934790 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:29.934744    6059 cpu_manager.go:414] "containerMap: removing container" podUID="1fce4902-178e-4c23-88d0-1d7190733721" containerName="revoke-keys"
Nov 03 18:47:29.935035 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:29.935015    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fce4902-178e-4c23-88d0-1d7190733721" containerName="revoke-keys"
Nov 03 18:47:29.935525 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:29.935489    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fce4902-178e-4c23-88d0-1d7190733721" containerName="revoke-keys"
Nov 03 18:47:29.940251 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:29.940219    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9jbg9"
Nov 03 18:47:29.990693 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:29.990659    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:47:29.991484 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:29.991457    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:47:30.038037 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:30.038000    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:47:30.042522 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:30.042316    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9jbg9"]
Nov 03 18:47:30.043005 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:30.042983    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:47:30.060706 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:30.060671    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90136c7-08d4-4ab1-88b2-a2f2a2dd3595-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-9jbg9\" (UID: \"e90136c7-08d4-4ab1-88b2-a2f2a2dd3595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9jbg9"
Nov 03 18:47:30.061035 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:30.061010    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90136c7-08d4-4ab1-88b2-a2f2a2dd3595-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-9jbg9\" (UID: \"e90136c7-08d4-4ab1-88b2-a2f2a2dd3595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9jbg9"
Nov 03 18:47:30.061268 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:30.061243    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcrrr\" (UniqueName: \"kubernetes.io/projected/e90136c7-08d4-4ab1-88b2-a2f2a2dd3595-kube-api-access-lcrrr\") pod \"maas-api-revoke-sub-79805ca9-9jbg9\" (UID: \"e90136c7-08d4-4ab1-88b2-a2f2a2dd3595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9jbg9"
Nov 03 18:47:30.163031 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:30.162981    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90136c7-08d4-4ab1-88b2-a2f2a2dd3595-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-9jbg9\" (UID: \"e90136c7-08d4-4ab1-88b2-a2f2a2dd3595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9jbg9"
Nov 03 18:47:30.163270 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:30.163098    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcrrr\" (UniqueName: \"kubernetes.io/projected/e90136c7-08d4-4ab1-88b2-a2f2a2dd3595-kube-api-access-lcrrr\") pod \"maas-api-revoke-sub-79805ca9-9jbg9\" (UID: \"e90136c7-08d4-4ab1-88b2-a2f2a2dd3595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9jbg9"
Nov 03 18:47:30.163270 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:30.163217    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90136c7-08d4-4ab1-88b2-a2f2a2dd3595-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-9jbg9\" (UID: \"e90136c7-08d4-4ab1-88b2-a2f2a2dd3595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9jbg9"
Nov 03 18:47:30.164340 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:30.164304    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90136c7-08d4-4ab1-88b2-a2f2a2dd3595-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-9jbg9\" (UID: \"e90136c7-08d4-4ab1-88b2-a2f2a2dd3595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9jbg9"
Nov 03 18:47:30.166411 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:30.166383    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90136c7-08d4-4ab1-88b2-a2f2a2dd3595-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-9jbg9\" (UID: \"e90136c7-08d4-4ab1-88b2-a2f2a2dd3595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9jbg9"
Nov 03 18:47:30.223524 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:30.223485    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcrrr\" (UniqueName: \"kubernetes.io/projected/e90136c7-08d4-4ab1-88b2-a2f2a2dd3595-kube-api-access-lcrrr\") pod \"maas-api-revoke-sub-79805ca9-9jbg9\" (UID: \"e90136c7-08d4-4ab1-88b2-a2f2a2dd3595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9jbg9"
Nov 03 18:47:30.270537 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:30.270485    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9jbg9"
Nov 03 18:47:30.708105 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:30.707502    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9jbg9"]
Nov 03 18:47:31.111700 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:31.111601    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9jbg9" event={"ID":"e90136c7-08d4-4ab1-88b2-a2f2a2dd3595","Type":"ContainerStarted","Data":"e052b6a85370a13c63657429d673fb1d73de07387028c3b7b2434d9e62e7b21e"}
Nov 03 18:47:32.101384 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:32.081712    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xpshn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:47:32.101384 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:32.081789    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn" podUID="d9e3abf7-c5e1-4414-858e-5f84f6a88670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:47:32.119850 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:32.119734    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zp9n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:47:32.152997 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:32.152937    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7" podUID="60b66cb6-2bce-4c02-aa7d-7eb25f29f237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:47:32.252444 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:32.225371    6059 generic.go:356] "Generic (PLEG): container finished" podID="e90136c7-08d4-4ab1-88b2-a2f2a2dd3595" containerID="3a5ebaf5835ce76e506950c4b51ddb3d9a039b4a2b86285163aeab5a0d08f8d2" exitCode=0
Nov 03 18:47:32.252444 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:32.225567    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9jbg9" event={"ID":"e90136c7-08d4-4ab1-88b2-a2f2a2dd3595","Type":"ContainerDied","Data":"3a5ebaf5835ce76e506950c4b51ddb3d9a039b4a2b86285163aeab5a0d08f8d2"}
Nov 03 18:47:32.325121 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:47:32.325069    6059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90136c7_08d4_4ab1_88b2_a2f2a2dd3595.slice/crio-3a5ebaf5835ce76e506950c4b51ddb3d9a039b4a2b86285163aeab5a0d08f8d2.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90136c7_08d4_4ab1_88b2_a2f2a2dd3595.slice/crio-conmon-3a5ebaf5835ce76e506950c4b51ddb3d9a039b4a2b86285163aeab5a0d08f8d2.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 18:47:32.326262 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:47:32.326188    6059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90136c7_08d4_4ab1_88b2_a2f2a2dd3595.slice/crio-3a5ebaf5835ce76e506950c4b51ddb3d9a039b4a2b86285163aeab5a0d08f8d2.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90136c7_08d4_4ab1_88b2_a2f2a2dd3595.slice/crio-conmon-3a5ebaf5835ce76e506950c4b51ddb3d9a039b4a2b86285163aeab5a0d08f8d2.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 18:47:33.631916 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:33.631878    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9jbg9"
Nov 03 18:47:33.675749 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:33.675706    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e90136c7-08d4-4ab1-88b2-a2f2a2dd3595-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90136c7-08d4-4ab1-88b2-a2f2a2dd3595-maas-api-openshift-ca\") pod \"e90136c7-08d4-4ab1-88b2-a2f2a2dd3595\" (UID: \"e90136c7-08d4-4ab1-88b2-a2f2a2dd3595\") "
Nov 03 18:47:33.675953 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:33.675892    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e90136c7-08d4-4ab1-88b2-a2f2a2dd3595-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90136c7-08d4-4ab1-88b2-a2f2a2dd3595-maas-api-xks-ca\") pod \"e90136c7-08d4-4ab1-88b2-a2f2a2dd3595\" (UID: \"e90136c7-08d4-4ab1-88b2-a2f2a2dd3595\") "
Nov 03 18:47:33.676023 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:33.675963    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e90136c7-08d4-4ab1-88b2-a2f2a2dd3595-kube-api-access-lcrrr\" (UniqueName: \"kubernetes.io/projected/e90136c7-08d4-4ab1-88b2-a2f2a2dd3595-kube-api-access-lcrrr\") pod \"e90136c7-08d4-4ab1-88b2-a2f2a2dd3595\" (UID: \"e90136c7-08d4-4ab1-88b2-a2f2a2dd3595\") "
Nov 03 18:47:33.676102 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:33.676076    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90136c7-08d4-4ab1-88b2-a2f2a2dd3595-maas-api-openshift-ca" pod "e90136c7-08d4-4ab1-88b2-a2f2a2dd3595" (UID: "e90136c7-08d4-4ab1-88b2-a2f2a2dd3595"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:47:33.676264 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:33.676248    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90136c7-08d4-4ab1-88b2-a2f2a2dd3595-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:47:33.678024 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:33.677998    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e90136c7-08d4-4ab1-88b2-a2f2a2dd3595-maas-api-xks-ca" pod "e90136c7-08d4-4ab1-88b2-a2f2a2dd3595" (UID: "e90136c7-08d4-4ab1-88b2-a2f2a2dd3595"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:47:33.678117 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:33.678087    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90136c7-08d4-4ab1-88b2-a2f2a2dd3595-kube-api-access-lcrrr" pod "e90136c7-08d4-4ab1-88b2-a2f2a2dd3595" (UID: "e90136c7-08d4-4ab1-88b2-a2f2a2dd3595"). InnerVolumeSpecName "kube-api-access-lcrrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:47:33.777637 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:33.777539    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcrrr\" (UniqueName: \"kubernetes.io/projected/e90136c7-08d4-4ab1-88b2-a2f2a2dd3595-kube-api-access-lcrrr\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:47:33.777637 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:33.777578    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90136c7-08d4-4ab1-88b2-a2f2a2dd3595-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:47:34.255425 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:34.255388    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9jbg9"
Nov 03 18:47:34.255425 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:34.255401    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9jbg9" event={"ID":"e90136c7-08d4-4ab1-88b2-a2f2a2dd3595","Type":"ContainerDied","Data":"e052b6a85370a13c63657429d673fb1d73de07387028c3b7b2434d9e62e7b21e"}
Nov 03 18:47:34.255659 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:34.255446    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e052b6a85370a13c63657429d673fb1d73de07387028c3b7b2434d9e62e7b21e"
Nov 03 18:47:36.861573 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:47:36.861535    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:47:40.155129 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:40.155102    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:47:40.176425 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:40.176397    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:47:40.182999 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:40.182972    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:47:40.199735 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:40.199708    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:47:47.448701 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:47.448600    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqkgt"]
Nov 03 18:47:47.449332 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:47.449305    6059 cpu_manager.go:414] "containerMap: removing container" podUID="e90136c7-08d4-4ab1-88b2-a2f2a2dd3595" containerName="revoke-keys"
Nov 03 18:47:47.449479 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:47.449335    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90136c7-08d4-4ab1-88b2-a2f2a2dd3595" containerName="revoke-keys"
Nov 03 18:47:47.449555 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:47.449513    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e90136c7-08d4-4ab1-88b2-a2f2a2dd3595" containerName="revoke-keys"
Nov 03 18:47:47.459830 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:47.459774    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqkgt"
Nov 03 18:47:47.509795 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:47.509742    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqkgt"]
Nov 03 18:47:47.510406 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:47.510338    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:47:47.535387 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:47.529800    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mbtq\" (UniqueName: \"kubernetes.io/projected/5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a-kube-api-access-8mbtq\") pod \"maas-api-revoke-sub-0b7e8edb-pqkgt\" (UID: \"5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqkgt"
Nov 03 18:47:47.535387 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:47.530095    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pqkgt\" (UID: \"5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqkgt"
Nov 03 18:47:47.535387 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:47.530160    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pqkgt\" (UID: \"5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqkgt"
Nov 03 18:47:47.558015 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:47.557983    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:47:47.558433 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:47.558412    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:47:47.558680 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:47.558664    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:47:47.631038 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:47.630992    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pqkgt\" (UID: \"5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqkgt"
Nov 03 18:47:47.631389 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:47.631336    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mbtq\" (UniqueName: \"kubernetes.io/projected/5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a-kube-api-access-8mbtq\") pod \"maas-api-revoke-sub-0b7e8edb-pqkgt\" (UID: \"5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqkgt"
Nov 03 18:47:47.631710 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:47.631675    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pqkgt\" (UID: \"5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqkgt"
Nov 03 18:47:47.633509 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:47.633486    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pqkgt\" (UID: \"5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqkgt"
Nov 03 18:47:47.634954 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:47.634931    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pqkgt\" (UID: \"5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqkgt"
Nov 03 18:47:47.692697 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:47.692660    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mbtq\" (UniqueName: \"kubernetes.io/projected/5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a-kube-api-access-8mbtq\") pod \"maas-api-revoke-sub-0b7e8edb-pqkgt\" (UID: \"5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqkgt"
Nov 03 18:47:47.786745 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:47.786695    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqkgt"
Nov 03 18:47:48.193174 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:48.193130    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqkgt"]
Nov 03 18:47:48.351581 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:48.351535    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqkgt" event={"ID":"5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a","Type":"ContainerStarted","Data":"966374a441025b82cb9982d67b1e99c8ebd5a8461ed9f56b9452b3fe9f5ed3d7"}
Nov 03 18:47:49.432380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:49.430259    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqkgt" event={"ID":"5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a","Type":"ContainerStarted","Data":"a68730cf06d9778bfd8d3aafda78e49f3d1519891b130da6ec9c1445d79a40d8"}
Nov 03 18:47:50.467382 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:50.466478    6059 generic.go:356] "Generic (PLEG): container finished" podID="5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a" containerID="a68730cf06d9778bfd8d3aafda78e49f3d1519891b130da6ec9c1445d79a40d8" exitCode=0
Nov 03 18:47:50.467382 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:50.466630    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqkgt" event={"ID":"5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a","Type":"ContainerDied","Data":"a68730cf06d9778bfd8d3aafda78e49f3d1519891b130da6ec9c1445d79a40d8"}
Nov 03 18:47:51.220609 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:51.220565    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqkgt"
Nov 03 18:47:51.227402 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:51.227343    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a-kube-api-access-8mbtq\" (UniqueName: \"kubernetes.io/projected/5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a-kube-api-access-8mbtq\") pod \"5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a\" (UID: \"5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a\") "
Nov 03 18:47:51.227613 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:51.227415    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a-maas-api-xks-ca\") pod \"5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a\" (UID: \"5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a\") "
Nov 03 18:47:51.227613 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:51.227519    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a-maas-api-openshift-ca\") pod \"5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a\" (UID: \"5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a\") "
Nov 03 18:47:51.228367 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:51.228307    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a-maas-api-openshift-ca" pod "5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a" (UID: "5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:47:51.250426 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:51.249309    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a-maas-api-xks-ca" pod "5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a" (UID: "5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:47:51.251146 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:51.251113    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a-kube-api-access-8mbtq" pod "5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a" (UID: "5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a"). InnerVolumeSpecName "kube-api-access-8mbtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:47:51.428975 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:51.428881    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:47:51.429198 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:51.429181    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mbtq\" (UniqueName: \"kubernetes.io/projected/5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a-kube-api-access-8mbtq\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:47:51.429303 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:51.429277    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:47:51.508338 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:51.507314    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqkgt" event={"ID":"5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a","Type":"ContainerDied","Data":"966374a441025b82cb9982d67b1e99c8ebd5a8461ed9f56b9452b3fe9f5ed3d7"}
Nov 03 18:47:51.508338 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:51.507418    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="966374a441025b82cb9982d67b1e99c8ebd5a8461ed9f56b9452b3fe9f5ed3d7"
Nov 03 18:47:51.508338 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:51.507495    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqkgt"
Nov 03 18:47:58.803750 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:58.803699    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pp7tl"]
Nov 03 18:47:58.814892 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:58.814858    6059 cpu_manager.go:414] "containerMap: removing container" podUID="5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a" containerName="revoke-keys"
Nov 03 18:47:58.815106 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:58.815083    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a" containerName="revoke-keys"
Nov 03 18:47:58.815659 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:58.815634    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a" containerName="revoke-keys"
Nov 03 18:47:58.819954 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:58.819930    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pp7tl"
Nov 03 18:47:58.866000 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:58.865967    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pp7tl"]
Nov 03 18:47:58.909915 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:58.909877    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f9265bc-0969-4f8b-8f9d-6947d6b78847-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pp7tl\" (UID: \"6f9265bc-0969-4f8b-8f9d-6947d6b78847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pp7tl"
Nov 03 18:47:58.910132 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:58.909947    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f9265bc-0969-4f8b-8f9d-6947d6b78847-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pp7tl\" (UID: \"6f9265bc-0969-4f8b-8f9d-6947d6b78847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pp7tl"
Nov 03 18:47:58.910132 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:58.910069    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl7b2\" (UniqueName: \"kubernetes.io/projected/6f9265bc-0969-4f8b-8f9d-6947d6b78847-kube-api-access-rl7b2\") pod \"maas-api-revoke-sub-3d4156f1-pp7tl\" (UID: \"6f9265bc-0969-4f8b-8f9d-6947d6b78847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pp7tl"
Nov 03 18:47:58.910972 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:58.910947    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:47:58.911324 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:58.911304    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:47:58.918098 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:58.918074    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:47:58.918217 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:58.918148    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:47:59.010809 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:59.010771    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f9265bc-0969-4f8b-8f9d-6947d6b78847-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pp7tl\" (UID: \"6f9265bc-0969-4f8b-8f9d-6947d6b78847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pp7tl"
Nov 03 18:47:59.011033 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:59.010850    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f9265bc-0969-4f8b-8f9d-6947d6b78847-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pp7tl\" (UID: \"6f9265bc-0969-4f8b-8f9d-6947d6b78847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pp7tl"
Nov 03 18:47:59.011033 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:59.010891    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl7b2\" (UniqueName: \"kubernetes.io/projected/6f9265bc-0969-4f8b-8f9d-6947d6b78847-kube-api-access-rl7b2\") pod \"maas-api-revoke-sub-3d4156f1-pp7tl\" (UID: \"6f9265bc-0969-4f8b-8f9d-6947d6b78847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pp7tl"
Nov 03 18:47:59.011605 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:59.011579    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f9265bc-0969-4f8b-8f9d-6947d6b78847-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pp7tl\" (UID: \"6f9265bc-0969-4f8b-8f9d-6947d6b78847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pp7tl"
Nov 03 18:47:59.013293 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:59.013270    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f9265bc-0969-4f8b-8f9d-6947d6b78847-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pp7tl\" (UID: \"6f9265bc-0969-4f8b-8f9d-6947d6b78847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pp7tl"
Nov 03 18:47:59.070440 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:59.070331    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl7b2\" (UniqueName: \"kubernetes.io/projected/6f9265bc-0969-4f8b-8f9d-6947d6b78847-kube-api-access-rl7b2\") pod \"maas-api-revoke-sub-3d4156f1-pp7tl\" (UID: \"6f9265bc-0969-4f8b-8f9d-6947d6b78847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pp7tl"
Nov 03 18:47:59.134703 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:59.134664    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pp7tl"
Nov 03 18:47:59.509051 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:59.508955    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pp7tl"]
Nov 03 18:47:59.510189 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:47:59.510159    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f9265bc_0969_4f8b_8f9d_6947d6b78847.slice/crio-8a30b12465b910d87fff69c347afe0647cf10c0c8893596791161431d91e632b WatchSource:0}: Error finding container 8a30b12465b910d87fff69c347afe0647cf10c0c8893596791161431d91e632b: Status 404 returned error can't find the container with id 8a30b12465b910d87fff69c347afe0647cf10c0c8893596791161431d91e632b
Nov 03 18:47:59.553981 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:47:59.553951    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pp7tl" event={"ID":"6f9265bc-0969-4f8b-8f9d-6947d6b78847","Type":"ContainerStarted","Data":"8a30b12465b910d87fff69c347afe0647cf10c0c8893596791161431d91e632b"}
Nov 03 18:48:00.561515 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:00.561480    6059 generic.go:356] "Generic (PLEG): container finished" podID="6f9265bc-0969-4f8b-8f9d-6947d6b78847" containerID="b1f5796476a02f7d962bc79728d947d917025e065a0df67f58c0dbf72148d8b7" exitCode=0
Nov 03 18:48:00.561910 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:00.561551    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pp7tl" event={"ID":"6f9265bc-0969-4f8b-8f9d-6947d6b78847","Type":"ContainerDied","Data":"b1f5796476a02f7d962bc79728d947d917025e065a0df67f58c0dbf72148d8b7"}
Nov 03 18:48:01.504845 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:01.504788    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:01.710209 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:01.710180    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pp7tl"
Nov 03 18:48:01.840003 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:01.839909    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f9265bc-0969-4f8b-8f9d-6947d6b78847-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f9265bc-0969-4f8b-8f9d-6947d6b78847-maas-api-openshift-ca\") pod \"6f9265bc-0969-4f8b-8f9d-6947d6b78847\" (UID: \"6f9265bc-0969-4f8b-8f9d-6947d6b78847\") "
Nov 03 18:48:01.840193 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:01.840053    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f9265bc-0969-4f8b-8f9d-6947d6b78847-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f9265bc-0969-4f8b-8f9d-6947d6b78847-maas-api-xks-ca\") pod \"6f9265bc-0969-4f8b-8f9d-6947d6b78847\" (UID: \"6f9265bc-0969-4f8b-8f9d-6947d6b78847\") "
Nov 03 18:48:01.840193 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:01.840142    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f9265bc-0969-4f8b-8f9d-6947d6b78847-kube-api-access-rl7b2\" (UniqueName: \"kubernetes.io/projected/6f9265bc-0969-4f8b-8f9d-6947d6b78847-kube-api-access-rl7b2\") pod \"6f9265bc-0969-4f8b-8f9d-6947d6b78847\" (UID: \"6f9265bc-0969-4f8b-8f9d-6947d6b78847\") "
Nov 03 18:48:01.840323 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:01.840263    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f9265bc-0969-4f8b-8f9d-6947d6b78847-maas-api-openshift-ca" pod "6f9265bc-0969-4f8b-8f9d-6947d6b78847" (UID: "6f9265bc-0969-4f8b-8f9d-6947d6b78847"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:48:01.840540 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:01.840518    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f9265bc-0969-4f8b-8f9d-6947d6b78847-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:48:01.842551 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:01.842525    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f9265bc-0969-4f8b-8f9d-6947d6b78847-maas-api-xks-ca" pod "6f9265bc-0969-4f8b-8f9d-6947d6b78847" (UID: "6f9265bc-0969-4f8b-8f9d-6947d6b78847"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:48:01.846523 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:01.846492    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9265bc-0969-4f8b-8f9d-6947d6b78847-kube-api-access-rl7b2" pod "6f9265bc-0969-4f8b-8f9d-6947d6b78847" (UID: "6f9265bc-0969-4f8b-8f9d-6947d6b78847"). InnerVolumeSpecName "kube-api-access-rl7b2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:48:01.941406 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:01.941342    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f9265bc-0969-4f8b-8f9d-6947d6b78847-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:48:01.941406 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:01.941400    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl7b2\" (UniqueName: \"kubernetes.io/projected/6f9265bc-0969-4f8b-8f9d-6947d6b78847-kube-api-access-rl7b2\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:48:02.575767 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:02.575729    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pp7tl"
Nov 03 18:48:02.575767 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:02.575750    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pp7tl" event={"ID":"6f9265bc-0969-4f8b-8f9d-6947d6b78847","Type":"ContainerDied","Data":"8a30b12465b910d87fff69c347afe0647cf10c0c8893596791161431d91e632b"}
Nov 03 18:48:02.575989 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:02.575810    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a30b12465b910d87fff69c347afe0647cf10c0c8893596791161431d91e632b"
Nov 03 18:48:06.861713 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:48:06.861668    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:48:17.062543 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:17.062488    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zp9n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:48:17.063176 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:17.063143    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7" podUID="60b66cb6-2bce-4c02-aa7d-7eb25f29f237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:48:17.088628 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:17.088479    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xpshn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:48:17.088628 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:17.088565    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn" podUID="d9e3abf7-c5e1-4414-858e-5f84f6a88670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:48:22.168937 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:22.168889    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-l2xpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:48:22.169703 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:22.169662    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" podUID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:48:22.279720 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:22.278991    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gxxkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:48:22.279720 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:22.279065    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" podUID="2159e743-a61b-4d76-9f78-9be36e825a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:48:23.877196 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:23.877154    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pp7tl"]
Nov 03 18:48:23.924679 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:23.924640    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pp7tl"]
Nov 03 18:48:24.003115 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:24.003081    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkrqc"]
Nov 03 18:48:24.003623 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:24.003603    6059 cpu_manager.go:414] "containerMap: removing container" podUID="6f9265bc-0969-4f8b-8f9d-6947d6b78847" containerName="revoke-keys"
Nov 03 18:48:24.003755 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:24.003624    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f9265bc-0969-4f8b-8f9d-6947d6b78847" containerName="revoke-keys"
Nov 03 18:48:24.003824 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:24.003771    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f9265bc-0969-4f8b-8f9d-6947d6b78847" containerName="revoke-keys"
Nov 03 18:48:24.007158 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:24.007133    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkrqc"
Nov 03 18:48:24.057098 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:24.057065    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:48:24.057312 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:24.057145    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:48:24.057312 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:24.057167    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:48:24.057312 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:24.057270    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:48:24.067409 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:24.067372    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkrqc"]
Nov 03 18:48:24.078625 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:24.078583    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrj2w\" (UniqueName: \"kubernetes.io/projected/0529bba0-8f97-4257-8182-3834249cc7f8-kube-api-access-zrj2w\") pod \"maas-api-revoke-sub-3d4156f1-fkrqc\" (UID: \"0529bba0-8f97-4257-8182-3834249cc7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkrqc"
Nov 03 18:48:24.078817 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:24.078721    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0529bba0-8f97-4257-8182-3834249cc7f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fkrqc\" (UID: \"0529bba0-8f97-4257-8182-3834249cc7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkrqc"
Nov 03 18:48:24.078978 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:24.078951    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0529bba0-8f97-4257-8182-3834249cc7f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fkrqc\" (UID: \"0529bba0-8f97-4257-8182-3834249cc7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkrqc"
Nov 03 18:48:24.180044 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:24.179953    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0529bba0-8f97-4257-8182-3834249cc7f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fkrqc\" (UID: \"0529bba0-8f97-4257-8182-3834249cc7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkrqc"
Nov 03 18:48:24.180044 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:24.180019    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrj2w\" (UniqueName: \"kubernetes.io/projected/0529bba0-8f97-4257-8182-3834249cc7f8-kube-api-access-zrj2w\") pod \"maas-api-revoke-sub-3d4156f1-fkrqc\" (UID: \"0529bba0-8f97-4257-8182-3834249cc7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkrqc"
Nov 03 18:48:24.180247 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:24.180080    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0529bba0-8f97-4257-8182-3834249cc7f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fkrqc\" (UID: \"0529bba0-8f97-4257-8182-3834249cc7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkrqc"
Nov 03 18:48:24.180804 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:24.180780    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0529bba0-8f97-4257-8182-3834249cc7f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fkrqc\" (UID: \"0529bba0-8f97-4257-8182-3834249cc7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkrqc"
Nov 03 18:48:24.182389 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:24.182367    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0529bba0-8f97-4257-8182-3834249cc7f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fkrqc\" (UID: \"0529bba0-8f97-4257-8182-3834249cc7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkrqc"
Nov 03 18:48:24.239192 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:24.239158    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrj2w\" (UniqueName: \"kubernetes.io/projected/0529bba0-8f97-4257-8182-3834249cc7f8-kube-api-access-zrj2w\") pod \"maas-api-revoke-sub-3d4156f1-fkrqc\" (UID: \"0529bba0-8f97-4257-8182-3834249cc7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkrqc"
Nov 03 18:48:24.320513 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:24.320471    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkrqc"
Nov 03 18:48:24.689474 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:24.689448    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkrqc"]
Nov 03 18:48:24.691583 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:48:24.691550    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0529bba0_8f97_4257_8182_3834249cc7f8.slice/crio-af9e1b23142f77b8d8bed74f0c264065a716190a8029575728c5e6f15edd080f WatchSource:0}: Error finding container af9e1b23142f77b8d8bed74f0c264065a716190a8029575728c5e6f15edd080f: Status 404 returned error can't find the container with id af9e1b23142f77b8d8bed74f0c264065a716190a8029575728c5e6f15edd080f
Nov 03 18:48:24.802588 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:24.802549    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkrqc" event={"ID":"0529bba0-8f97-4257-8182-3834249cc7f8","Type":"ContainerStarted","Data":"af9e1b23142f77b8d8bed74f0c264065a716190a8029575728c5e6f15edd080f"}
Nov 03 18:48:25.335836 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:25.335804    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f9265bc-0969-4f8b-8f9d-6947d6b78847" path="/var/lib/kubelet/pods/6f9265bc-0969-4f8b-8f9d-6947d6b78847/volumes"
Nov 03 18:48:25.406984 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:25.406948    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:25.812905 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:25.812873    6059 generic.go:356] "Generic (PLEG): container finished" podID="0529bba0-8f97-4257-8182-3834249cc7f8" containerID="a275bd2ae7a4e940f577ed55c6296fa0ec97d8114b34d30478b053a8d585dc8e" exitCode=0
Nov 03 18:48:25.813103 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:25.812937    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkrqc" event={"ID":"0529bba0-8f97-4257-8182-3834249cc7f8","Type":"ContainerDied","Data":"a275bd2ae7a4e940f577ed55c6296fa0ec97d8114b34d30478b053a8d585dc8e"}
Nov 03 18:48:26.960132 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:26.960102    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkrqc"
Nov 03 18:48:27.012418 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:27.012381    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0529bba0-8f97-4257-8182-3834249cc7f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0529bba0-8f97-4257-8182-3834249cc7f8-maas-api-xks-ca\") pod \"0529bba0-8f97-4257-8182-3834249cc7f8\" (UID: \"0529bba0-8f97-4257-8182-3834249cc7f8\") "
Nov 03 18:48:27.012629 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:27.012487    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0529bba0-8f97-4257-8182-3834249cc7f8-kube-api-access-zrj2w\" (UniqueName: \"kubernetes.io/projected/0529bba0-8f97-4257-8182-3834249cc7f8-kube-api-access-zrj2w\") pod \"0529bba0-8f97-4257-8182-3834249cc7f8\" (UID: \"0529bba0-8f97-4257-8182-3834249cc7f8\") "
Nov 03 18:48:27.012629 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:27.012579    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0529bba0-8f97-4257-8182-3834249cc7f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0529bba0-8f97-4257-8182-3834249cc7f8-maas-api-openshift-ca\") pod \"0529bba0-8f97-4257-8182-3834249cc7f8\" (UID: \"0529bba0-8f97-4257-8182-3834249cc7f8\") "
Nov 03 18:48:27.012943 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:27.012915    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0529bba0-8f97-4257-8182-3834249cc7f8-maas-api-openshift-ca" pod "0529bba0-8f97-4257-8182-3834249cc7f8" (UID: "0529bba0-8f97-4257-8182-3834249cc7f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:48:27.014539 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:27.014513    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0529bba0-8f97-4257-8182-3834249cc7f8-kube-api-access-zrj2w" pod "0529bba0-8f97-4257-8182-3834249cc7f8" (UID: "0529bba0-8f97-4257-8182-3834249cc7f8"). InnerVolumeSpecName "kube-api-access-zrj2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:48:27.014657 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:27.014638    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0529bba0-8f97-4257-8182-3834249cc7f8-maas-api-xks-ca" pod "0529bba0-8f97-4257-8182-3834249cc7f8" (UID: "0529bba0-8f97-4257-8182-3834249cc7f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:48:27.113528 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:27.113440    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrj2w\" (UniqueName: \"kubernetes.io/projected/0529bba0-8f97-4257-8182-3834249cc7f8-kube-api-access-zrj2w\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:48:27.113528 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:27.113472    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0529bba0-8f97-4257-8182-3834249cc7f8-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:48:27.113528 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:27.113482    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0529bba0-8f97-4257-8182-3834249cc7f8-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:48:27.825943 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:27.825911    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkrqc"
Nov 03 18:48:27.825943 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:27.825935    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkrqc" event={"ID":"0529bba0-8f97-4257-8182-3834249cc7f8","Type":"ContainerDied","Data":"af9e1b23142f77b8d8bed74f0c264065a716190a8029575728c5e6f15edd080f"}
Nov 03 18:48:27.826178 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:48:27.825989    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af9e1b23142f77b8d8bed74f0c264065a716190a8029575728c5e6f15edd080f"
Nov 03 18:48:36.860943 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:48:36.860900    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:49:01.864931 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:01.864892    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:03.884351 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:03.884271    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d"]
Nov 03 18:49:03.884972 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:03.884949    6059 cpu_manager.go:414] "containerMap: removing container" podUID="0529bba0-8f97-4257-8182-3834249cc7f8" containerName="revoke-keys"
Nov 03 18:49:03.885055 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:03.884975    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0529bba0-8f97-4257-8182-3834249cc7f8" containerName="revoke-keys"
Nov 03 18:49:03.885121 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:03.885111    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0529bba0-8f97-4257-8182-3834249cc7f8" containerName="revoke-keys"
Nov 03 18:49:03.888936 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:03.888910    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d"
Nov 03 18:49:03.938928 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:03.938894    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:49:03.958979 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:03.958929    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce9d50b-a49c-41e2-9111-33b9faf3c5ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-c5h4d\" (UID: \"5ce9d50b-a49c-41e2-9111-33b9faf3c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d"
Nov 03 18:49:03.959296 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:03.959264    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw4dv\" (UniqueName: \"kubernetes.io/projected/5ce9d50b-a49c-41e2-9111-33b9faf3c5ff-kube-api-access-tw4dv\") pod \"maas-api-revoke-sub-e2fc9c5b-c5h4d\" (UID: \"5ce9d50b-a49c-41e2-9111-33b9faf3c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d"
Nov 03 18:49:03.959465 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:03.959447    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce9d50b-a49c-41e2-9111-33b9faf3c5ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-c5h4d\" (UID: \"5ce9d50b-a49c-41e2-9111-33b9faf3c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d"
Nov 03 18:49:03.986475 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:03.986436    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:49:03.986678 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:03.986477    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:49:03.986678 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:03.986499    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:49:03.988977 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:03.988957    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d"]
Nov 03 18:49:04.060328 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:04.060287    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce9d50b-a49c-41e2-9111-33b9faf3c5ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-c5h4d\" (UID: \"5ce9d50b-a49c-41e2-9111-33b9faf3c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d"
Nov 03 18:49:04.060569 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:04.060375    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce9d50b-a49c-41e2-9111-33b9faf3c5ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-c5h4d\" (UID: \"5ce9d50b-a49c-41e2-9111-33b9faf3c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d"
Nov 03 18:49:04.060569 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:04.060432    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw4dv\" (UniqueName: \"kubernetes.io/projected/5ce9d50b-a49c-41e2-9111-33b9faf3c5ff-kube-api-access-tw4dv\") pod \"maas-api-revoke-sub-e2fc9c5b-c5h4d\" (UID: \"5ce9d50b-a49c-41e2-9111-33b9faf3c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d"
Nov 03 18:49:04.061035 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:04.061007    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce9d50b-a49c-41e2-9111-33b9faf3c5ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-c5h4d\" (UID: \"5ce9d50b-a49c-41e2-9111-33b9faf3c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d"
Nov 03 18:49:04.062937 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:04.062909    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce9d50b-a49c-41e2-9111-33b9faf3c5ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-c5h4d\" (UID: \"5ce9d50b-a49c-41e2-9111-33b9faf3c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d"
Nov 03 18:49:04.119445 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:04.119412    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw4dv\" (UniqueName: \"kubernetes.io/projected/5ce9d50b-a49c-41e2-9111-33b9faf3c5ff-kube-api-access-tw4dv\") pod \"maas-api-revoke-sub-e2fc9c5b-c5h4d\" (UID: \"5ce9d50b-a49c-41e2-9111-33b9faf3c5ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d"
Nov 03 18:49:04.203220 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:04.203126    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d"
Nov 03 18:49:04.574940 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:04.574908    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d"]
Nov 03 18:49:04.578306 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:49:04.578272    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce9d50b_a49c_41e2_9111_33b9faf3c5ff.slice/crio-1560660ad3e3d2f7b6abe15b665ba0e2f3f2d3a7f34b419ac455ad94ee086801 WatchSource:0}: Error finding container 1560660ad3e3d2f7b6abe15b665ba0e2f3f2d3a7f34b419ac455ad94ee086801: Status 404 returned error can't find the container with id 1560660ad3e3d2f7b6abe15b665ba0e2f3f2d3a7f34b419ac455ad94ee086801
Nov 03 18:49:05.133491 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:05.133398    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d" event={"ID":"5ce9d50b-a49c-41e2-9111-33b9faf3c5ff","Type":"ContainerStarted","Data":"26c8ed6d56198395233fa0c78b58499bd268c1bb88a5557b87d3575ef2ed157a"}
Nov 03 18:49:05.133491 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:05.133480    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d" event={"ID":"5ce9d50b-a49c-41e2-9111-33b9faf3c5ff","Type":"ContainerStarted","Data":"1560660ad3e3d2f7b6abe15b665ba0e2f3f2d3a7f34b419ac455ad94ee086801"}
Nov 03 18:49:05.244790 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:05.244731    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d" podStartSLOduration=2.244713333 podStartE2EDuration="2.244713333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:49:05.243656029 +0000 UTC m=+5486.041368811" watchObservedRunningTime="2026-11-03 18:49:05.244713333 +0000 UTC m=+5486.042426114"
Nov 03 18:49:06.140414 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:06.140375    6059 generic.go:356] "Generic (PLEG): container finished" podID="5ce9d50b-a49c-41e2-9111-33b9faf3c5ff" containerID="26c8ed6d56198395233fa0c78b58499bd268c1bb88a5557b87d3575ef2ed157a" exitCode=0
Nov 03 18:49:06.140823 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:06.140461    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d" event={"ID":"5ce9d50b-a49c-41e2-9111-33b9faf3c5ff","Type":"ContainerDied","Data":"26c8ed6d56198395233fa0c78b58499bd268c1bb88a5557b87d3575ef2ed157a"}
Nov 03 18:49:06.861106 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:49:06.861068    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:49:07.283820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:07.283792    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d"
Nov 03 18:49:07.396176 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:07.396081    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce9d50b-a49c-41e2-9111-33b9faf3c5ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce9d50b-a49c-41e2-9111-33b9faf3c5ff-maas-api-openshift-ca\") pod \"5ce9d50b-a49c-41e2-9111-33b9faf3c5ff\" (UID: \"5ce9d50b-a49c-41e2-9111-33b9faf3c5ff\") "
Nov 03 18:49:07.396176 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:07.396152    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ce9d50b-a49c-41e2-9111-33b9faf3c5ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce9d50b-a49c-41e2-9111-33b9faf3c5ff-maas-api-xks-ca\") pod \"5ce9d50b-a49c-41e2-9111-33b9faf3c5ff\" (UID: \"5ce9d50b-a49c-41e2-9111-33b9faf3c5ff\") "
Nov 03 18:49:07.396419 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:07.396204    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce9d50b-a49c-41e2-9111-33b9faf3c5ff-kube-api-access-tw4dv\" (UniqueName: \"kubernetes.io/projected/5ce9d50b-a49c-41e2-9111-33b9faf3c5ff-kube-api-access-tw4dv\") pod \"5ce9d50b-a49c-41e2-9111-33b9faf3c5ff\" (UID: \"5ce9d50b-a49c-41e2-9111-33b9faf3c5ff\") "
Nov 03 18:49:07.396823 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:07.396535    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce9d50b-a49c-41e2-9111-33b9faf3c5ff-maas-api-openshift-ca" pod "5ce9d50b-a49c-41e2-9111-33b9faf3c5ff" (UID: "5ce9d50b-a49c-41e2-9111-33b9faf3c5ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:49:07.398497 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:07.398470    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce9d50b-a49c-41e2-9111-33b9faf3c5ff-kube-api-access-tw4dv" pod "5ce9d50b-a49c-41e2-9111-33b9faf3c5ff" (UID: "5ce9d50b-a49c-41e2-9111-33b9faf3c5ff"). InnerVolumeSpecName "kube-api-access-tw4dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:49:07.398608 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:07.398538    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ce9d50b-a49c-41e2-9111-33b9faf3c5ff-maas-api-xks-ca" pod "5ce9d50b-a49c-41e2-9111-33b9faf3c5ff" (UID: "5ce9d50b-a49c-41e2-9111-33b9faf3c5ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:49:07.497699 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:07.497649    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce9d50b-a49c-41e2-9111-33b9faf3c5ff-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:49:07.497699 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:07.497694    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce9d50b-a49c-41e2-9111-33b9faf3c5ff-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:49:07.497699 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:07.497706    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw4dv\" (UniqueName: \"kubernetes.io/projected/5ce9d50b-a49c-41e2-9111-33b9faf3c5ff-kube-api-access-tw4dv\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:49:08.152529 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:08.152494    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d" event={"ID":"5ce9d50b-a49c-41e2-9111-33b9faf3c5ff","Type":"ContainerDied","Data":"1560660ad3e3d2f7b6abe15b665ba0e2f3f2d3a7f34b419ac455ad94ee086801"}
Nov 03 18:49:08.152529 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:08.152521    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d"
Nov 03 18:49:08.152839 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:08.152545    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1560660ad3e3d2f7b6abe15b665ba0e2f3f2d3a7f34b419ac455ad94ee086801"
Nov 03 18:49:14.698602 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:14.698480    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bsq4t"]
Nov 03 18:49:14.704557 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:14.698913    6059 cpu_manager.go:414] "containerMap: removing container" podUID="5ce9d50b-a49c-41e2-9111-33b9faf3c5ff" containerName="revoke-keys"
Nov 03 18:49:14.704557 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:14.698925    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce9d50b-a49c-41e2-9111-33b9faf3c5ff" containerName="revoke-keys"
Nov 03 18:49:14.704557 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:14.699035    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce9d50b-a49c-41e2-9111-33b9faf3c5ff" containerName="revoke-keys"
Nov 03 18:49:14.704557 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:14.702982    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bsq4t"
Nov 03 18:49:14.777909 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:14.777872    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdpxw\" (UniqueName: \"kubernetes.io/projected/c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9-kube-api-access-sdpxw\") pod \"maas-api-revoke-sub-5a29efb3-bsq4t\" (UID: \"c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bsq4t"
Nov 03 18:49:14.778089 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:14.777918    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-bsq4t\" (UID: \"c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bsq4t"
Nov 03 18:49:14.778089 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:14.778030    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-bsq4t\" (UID: \"c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bsq4t"
Nov 03 18:49:14.800039 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:14.799998    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:49:14.800231 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:14.800048    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:49:14.800231 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:14.800066    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:49:14.800231 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:14.800106    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:49:14.802685 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:14.802662    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bsq4t"]
Nov 03 18:49:14.879636 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:14.879597    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdpxw\" (UniqueName: \"kubernetes.io/projected/c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9-kube-api-access-sdpxw\") pod \"maas-api-revoke-sub-5a29efb3-bsq4t\" (UID: \"c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bsq4t"
Nov 03 18:49:14.879822 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:14.879639    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-bsq4t\" (UID: \"c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bsq4t"
Nov 03 18:49:14.879822 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:14.879699    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-bsq4t\" (UID: \"c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bsq4t"
Nov 03 18:49:14.880593 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:14.880566    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-bsq4t\" (UID: \"c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bsq4t"
Nov 03 18:49:14.882178 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:14.882154    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-bsq4t\" (UID: \"c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bsq4t"
Nov 03 18:49:14.942710 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:14.942670    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdpxw\" (UniqueName: \"kubernetes.io/projected/c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9-kube-api-access-sdpxw\") pod \"maas-api-revoke-sub-5a29efb3-bsq4t\" (UID: \"c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bsq4t"
Nov 03 18:49:15.017105 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:15.017069    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bsq4t"
Nov 03 18:49:15.385074 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:15.385047    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bsq4t"]
Nov 03 18:49:15.387579 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:49:15.387537    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ff47a7_5539_4ebf_94c3_fa8daec1a1b9.slice/crio-48a33aec8ca85ac44da349e13650180524a8ba1778e8af10ec163ac0ee668cca WatchSource:0}: Error finding container 48a33aec8ca85ac44da349e13650180524a8ba1778e8af10ec163ac0ee668cca: Status 404 returned error can't find the container with id 48a33aec8ca85ac44da349e13650180524a8ba1778e8af10ec163ac0ee668cca
Nov 03 18:49:16.201024 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:16.200991    6059 generic.go:356] "Generic (PLEG): container finished" podID="c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9" containerID="684d73f6d6a6a3bbe2e4a10fb36932f370b3aca84c85b2b33e6b189d8f371419" exitCode=0
Nov 03 18:49:16.201469 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:16.201075    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bsq4t" event={"ID":"c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9","Type":"ContainerDied","Data":"684d73f6d6a6a3bbe2e4a10fb36932f370b3aca84c85b2b33e6b189d8f371419"}
Nov 03 18:49:16.201469 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:16.201179    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bsq4t" event={"ID":"c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9","Type":"ContainerStarted","Data":"48a33aec8ca85ac44da349e13650180524a8ba1778e8af10ec163ac0ee668cca"}
Nov 03 18:49:17.419219 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:17.419185    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bsq4t"
Nov 03 18:49:17.510761 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:17.510653    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9-maas-api-openshift-ca\") pod \"c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9\" (UID: \"c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9\") "
Nov 03 18:49:17.510948 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:17.510799    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9-kube-api-access-sdpxw\" (UniqueName: \"kubernetes.io/projected/c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9-kube-api-access-sdpxw\") pod \"c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9\" (UID: \"c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9\") "
Nov 03 18:49:17.510948 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:17.510839    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9-maas-api-xks-ca\") pod \"c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9\" (UID: \"c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9\") "
Nov 03 18:49:17.511835 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:17.511805    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9-maas-api-openshift-ca" pod "c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9" (UID: "c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:49:17.514428 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:17.514382    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9-maas-api-xks-ca" pod "c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9" (UID: "c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:49:17.533381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:17.533028    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9-kube-api-access-sdpxw" pod "c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9" (UID: "c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9"). InnerVolumeSpecName "kube-api-access-sdpxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:49:17.613100 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:17.613003    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:49:17.613100 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:17.613051    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdpxw\" (UniqueName: \"kubernetes.io/projected/c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9-kube-api-access-sdpxw\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:49:17.613100 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:17.613069    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:49:18.047719 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:18.047666    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bsl25"]
Nov 03 18:49:18.094945 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:18.094901    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-bsl25"]
Nov 03 18:49:18.221610 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:18.221579    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bsq4t"
Nov 03 18:49:18.221828 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:18.221605    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bsq4t" event={"ID":"c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9","Type":"ContainerDied","Data":"48a33aec8ca85ac44da349e13650180524a8ba1778e8af10ec163ac0ee668cca"}
Nov 03 18:49:18.221828 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:18.221670    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48a33aec8ca85ac44da349e13650180524a8ba1778e8af10ec163ac0ee668cca"
Nov 03 18:49:19.373381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:19.370296    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7ddbd0f-a039-4be4-b5e9-b68b36a4d287" path="/var/lib/kubelet/pods/c7ddbd0f-a039-4be4-b5e9-b68b36a4d287/volumes"
Nov 03 18:49:31.547279 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:31.547237    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkrqc"]
Nov 03 18:49:31.594585 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:31.594547    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkrqc"]
Nov 03 18:49:31.671290 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:31.671254    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xz9zx"]
Nov 03 18:49:31.671744 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:31.671729    6059 cpu_manager.go:414] "containerMap: removing container" podUID="c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9" containerName="revoke-keys"
Nov 03 18:49:31.671817 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:31.671746    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9" containerName="revoke-keys"
Nov 03 18:49:31.671857 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:31.671822    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9" containerName="revoke-keys"
Nov 03 18:49:31.675131 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:31.675105    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xz9zx"
Nov 03 18:49:31.714687 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:31.714646    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f41601e-94f8-48ef-9868-e9ffdb853b0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xz9zx\" (UID: \"1f41601e-94f8-48ef-9868-e9ffdb853b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xz9zx"
Nov 03 18:49:31.714861 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:31.714793    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f41601e-94f8-48ef-9868-e9ffdb853b0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xz9zx\" (UID: \"1f41601e-94f8-48ef-9868-e9ffdb853b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xz9zx"
Nov 03 18:49:31.714945 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:31.714860    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqw78\" (UniqueName: \"kubernetes.io/projected/1f41601e-94f8-48ef-9868-e9ffdb853b0c-kube-api-access-tqw78\") pod \"maas-api-revoke-sub-3d4156f1-xz9zx\" (UID: \"1f41601e-94f8-48ef-9868-e9ffdb853b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xz9zx"
Nov 03 18:49:31.725218 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:31.725191    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:49:31.725397 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:31.725306    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:49:31.725397 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:31.725306    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:49:31.725397 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:31.725388    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:49:31.735469 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:31.735437    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xz9zx"]
Nov 03 18:49:31.815516 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:31.815431    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f41601e-94f8-48ef-9868-e9ffdb853b0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xz9zx\" (UID: \"1f41601e-94f8-48ef-9868-e9ffdb853b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xz9zx"
Nov 03 18:49:31.815516 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:31.815510    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f41601e-94f8-48ef-9868-e9ffdb853b0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xz9zx\" (UID: \"1f41601e-94f8-48ef-9868-e9ffdb853b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xz9zx"
Nov 03 18:49:31.815729 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:31.815535    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqw78\" (UniqueName: \"kubernetes.io/projected/1f41601e-94f8-48ef-9868-e9ffdb853b0c-kube-api-access-tqw78\") pod \"maas-api-revoke-sub-3d4156f1-xz9zx\" (UID: \"1f41601e-94f8-48ef-9868-e9ffdb853b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xz9zx"
Nov 03 18:49:31.816168 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:31.816144    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f41601e-94f8-48ef-9868-e9ffdb853b0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xz9zx\" (UID: \"1f41601e-94f8-48ef-9868-e9ffdb853b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xz9zx"
Nov 03 18:49:31.817889 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:31.817871    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f41601e-94f8-48ef-9868-e9ffdb853b0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xz9zx\" (UID: \"1f41601e-94f8-48ef-9868-e9ffdb853b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xz9zx"
Nov 03 18:49:31.874279 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:31.874247    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqw78\" (UniqueName: \"kubernetes.io/projected/1f41601e-94f8-48ef-9868-e9ffdb853b0c-kube-api-access-tqw78\") pod \"maas-api-revoke-sub-3d4156f1-xz9zx\" (UID: \"1f41601e-94f8-48ef-9868-e9ffdb853b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xz9zx"
Nov 03 18:49:31.988630 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:31.988591    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xz9zx"
Nov 03 18:49:32.365080 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:32.365057    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xz9zx"]
Nov 03 18:49:32.367059 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:49:32.367026    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f41601e_94f8_48ef_9868_e9ffdb853b0c.slice/crio-1a29e0a6ba6fa53c61c4cb639a3d9c43dd52bac2490ec335ec2da777f7f906f2 WatchSource:0}: Error finding container 1a29e0a6ba6fa53c61c4cb639a3d9c43dd52bac2490ec335ec2da777f7f906f2: Status 404 returned error can't find the container with id 1a29e0a6ba6fa53c61c4cb639a3d9c43dd52bac2490ec335ec2da777f7f906f2
Nov 03 18:49:33.330985 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:33.330942    6059 generic.go:356] "Generic (PLEG): container finished" podID="1f41601e-94f8-48ef-9868-e9ffdb853b0c" containerID="67d8e3301db178dfe350156b3fbb049adcc187f30ef1fc27d6815b05882a1cc8" exitCode=0
Nov 03 18:49:33.335947 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:33.335921    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0529bba0-8f97-4257-8182-3834249cc7f8" path="/var/lib/kubelet/pods/0529bba0-8f97-4257-8182-3834249cc7f8/volumes"
Nov 03 18:49:33.336232 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:33.336212    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xz9zx" event={"ID":"1f41601e-94f8-48ef-9868-e9ffdb853b0c","Type":"ContainerDied","Data":"67d8e3301db178dfe350156b3fbb049adcc187f30ef1fc27d6815b05882a1cc8"}
Nov 03 18:49:33.336273 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:33.336257    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xz9zx" event={"ID":"1f41601e-94f8-48ef-9868-e9ffdb853b0c","Type":"ContainerStarted","Data":"1a29e0a6ba6fa53c61c4cb639a3d9c43dd52bac2490ec335ec2da777f7f906f2"}
Nov 03 18:49:34.483260 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:34.483233    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xz9zx"
Nov 03 18:49:34.542680 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:34.542635    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f41601e-94f8-48ef-9868-e9ffdb853b0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f41601e-94f8-48ef-9868-e9ffdb853b0c-maas-api-xks-ca\") pod \"1f41601e-94f8-48ef-9868-e9ffdb853b0c\" (UID: \"1f41601e-94f8-48ef-9868-e9ffdb853b0c\") "
Nov 03 18:49:34.542893 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:34.542696    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f41601e-94f8-48ef-9868-e9ffdb853b0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f41601e-94f8-48ef-9868-e9ffdb853b0c-maas-api-openshift-ca\") pod \"1f41601e-94f8-48ef-9868-e9ffdb853b0c\" (UID: \"1f41601e-94f8-48ef-9868-e9ffdb853b0c\") "
Nov 03 18:49:34.542893 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:34.542718    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f41601e-94f8-48ef-9868-e9ffdb853b0c-kube-api-access-tqw78\" (UniqueName: \"kubernetes.io/projected/1f41601e-94f8-48ef-9868-e9ffdb853b0c-kube-api-access-tqw78\") pod \"1f41601e-94f8-48ef-9868-e9ffdb853b0c\" (UID: \"1f41601e-94f8-48ef-9868-e9ffdb853b0c\") "
Nov 03 18:49:34.543122 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:34.543093    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f41601e-94f8-48ef-9868-e9ffdb853b0c-maas-api-openshift-ca" pod "1f41601e-94f8-48ef-9868-e9ffdb853b0c" (UID: "1f41601e-94f8-48ef-9868-e9ffdb853b0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:49:34.544924 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:34.544892    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f41601e-94f8-48ef-9868-e9ffdb853b0c-maas-api-xks-ca" pod "1f41601e-94f8-48ef-9868-e9ffdb853b0c" (UID: "1f41601e-94f8-48ef-9868-e9ffdb853b0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:49:34.545043 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:34.544983    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f41601e-94f8-48ef-9868-e9ffdb853b0c-kube-api-access-tqw78" pod "1f41601e-94f8-48ef-9868-e9ffdb853b0c" (UID: "1f41601e-94f8-48ef-9868-e9ffdb853b0c"). InnerVolumeSpecName "kube-api-access-tqw78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:49:34.644147 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:34.644049    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f41601e-94f8-48ef-9868-e9ffdb853b0c-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:49:34.644147 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:34.644085    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f41601e-94f8-48ef-9868-e9ffdb853b0c-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:49:34.644147 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:34.644096    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqw78\" (UniqueName: \"kubernetes.io/projected/1f41601e-94f8-48ef-9868-e9ffdb853b0c-kube-api-access-tqw78\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:49:35.345087 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:35.345058    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xz9zx"
Nov 03 18:49:35.345250 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:35.345055    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xz9zx" event={"ID":"1f41601e-94f8-48ef-9868-e9ffdb853b0c","Type":"ContainerDied","Data":"1a29e0a6ba6fa53c61c4cb639a3d9c43dd52bac2490ec335ec2da777f7f906f2"}
Nov 03 18:49:35.345250 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:35.345179    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a29e0a6ba6fa53c61c4cb639a3d9c43dd52bac2490ec335ec2da777f7f906f2"
Nov 03 18:49:36.862331 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:49:36.862287    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:49:47.820078 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:47.820038    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:48.329054 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.326529    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"]
Nov 03 18:49:48.329054 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.327267    6059 cpu_manager.go:414] "containerMap: removing container" podUID="1f41601e-94f8-48ef-9868-e9ffdb853b0c" containerName="revoke-keys"
Nov 03 18:49:48.329054 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.327288    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f41601e-94f8-48ef-9868-e9ffdb853b0c" containerName="revoke-keys"
Nov 03 18:49:48.329054 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.327416    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f41601e-94f8-48ef-9868-e9ffdb853b0c" containerName="revoke-keys"
Nov 03 18:49:48.338935 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.338824    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.392416 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.390768    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-27a21b-openshift-default-dockercfg-hplwr\""
Nov 03 18:49:48.456382 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.453004    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-workload-socket\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.456723 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.456690    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907144cd-9c12-45d8-a0c8-5529607d119e-istio-podinfo\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.460381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.456887    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-workload-certs\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.460649 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.460609    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh96s\" (UniqueName: \"kubernetes.io/projected/907144cd-9c12-45d8-a0c8-5529607d119e-kube-api-access-nh96s\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.460869 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.460849    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-istio-envoy\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.461084 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.461064    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-istio-data\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.464468 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.464434    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907144cd-9c12-45d8-a0c8-5529607d119e-istiod-ca-cert\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.464744 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.464719    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-credential-socket\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.489506 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.489460    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/907144cd-9c12-45d8-a0c8-5529607d119e-istio-token\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.489888 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.459427    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"]
Nov 03 18:49:48.544417 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.540073    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:48.604865 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.604766    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-istio-envoy\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.624395 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.619624    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-istio-data\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.624395 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.619711    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907144cd-9c12-45d8-a0c8-5529607d119e-istiod-ca-cert\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.624395 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.619836    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-credential-socket\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.624395 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.619868    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/907144cd-9c12-45d8-a0c8-5529607d119e-istio-token\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.624395 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.619961    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-workload-socket\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.624395 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.619994    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907144cd-9c12-45d8-a0c8-5529607d119e-istio-podinfo\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.624395 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.620051    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-workload-certs\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.624395 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.620078    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh96s\" (UniqueName: \"kubernetes.io/projected/907144cd-9c12-45d8-a0c8-5529607d119e-kube-api-access-nh96s\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.624395 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.621197    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907144cd-9c12-45d8-a0c8-5529607d119e-istiod-ca-cert\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.624395 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.621481    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-credential-socket\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.624395 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.621981    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-workload-socket\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.624395 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.622721    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-workload-certs\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.666232 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.666141    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-istio-envoy\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.666493 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.666450    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-istio-data\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.686393 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.672958    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907144cd-9c12-45d8-a0c8-5529607d119e-istio-podinfo\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.725679 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.725583    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:48.726050 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.725643    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh96s\" (UniqueName: \"kubernetes.io/projected/907144cd-9c12-45d8-a0c8-5529607d119e-kube-api-access-nh96s\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:48.750381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:48.728339    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/907144cd-9c12-45d8-a0c8-5529607d119e-istio-token\") pod \"e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") " pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:49.005907 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:49.005576    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:49.776914 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:49.776886    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"]
Nov 03 18:49:49.778640 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:49:49.778608    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907144cd_9c12_45d8_a0c8_5529607d119e.slice/crio-e3ef705f1424ed21b19184b40ced19b1f028ca34338b97e90e973c5c5c528a95 WatchSource:0}: Error finding container e3ef705f1424ed21b19184b40ced19b1f028ca34338b97e90e973c5c5c528a95: Status 404 returned error can't find the container with id e3ef705f1424ed21b19184b40ced19b1f028ca34338b97e90e973c5c5c528a95
Nov 03 18:49:49.781184 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:49.781155    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:49:49.781258 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:49.781225    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:49:49.781298 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:49.781255    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:49:50.645177 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:50.645142    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b" event={"ID":"907144cd-9c12-45d8-a0c8-5529607d119e","Type":"ContainerStarted","Data":"6e6348351ede0e49367edce747a41cbf99f13dc7144839c7c5a26b9dc4f818b9"}
Nov 03 18:49:50.645600 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:50.645192    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b" event={"ID":"907144cd-9c12-45d8-a0c8-5529607d119e","Type":"ContainerStarted","Data":"e3ef705f1424ed21b19184b40ced19b1f028ca34338b97e90e973c5c5c528a95"}
Nov 03 18:49:50.809914 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:50.809860    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b" podStartSLOduration=2.809846739 podStartE2EDuration="2.809846739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:49:50.808013115 +0000 UTC m=+5531.605725897" watchObservedRunningTime="2026-11-03 18:49:50.809846739 +0000 UTC m=+5531.607559519"
Nov 03 18:49:51.012403 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:51.012345    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:51.018069 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:51.018040    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:51.651131 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:51.651098    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:51.652295 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:51.652272    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:49:53.817057 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:53.817011    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:53.992758 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:53.992722    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:54.053540 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:54.053498    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc3b6eae-dngd5"]
Nov 03 18:49:54.100916 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:54.100828    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc3b6eae-dngd5"]
Nov 03 18:49:54.177790 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:54.177760    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:55.335621 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:55.335588    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c3ccc35-75e7-4132-b9ee-c861e695778c" path="/var/lib/kubelet/pods/9c3ccc35-75e7-4132-b9ee-c861e695778c/volumes"
Nov 03 18:49:57.023184 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.023145    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:57.205425 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.205382    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:57.392886 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.392795    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2"]
Nov 03 18:49:57.396885 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.396858    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2"
Nov 03 18:49:57.494043 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.493996    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-27a21b\""
Nov 03 18:49:57.494043 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.494006    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-b42bd92a-dockercfg-4ncnz\""
Nov 03 18:49:57.496627 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.496594    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2"]
Nov 03 18:49:57.505908 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.505875    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d222j\" (UniqueName: \"kubernetes.io/projected/7297a2df-851c-4656-98cc-197ec88efe7b-kube-api-access-d222j\") pod \"payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2\" (UID: \"7297a2df-851c-4656-98cc-197ec88efe7b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2"
Nov 03 18:49:57.506084 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.506059    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7297a2df-851c-4656-98cc-197ec88efe7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2\" (UID: \"7297a2df-851c-4656-98cc-197ec88efe7b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2"
Nov 03 18:49:57.555597 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.555562    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v"]
Nov 03 18:49:57.559946 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.559902    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v"
Nov 03 18:49:57.606790 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.606743    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v\" (UID: \"b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v"
Nov 03 18:49:57.606960 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.606904    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d222j\" (UniqueName: \"kubernetes.io/projected/7297a2df-851c-4656-98cc-197ec88efe7b-kube-api-access-d222j\") pod \"payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2\" (UID: \"7297a2df-851c-4656-98cc-197ec88efe7b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2"
Nov 03 18:49:57.606960 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.606942    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mnxm\" (UniqueName: \"kubernetes.io/projected/b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397-kube-api-access-6mnxm\") pod \"payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v\" (UID: \"b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v"
Nov 03 18:49:57.607077 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.606983    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7297a2df-851c-4656-98cc-197ec88efe7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2\" (UID: \"7297a2df-851c-4656-98cc-197ec88efe7b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2"
Nov 03 18:49:57.607703 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.607683    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7297a2df-851c-4656-98cc-197ec88efe7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2\" (UID: \"7297a2df-851c-4656-98cc-197ec88efe7b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2"
Nov 03 18:49:57.615876 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.615850    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v"]
Nov 03 18:49:57.666464 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.666388    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d222j\" (UniqueName: \"kubernetes.io/projected/7297a2df-851c-4656-98cc-197ec88efe7b-kube-api-access-d222j\") pod \"payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2\" (UID: \"7297a2df-851c-4656-98cc-197ec88efe7b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2"
Nov 03 18:49:57.707809 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.707768    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v\" (UID: \"b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v"
Nov 03 18:49:57.707996 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.707849    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mnxm\" (UniqueName: \"kubernetes.io/projected/b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397-kube-api-access-6mnxm\") pod \"payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v\" (UID: \"b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v"
Nov 03 18:49:57.708652 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.708629    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v\" (UID: \"b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v"
Nov 03 18:49:57.709849 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.709829    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2"
Nov 03 18:49:57.782606 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.768425    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mnxm\" (UniqueName: \"kubernetes.io/projected/b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397-kube-api-access-6mnxm\") pod \"payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v\" (UID: \"b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v"
Nov 03 18:49:57.872974 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:57.872932    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v"
Nov 03 18:49:58.080963 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:58.080934    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2"]
Nov 03 18:49:58.082783 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:49:58.082748    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7297a2df_851c_4656_98cc_197ec88efe7b.slice/crio-02db33e4f40eece9c14a789432cf3d776247c1c4e91ae95d28544cf8bca00248 WatchSource:0}: Error finding container 02db33e4f40eece9c14a789432cf3d776247c1c4e91ae95d28544cf8bca00248: Status 404 returned error can't find the container with id 02db33e4f40eece9c14a789432cf3d776247c1c4e91ae95d28544cf8bca00248
Nov 03 18:49:58.243919 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:58.243890    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v"]
Nov 03 18:49:58.245828 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:49:58.245788    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ea1469_8ca5_4a6f_90c9_dc3cb09e9397.slice/crio-671a7f25a58e4878db90e68a9b6bb86428b841de439bfec07ce1081458a32646 WatchSource:0}: Error finding container 671a7f25a58e4878db90e68a9b6bb86428b841de439bfec07ce1081458a32646: Status 404 returned error can't find the container with id 671a7f25a58e4878db90e68a9b6bb86428b841de439bfec07ce1081458a32646
Nov 03 18:49:58.694033 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:58.693942    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2" event={"ID":"7297a2df-851c-4656-98cc-197ec88efe7b","Type":"ContainerStarted","Data":"344b4d8402a07ec67893c4b8c793e1f834ae2110ed38f1eab3449d37d646354b"}
Nov 03 18:49:58.694033 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:58.694016    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2"
Nov 03 18:49:58.694256 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:58.694046    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2" event={"ID":"7297a2df-851c-4656-98cc-197ec88efe7b","Type":"ContainerStarted","Data":"02db33e4f40eece9c14a789432cf3d776247c1c4e91ae95d28544cf8bca00248"}
Nov 03 18:49:58.695653 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:58.695628    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v" event={"ID":"b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397","Type":"ContainerStarted","Data":"a27f5faeecbbc26b7f601c6abdbab37091cf3498da376dbce7c9022f338eaf04"}
Nov 03 18:49:58.695770 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:58.695673    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v" event={"ID":"b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397","Type":"ContainerStarted","Data":"671a7f25a58e4878db90e68a9b6bb86428b841de439bfec07ce1081458a32646"}
Nov 03 18:49:58.695770 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:58.695693    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v"
Nov 03 18:49:58.805988 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:58.805928    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2" podStartSLOduration=1.8059105629999999 podStartE2EDuration="1.805910563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:49:58.804447585 +0000 UTC m=+5539.602160366" watchObservedRunningTime="2026-11-03 18:49:58.805910563 +0000 UTC m=+5539.603623344"
Nov 03 18:49:58.915725 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:49:58.915670    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v" podStartSLOduration=1.9156523330000002 podStartE2EDuration="1.915652333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:49:58.91529109 +0000 UTC m=+5539.713003874" watchObservedRunningTime="2026-11-03 18:49:58.915652333 +0000 UTC m=+5539.713365118"
Nov 03 18:50:00.247195 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:00.247156    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:00.469109 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:00.469063    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:02.218534 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:02.218491    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:02.400899 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:02.400861    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:05.280456 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:05.280415    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:06.881901 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:50:06.881863    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:50:09.705742 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:09.705709    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v"
Nov 03 18:50:09.706099 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:09.705779    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2"
Nov 03 18:50:10.337959 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:10.337921    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:15.283493 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:15.283453    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:15.814628 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:15.814592    6059 generic.go:356] "Generic (PLEG): container finished" podID="907144cd-9c12-45d8-a0c8-5529607d119e" containerID="6e6348351ede0e49367edce747a41cbf99f13dc7144839c7c5a26b9dc4f818b9" exitCode=0
Nov 03 18:50:15.814820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:15.814641    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b" event={"ID":"907144cd-9c12-45d8-a0c8-5529607d119e","Type":"ContainerDied","Data":"6e6348351ede0e49367edce747a41cbf99f13dc7144839c7c5a26b9dc4f818b9"}
Nov 03 18:50:15.815064 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:15.815049    6059 scope.go:122] "RemoveContainer" containerID="6e6348351ede0e49367edce747a41cbf99f13dc7144839c7c5a26b9dc4f818b9"
Nov 03 18:50:15.816309 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:15.816284    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:50:15.816388 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:15.816348    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:50:15.816431 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:15.816422    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:50:16.822764 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:16.822721    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b" event={"ID":"907144cd-9c12-45d8-a0c8-5529607d119e","Type":"ContainerStarted","Data":"d2bbb5f4a4899f2fc509073dbc8b2751adb4da2127734e9f0d55eabe34a5d518"}
Nov 03 18:50:17.012327 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:17.012286    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:50:17.012575 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:17.012554    6059 patch_prober.go:29] interesting pod/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Nov 03 18:50:17.012624 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:17.012600    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b" podUID="907144cd-9c12-45d8-a0c8-5529607d119e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Nov 03 18:50:18.012748 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:18.012710    6059 patch_prober.go:29] interesting pod/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Nov 03 18:50:18.013150 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:18.012773    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b" podUID="907144cd-9c12-45d8-a0c8-5529607d119e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Nov 03 18:50:18.574554 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:18.574513    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:19.012076 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:19.012043    6059 patch_prober.go:29] interesting pod/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Nov 03 18:50:19.012298 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:19.012095    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b" podUID="907144cd-9c12-45d8-a0c8-5529607d119e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Nov 03 18:50:19.125008 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:19.124971    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:19.397508 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:19.397400    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:20.012168 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:20.012138    6059 patch_prober.go:29] interesting pod/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Nov 03 18:50:20.012384 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:20.012197    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b" podUID="907144cd-9c12-45d8-a0c8-5529607d119e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Nov 03 18:50:20.751381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:20.748411    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:21.011946 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:21.011858    6059 patch_prober.go:29] interesting pod/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Nov 03 18:50:21.011946 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:21.011911    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b" podUID="907144cd-9c12-45d8-a0c8-5529607d119e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Nov 03 18:50:21.653382 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:21.653205    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:50:22.016283 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:22.016247    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:50:22.017447 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:22.017414    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:50:23.755539 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:23.755505    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:25.701182 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:25.701149    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:30.599797 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:30.599745    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:31.046219 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:31.046184    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g"]
Nov 03 18:50:31.093233 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:31.093194    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-phs9g"]
Nov 03 18:50:31.335408 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:31.335312    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d17028ef-cc8f-4851-8bb3-9f223966c36c" path="/var/lib/kubelet/pods/d17028ef-cc8f-4851-8bb3-9f223966c36c/volumes"
Nov 03 18:50:35.498566 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:35.498527    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:36.861812 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:50:36.861775    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:50:40.494493 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:40.494458    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:44.084804 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:44.084765    6059 scope.go:122] "RemoveContainer" containerID="0e9d2612e04d42ecc0054486534a56149b0f5155ff947bb178a18087cbbead21"
Nov 03 18:50:45.500928 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:45.500895    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:50.482943 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:50.482910    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:55.400434 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:50:55.400398    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:51:00.291369 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:00.291324    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:51:02.562672 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:02.562635    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:51:05.203474 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:05.203433    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:51:06.744249 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:06.744216    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls"]
Nov 03 18:51:06.748918 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:06.748894    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls"
Nov 03 18:51:06.798787 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:06.798746    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:51:06.798787 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:06.798764    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4c7s\" (UniqueName: \"kubernetes.io/projected/7e799c8c-1dea-418d-baa4-73c655cfaa00-kube-api-access-t4c7s\") pod \"maas-api-revoke-sub-013a378c-g9hls\" (UID: \"7e799c8c-1dea-418d-baa4-73c655cfaa00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls"
Nov 03 18:51:06.799061 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:06.798942    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e799c8c-1dea-418d-baa4-73c655cfaa00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-013a378c-g9hls\" (UID: \"7e799c8c-1dea-418d-baa4-73c655cfaa00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls"
Nov 03 18:51:06.799061 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:06.798993    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e799c8c-1dea-418d-baa4-73c655cfaa00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-013a378c-g9hls\" (UID: \"7e799c8c-1dea-418d-baa4-73c655cfaa00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls"
Nov 03 18:51:06.846232 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:06.846197    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:51:06.846232 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:06.846201    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:51:06.846232 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:06.846241    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:51:06.849810 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:06.849781    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls"]
Nov 03 18:51:06.862183 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:51:06.862147    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:51:06.900574 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:06.900539    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e799c8c-1dea-418d-baa4-73c655cfaa00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-013a378c-g9hls\" (UID: \"7e799c8c-1dea-418d-baa4-73c655cfaa00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls"
Nov 03 18:51:06.900798 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:06.900603    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4c7s\" (UniqueName: \"kubernetes.io/projected/7e799c8c-1dea-418d-baa4-73c655cfaa00-kube-api-access-t4c7s\") pod \"maas-api-revoke-sub-013a378c-g9hls\" (UID: \"7e799c8c-1dea-418d-baa4-73c655cfaa00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls"
Nov 03 18:51:06.900798 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:06.900687    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e799c8c-1dea-418d-baa4-73c655cfaa00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-013a378c-g9hls\" (UID: \"7e799c8c-1dea-418d-baa4-73c655cfaa00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls"
Nov 03 18:51:06.901260 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:06.901236    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e799c8c-1dea-418d-baa4-73c655cfaa00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-013a378c-g9hls\" (UID: \"7e799c8c-1dea-418d-baa4-73c655cfaa00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls"
Nov 03 18:51:06.903001 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:06.902978    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e799c8c-1dea-418d-baa4-73c655cfaa00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-013a378c-g9hls\" (UID: \"7e799c8c-1dea-418d-baa4-73c655cfaa00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls"
Nov 03 18:51:06.959852 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:06.959815    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4c7s\" (UniqueName: \"kubernetes.io/projected/7e799c8c-1dea-418d-baa4-73c655cfaa00-kube-api-access-t4c7s\") pod \"maas-api-revoke-sub-013a378c-g9hls\" (UID: \"7e799c8c-1dea-418d-baa4-73c655cfaa00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls"
Nov 03 18:51:07.046820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:07.046680    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7msdc"]
Nov 03 18:51:07.049292 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:07.049261    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7msdc"]
Nov 03 18:51:07.061990 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:07.061955    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls"
Nov 03 18:51:07.335653 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:07.335563    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f68896ba-8099-4ae5-a76d-262bd32fc504" path="/var/lib/kubelet/pods/f68896ba-8099-4ae5-a76d-262bd32fc504/volumes"
Nov 03 18:51:07.439145 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:07.439111    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls"]
Nov 03 18:51:07.441823 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:51:07.441794    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e799c8c_1dea_418d_baa4_73c655cfaa00.slice/crio-c1e649a6617c586219c58b126f4c056637e757d9957344482fbbaf2ab31c60cc WatchSource:0}: Error finding container c1e649a6617c586219c58b126f4c056637e757d9957344482fbbaf2ab31c60cc: Status 404 returned error can't find the container with id c1e649a6617c586219c58b126f4c056637e757d9957344482fbbaf2ab31c60cc
Nov 03 18:51:08.164872 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:08.164834    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls" event={"ID":"7e799c8c-1dea-418d-baa4-73c655cfaa00","Type":"ContainerStarted","Data":"873771e53fc11890ab934110ea6a1d89209fc08b538bd43756f251525db744d7"}
Nov 03 18:51:08.165285 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:08.164892    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls" event={"ID":"7e799c8c-1dea-418d-baa4-73c655cfaa00","Type":"ContainerStarted","Data":"c1e649a6617c586219c58b126f4c056637e757d9957344482fbbaf2ab31c60cc"}
Nov 03 18:51:08.276160 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:08.276105    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls" podStartSLOduration=2.276088574 podStartE2EDuration="2.276088574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:08.275691963 +0000 UTC m=+5609.073404745" watchObservedRunningTime="2026-11-03 18:51:08.276088574 +0000 UTC m=+5609.073801352"
Nov 03 18:51:09.172270 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:09.172231    6059 generic.go:356] "Generic (PLEG): container finished" podID="7e799c8c-1dea-418d-baa4-73c655cfaa00" containerID="873771e53fc11890ab934110ea6a1d89209fc08b538bd43756f251525db744d7" exitCode=0
Nov 03 18:51:09.172702 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:09.172292    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls" event={"ID":"7e799c8c-1dea-418d-baa4-73c655cfaa00","Type":"ContainerDied","Data":"873771e53fc11890ab934110ea6a1d89209fc08b538bd43756f251525db744d7"}
Nov 03 18:51:10.096535 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:10.096499    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:51:10.329232 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:10.329202    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls"
Nov 03 18:51:10.438677 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:10.438582    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e799c8c-1dea-418d-baa4-73c655cfaa00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e799c8c-1dea-418d-baa4-73c655cfaa00-maas-api-openshift-ca\") pod \"7e799c8c-1dea-418d-baa4-73c655cfaa00\" (UID: \"7e799c8c-1dea-418d-baa4-73c655cfaa00\") "
Nov 03 18:51:10.438677 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:10.438661    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e799c8c-1dea-418d-baa4-73c655cfaa00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e799c8c-1dea-418d-baa4-73c655cfaa00-maas-api-xks-ca\") pod \"7e799c8c-1dea-418d-baa4-73c655cfaa00\" (UID: \"7e799c8c-1dea-418d-baa4-73c655cfaa00\") "
Nov 03 18:51:10.438922 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:10.438770    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e799c8c-1dea-418d-baa4-73c655cfaa00-kube-api-access-t4c7s\" (UniqueName: \"kubernetes.io/projected/7e799c8c-1dea-418d-baa4-73c655cfaa00-kube-api-access-t4c7s\") pod \"7e799c8c-1dea-418d-baa4-73c655cfaa00\" (UID: \"7e799c8c-1dea-418d-baa4-73c655cfaa00\") "
Nov 03 18:51:10.439018 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:10.438974    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e799c8c-1dea-418d-baa4-73c655cfaa00-maas-api-openshift-ca" pod "7e799c8c-1dea-418d-baa4-73c655cfaa00" (UID: "7e799c8c-1dea-418d-baa4-73c655cfaa00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:51:10.439268 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:10.439246    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e799c8c-1dea-418d-baa4-73c655cfaa00-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:51:10.440844 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:10.440814    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e799c8c-1dea-418d-baa4-73c655cfaa00-maas-api-xks-ca" pod "7e799c8c-1dea-418d-baa4-73c655cfaa00" (UID: "7e799c8c-1dea-418d-baa4-73c655cfaa00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:51:10.440954 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:10.440869    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e799c8c-1dea-418d-baa4-73c655cfaa00-kube-api-access-t4c7s" pod "7e799c8c-1dea-418d-baa4-73c655cfaa00" (UID: "7e799c8c-1dea-418d-baa4-73c655cfaa00"). InnerVolumeSpecName "kube-api-access-t4c7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:51:10.540223 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:10.540179    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e799c8c-1dea-418d-baa4-73c655cfaa00-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:51:10.540223 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:10.540215    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4c7s\" (UniqueName: \"kubernetes.io/projected/7e799c8c-1dea-418d-baa4-73c655cfaa00-kube-api-access-t4c7s\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:51:11.193816 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:11.193780    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls"
Nov 03 18:51:11.194000 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:11.193816    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls" event={"ID":"7e799c8c-1dea-418d-baa4-73c655cfaa00","Type":"ContainerDied","Data":"c1e649a6617c586219c58b126f4c056637e757d9957344482fbbaf2ab31c60cc"}
Nov 03 18:51:11.194000 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:11.193900    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1e649a6617c586219c58b126f4c056637e757d9957344482fbbaf2ab31c60cc"
Nov 03 18:51:14.995936 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:14.995893    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:51:18.045209 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:18.045173    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq"]
Nov 03 18:51:18.092485 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:18.092446    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bjvmq"]
Nov 03 18:51:18.477608 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:18.477557    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:51:18.943199 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:18.943099    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:51:19.335318 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:19.335280    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a7a8ac1-5141-4f40-b054-59dac8b00a28" path="/var/lib/kubelet/pods/6a7a8ac1-5141-4f40-b054-59dac8b00a28/volumes"
Nov 03 18:51:19.910097 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:19.910058    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:51:23.520263 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:23.520218    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:51:24.890715 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:24.890681    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:51:25.067734 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:25.067683    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:51:25.515179 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:25.515144    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v"]
Nov 03 18:51:25.691684 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:25.691637    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2"]
Nov 03 18:51:25.977117 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:25.977085    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:51:26.159419 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:26.159385    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:51:29.518677 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:29.518628    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:51:30.515785 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:30.515728    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v" podUID="b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397" containerName="payload-processing" containerID="cri-o://a27f5faeecbbc26b7f601c6abdbab37091cf3498da376dbce7c9022f338eaf04" gracePeriod=25
Nov 03 18:51:30.692450 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:30.692384    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2" podUID="7297a2df-851c-4656-98cc-197ec88efe7b" containerName="payload-pre-processing" containerID="cri-o://344b4d8402a07ec67893c4b8c793e1f834ae2110ed38f1eab3449d37d646354b" gracePeriod=25
Nov 03 18:51:30.815790 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:30.815689    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:51:36.538261 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:36.538225    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:51:36.716020 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:36.715985    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:51:36.861738 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:51:36.861634    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:51:39.703439 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:39.703402    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2" podUID="7297a2df-851c-4656-98cc-197ec88efe7b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:51:39.703870 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:39.703402    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v" podUID="b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:51:44.119375 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:44.119324    6059 scope.go:122] "RemoveContainer" containerID="178f9e12cc1cff5a7426c2fbc3d1bc6d49463752c06d28a7d7e37571710592e0"
Nov 03 18:51:44.128994 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:44.128972    6059 scope.go:122] "RemoveContainer" containerID="095698a424fcc4205b888ede777529dedf6b9db05fedbb37eda4e0b28ca45683"
Nov 03 18:51:49.703638 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:49.703592    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2" podUID="7297a2df-851c-4656-98cc-197ec88efe7b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:51:49.704041 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:49.703592    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v" podUID="b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:51:50.821787 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:50.821760    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v"
Nov 03 18:51:50.897182 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:50.897141    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397-plugins-config-volume\") pod \"b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397\" (UID: \"b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397\") "
Nov 03 18:51:50.897406 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:50.897194    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397-kube-api-access-6mnxm\" (UniqueName: \"kubernetes.io/projected/b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397-kube-api-access-6mnxm\") pod \"b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397\" (UID: \"b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397\") "
Nov 03 18:51:50.897551 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:50.897528    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397-plugins-config-volume" pod "b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397" (UID: "b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:51:50.897768 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:50.897752    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:51:50.899298 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:50.899272    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397-kube-api-access-6mnxm" pod "b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397" (UID: "b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397"). InnerVolumeSpecName "kube-api-access-6mnxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:51:50.938233 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:50.938204    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2"
Nov 03 18:51:50.998781 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:50.998742    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mnxm\" (UniqueName: \"kubernetes.io/projected/b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397-kube-api-access-6mnxm\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:51:51.100148 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.100051    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7297a2df-851c-4656-98cc-197ec88efe7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7297a2df-851c-4656-98cc-197ec88efe7b-plugins-config-volume\") pod \"7297a2df-851c-4656-98cc-197ec88efe7b\" (UID: \"7297a2df-851c-4656-98cc-197ec88efe7b\") "
Nov 03 18:51:51.100335 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.100148    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7297a2df-851c-4656-98cc-197ec88efe7b-kube-api-access-d222j\" (UniqueName: \"kubernetes.io/projected/7297a2df-851c-4656-98cc-197ec88efe7b-kube-api-access-d222j\") pod \"7297a2df-851c-4656-98cc-197ec88efe7b\" (UID: \"7297a2df-851c-4656-98cc-197ec88efe7b\") "
Nov 03 18:51:51.100562 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.100536    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7297a2df-851c-4656-98cc-197ec88efe7b-plugins-config-volume" pod "7297a2df-851c-4656-98cc-197ec88efe7b" (UID: "7297a2df-851c-4656-98cc-197ec88efe7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:51:51.100694 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.100678    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7297a2df-851c-4656-98cc-197ec88efe7b-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:51:51.102337 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.102315    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7297a2df-851c-4656-98cc-197ec88efe7b-kube-api-access-d222j" pod "7297a2df-851c-4656-98cc-197ec88efe7b" (UID: "7297a2df-851c-4656-98cc-197ec88efe7b"). InnerVolumeSpecName "kube-api-access-d222j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:51:51.202241 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.202201    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d222j\" (UniqueName: \"kubernetes.io/projected/7297a2df-851c-4656-98cc-197ec88efe7b-kube-api-access-d222j\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:51:51.441268 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.441163    6059 generic.go:356] "Generic (PLEG): container finished" podID="7297a2df-851c-4656-98cc-197ec88efe7b" containerID="344b4d8402a07ec67893c4b8c793e1f834ae2110ed38f1eab3449d37d646354b" exitCode=0
Nov 03 18:51:51.441268 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.441219    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2"
Nov 03 18:51:51.441550 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.441266    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2" event={"ID":"7297a2df-851c-4656-98cc-197ec88efe7b","Type":"ContainerDied","Data":"344b4d8402a07ec67893c4b8c793e1f834ae2110ed38f1eab3449d37d646354b"}
Nov 03 18:51:51.441550 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.441320    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2" event={"ID":"7297a2df-851c-4656-98cc-197ec88efe7b","Type":"ContainerDied","Data":"02db33e4f40eece9c14a789432cf3d776247c1c4e91ae95d28544cf8bca00248"}
Nov 03 18:51:51.441550 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.441350    6059 scope.go:122] "RemoveContainer" containerID="344b4d8402a07ec67893c4b8c793e1f834ae2110ed38f1eab3449d37d646354b"
Nov 03 18:51:51.442947 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.442918    6059 generic.go:356] "Generic (PLEG): container finished" podID="b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397" containerID="a27f5faeecbbc26b7f601c6abdbab37091cf3498da376dbce7c9022f338eaf04" exitCode=0
Nov 03 18:51:51.443070 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.442961    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v"
Nov 03 18:51:51.443070 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.442989    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v" event={"ID":"b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397","Type":"ContainerDied","Data":"a27f5faeecbbc26b7f601c6abdbab37091cf3498da376dbce7c9022f338eaf04"}
Nov 03 18:51:51.443070 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.443047    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v" event={"ID":"b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397","Type":"ContainerDied","Data":"671a7f25a58e4878db90e68a9b6bb86428b841de439bfec07ce1081458a32646"}
Nov 03 18:51:51.450821 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.450799    6059 scope.go:122] "RemoveContainer" containerID="344b4d8402a07ec67893c4b8c793e1f834ae2110ed38f1eab3449d37d646354b"
Nov 03 18:51:51.451141 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:51:51.451118    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"344b4d8402a07ec67893c4b8c793e1f834ae2110ed38f1eab3449d37d646354b\": container with ID starting with 344b4d8402a07ec67893c4b8c793e1f834ae2110ed38f1eab3449d37d646354b not found: ID does not exist" containerID="344b4d8402a07ec67893c4b8c793e1f834ae2110ed38f1eab3449d37d646354b"
Nov 03 18:51:51.451246 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.451148    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"344b4d8402a07ec67893c4b8c793e1f834ae2110ed38f1eab3449d37d646354b"} err="failed to get container status \"344b4d8402a07ec67893c4b8c793e1f834ae2110ed38f1eab3449d37d646354b\": rpc error: code = NotFound desc = could not find container \"344b4d8402a07ec67893c4b8c793e1f834ae2110ed38f1eab3449d37d646354b\": container with ID starting with 344b4d8402a07ec67893c4b8c793e1f834ae2110ed38f1eab3449d37d646354b not found: ID does not exist"
Nov 03 18:51:51.451246 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.451166    6059 scope.go:122] "RemoveContainer" containerID="a27f5faeecbbc26b7f601c6abdbab37091cf3498da376dbce7c9022f338eaf04"
Nov 03 18:51:51.460027 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.460005    6059 scope.go:122] "RemoveContainer" containerID="a27f5faeecbbc26b7f601c6abdbab37091cf3498da376dbce7c9022f338eaf04"
Nov 03 18:51:51.460337 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:51:51.460315    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a27f5faeecbbc26b7f601c6abdbab37091cf3498da376dbce7c9022f338eaf04\": container with ID starting with a27f5faeecbbc26b7f601c6abdbab37091cf3498da376dbce7c9022f338eaf04 not found: ID does not exist" containerID="a27f5faeecbbc26b7f601c6abdbab37091cf3498da376dbce7c9022f338eaf04"
Nov 03 18:51:51.460402 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.460346    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27f5faeecbbc26b7f601c6abdbab37091cf3498da376dbce7c9022f338eaf04"} err="failed to get container status \"a27f5faeecbbc26b7f601c6abdbab37091cf3498da376dbce7c9022f338eaf04\": rpc error: code = NotFound desc = could not find container \"a27f5faeecbbc26b7f601c6abdbab37091cf3498da376dbce7c9022f338eaf04\": container with ID starting with a27f5faeecbbc26b7f601c6abdbab37091cf3498da376dbce7c9022f338eaf04 not found: ID does not exist"
Nov 03 18:51:51.598977 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.598938    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v"]
Nov 03 18:51:51.602952 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.602922    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-27a21b-6ff6f98db28r9v"]
Nov 03 18:51:51.706978 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.706951    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2"]
Nov 03 18:51:51.711403 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:51.711375    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-27a21b-9f599z2fm2"]
Nov 03 18:51:53.335108 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:53.335071    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7297a2df-851c-4656-98cc-197ec88efe7b" path="/var/lib/kubelet/pods/7297a2df-851c-4656-98cc-197ec88efe7b/volumes"
Nov 03 18:51:53.335525 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:53.335480    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397" path="/var/lib/kubelet/pods/b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397/volumes"
Nov 03 18:51:54.050114 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:54.050078    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-b98zt"]
Nov 03 18:51:54.097312 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:54.097275    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-b98zt"]
Nov 03 18:51:55.334873 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:55.334834    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75c24353-2065-48bc-95c5-e6e590de013d" path="/var/lib/kubelet/pods/75c24353-2065-48bc-95c5-e6e590de013d/volumes"
Nov 03 18:51:59.976240 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:51:59.974721    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:00.565104 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:00.565071    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:01.994428 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:01.994351    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:02.593646 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:02.593613    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"]
Nov 03 18:52:02.593888 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:02.593862    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b" podUID="907144cd-9c12-45d8-a0c8-5529607d119e" containerName="istio-proxy" containerID="cri-o://d2bbb5f4a4899f2fc509073dbc8b2751adb4da2127734e9f0d55eabe34a5d518" gracePeriod=30
Nov 03 18:52:02.636867 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:02.636839    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:02.912899 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:02.912814    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:04.206443 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.206395    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xz9zx"]
Nov 03 18:52:04.253635 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.253596    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xz9zx"]
Nov 03 18:52:04.364803 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.364767    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn"]
Nov 03 18:52:04.365277 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.365264    6059 cpu_manager.go:414] "containerMap: removing container" podUID="7297a2df-851c-4656-98cc-197ec88efe7b" containerName="payload-pre-processing"
Nov 03 18:52:04.365329 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.365278    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7297a2df-851c-4656-98cc-197ec88efe7b" containerName="payload-pre-processing"
Nov 03 18:52:04.365329 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.365296    6059 cpu_manager.go:414] "containerMap: removing container" podUID="b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397" containerName="payload-processing"
Nov 03 18:52:04.365329 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.365303    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397" containerName="payload-processing"
Nov 03 18:52:04.365329 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.365313    6059 cpu_manager.go:414] "containerMap: removing container" podUID="7e799c8c-1dea-418d-baa4-73c655cfaa00" containerName="revoke-keys"
Nov 03 18:52:04.365329 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.365318    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e799c8c-1dea-418d-baa4-73c655cfaa00" containerName="revoke-keys"
Nov 03 18:52:04.365519 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.365394    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1ea1469-8ca5-4a6f-90c9-dc3cb09e9397" containerName="payload-processing"
Nov 03 18:52:04.365519 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.365405    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e799c8c-1dea-418d-baa4-73c655cfaa00" containerName="revoke-keys"
Nov 03 18:52:04.365519 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.365416    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7297a2df-851c-4656-98cc-197ec88efe7b" containerName="payload-pre-processing"
Nov 03 18:52:04.369014 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.368991    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn"
Nov 03 18:52:04.418707 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.418660    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:52:04.418707 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.418702    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:52:04.418993 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.418819    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:52:04.418993 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.418910    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:52:04.428779 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.428746    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn"]
Nov 03 18:52:04.445143 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.445112    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12b4c264-b6d4-486c-b2e1-30714f0742d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mn7cn\" (UID: \"12b4c264-b6d4-486c-b2e1-30714f0742d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn"
Nov 03 18:52:04.445329 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.445161    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkcmk\" (UniqueName: \"kubernetes.io/projected/12b4c264-b6d4-486c-b2e1-30714f0742d5-kube-api-access-rkcmk\") pod \"maas-api-revoke-sub-3d4156f1-mn7cn\" (UID: \"12b4c264-b6d4-486c-b2e1-30714f0742d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn"
Nov 03 18:52:04.445329 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.445282    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12b4c264-b6d4-486c-b2e1-30714f0742d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mn7cn\" (UID: \"12b4c264-b6d4-486c-b2e1-30714f0742d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn"
Nov 03 18:52:04.546611 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.546524    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkcmk\" (UniqueName: \"kubernetes.io/projected/12b4c264-b6d4-486c-b2e1-30714f0742d5-kube-api-access-rkcmk\") pod \"maas-api-revoke-sub-3d4156f1-mn7cn\" (UID: \"12b4c264-b6d4-486c-b2e1-30714f0742d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn"
Nov 03 18:52:04.546611 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.546573    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12b4c264-b6d4-486c-b2e1-30714f0742d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mn7cn\" (UID: \"12b4c264-b6d4-486c-b2e1-30714f0742d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn"
Nov 03 18:52:04.546857 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.546674    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12b4c264-b6d4-486c-b2e1-30714f0742d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mn7cn\" (UID: \"12b4c264-b6d4-486c-b2e1-30714f0742d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn"
Nov 03 18:52:04.547304 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.547283    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12b4c264-b6d4-486c-b2e1-30714f0742d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mn7cn\" (UID: \"12b4c264-b6d4-486c-b2e1-30714f0742d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn"
Nov 03 18:52:04.549059 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.549039    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12b4c264-b6d4-486c-b2e1-30714f0742d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mn7cn\" (UID: \"12b4c264-b6d4-486c-b2e1-30714f0742d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn"
Nov 03 18:52:04.605056 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.605018    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkcmk\" (UniqueName: \"kubernetes.io/projected/12b4c264-b6d4-486c-b2e1-30714f0742d5-kube-api-access-rkcmk\") pod \"maas-api-revoke-sub-3d4156f1-mn7cn\" (UID: \"12b4c264-b6d4-486c-b2e1-30714f0742d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn"
Nov 03 18:52:04.682567 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:04.682529    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn"
Nov 03 18:52:05.054416 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:05.054346    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn"]
Nov 03 18:52:05.056614 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:52:05.056580    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b4c264_b6d4_486c_b2e1_30714f0742d5.slice/crio-79432c9a730c7ed4409410bf2a9ad7b586100de484bbeb50962fd26f083574fc WatchSource:0}: Error finding container 79432c9a730c7ed4409410bf2a9ad7b586100de484bbeb50962fd26f083574fc: Status 404 returned error can't find the container with id 79432c9a730c7ed4409410bf2a9ad7b586100de484bbeb50962fd26f083574fc
Nov 03 18:52:05.336579 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:05.336548    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f41601e-94f8-48ef-9868-e9ffdb853b0c" path="/var/lib/kubelet/pods/1f41601e-94f8-48ef-9868-e9ffdb853b0c/volumes"
Nov 03 18:52:05.532518 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:05.532477    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn" event={"ID":"12b4c264-b6d4-486c-b2e1-30714f0742d5","Type":"ContainerStarted","Data":"30ff0a4c035610200b757afbd0e92d246aad7f46a850eb8ca906acf4205869ff"}
Nov 03 18:52:05.532693 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:05.532529    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn" event={"ID":"12b4c264-b6d4-486c-b2e1-30714f0742d5","Type":"ContainerStarted","Data":"79432c9a730c7ed4409410bf2a9ad7b586100de484bbeb50962fd26f083574fc"}
Nov 03 18:52:05.644042 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:05.643933    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn" podStartSLOduration=1.6439177520000001 podStartE2EDuration="1.643917752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:52:05.642702099 +0000 UTC m=+5666.440414880" watchObservedRunningTime="2026-11-03 18:52:05.643917752 +0000 UTC m=+5666.441630564"
Nov 03 18:52:05.865487 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:05.865453    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:06.043874 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:06.043836    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:06.540341 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:06.540303    6059 generic.go:356] "Generic (PLEG): container finished" podID="12b4c264-b6d4-486c-b2e1-30714f0742d5" containerID="30ff0a4c035610200b757afbd0e92d246aad7f46a850eb8ca906acf4205869ff" exitCode=0
Nov 03 18:52:06.540762 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:06.540383    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn" event={"ID":"12b4c264-b6d4-486c-b2e1-30714f0742d5","Type":"ContainerDied","Data":"30ff0a4c035610200b757afbd0e92d246aad7f46a850eb8ca906acf4205869ff"}
Nov 03 18:52:06.863978 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:52:06.863890    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:52:07.018888 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.018847    6059 patch_prober.go:29] interesting pod/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:52:07.019093 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.018905    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b" podUID="907144cd-9c12-45d8-a0c8-5529607d119e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:52:07.722651 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.722622    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn"
Nov 03 18:52:07.781156 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.781124    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12b4c264-b6d4-486c-b2e1-30714f0742d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12b4c264-b6d4-486c-b2e1-30714f0742d5-maas-api-xks-ca\") pod \"12b4c264-b6d4-486c-b2e1-30714f0742d5\" (UID: \"12b4c264-b6d4-486c-b2e1-30714f0742d5\") "
Nov 03 18:52:07.781349 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.781188    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12b4c264-b6d4-486c-b2e1-30714f0742d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12b4c264-b6d4-486c-b2e1-30714f0742d5-maas-api-openshift-ca\") pod \"12b4c264-b6d4-486c-b2e1-30714f0742d5\" (UID: \"12b4c264-b6d4-486c-b2e1-30714f0742d5\") "
Nov 03 18:52:07.781349 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.781230    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12b4c264-b6d4-486c-b2e1-30714f0742d5-kube-api-access-rkcmk\" (UniqueName: \"kubernetes.io/projected/12b4c264-b6d4-486c-b2e1-30714f0742d5-kube-api-access-rkcmk\") pod \"12b4c264-b6d4-486c-b2e1-30714f0742d5\" (UID: \"12b4c264-b6d4-486c-b2e1-30714f0742d5\") "
Nov 03 18:52:07.781679 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.781651    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b4c264-b6d4-486c-b2e1-30714f0742d5-maas-api-openshift-ca" pod "12b4c264-b6d4-486c-b2e1-30714f0742d5" (UID: "12b4c264-b6d4-486c-b2e1-30714f0742d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:52:07.783748 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.783695    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b4c264-b6d4-486c-b2e1-30714f0742d5-maas-api-xks-ca" pod "12b4c264-b6d4-486c-b2e1-30714f0742d5" (UID: "12b4c264-b6d4-486c-b2e1-30714f0742d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:52:07.784079 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.784053    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b4c264-b6d4-486c-b2e1-30714f0742d5-kube-api-access-rkcmk" pod "12b4c264-b6d4-486c-b2e1-30714f0742d5" (UID: "12b4c264-b6d4-486c-b2e1-30714f0742d5"). InnerVolumeSpecName "kube-api-access-rkcmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:52:07.831845 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.831818    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:52:07.882231 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.882198    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/907144cd-9c12-45d8-a0c8-5529607d119e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907144cd-9c12-45d8-a0c8-5529607d119e-istio-podinfo\") pod \"907144cd-9c12-45d8-a0c8-5529607d119e\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") "
Nov 03 18:52:07.882450 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.882262    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-workload-socket\") pod \"907144cd-9c12-45d8-a0c8-5529607d119e\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") "
Nov 03 18:52:07.882450 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.882285    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-istio-envoy\") pod \"907144cd-9c12-45d8-a0c8-5529607d119e\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") "
Nov 03 18:52:07.882450 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.882415    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-istio-data\") pod \"907144cd-9c12-45d8-a0c8-5529607d119e\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") "
Nov 03 18:52:07.882645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.882458    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/907144cd-9c12-45d8-a0c8-5529607d119e-istio-token\" (UniqueName: \"kubernetes.io/projected/907144cd-9c12-45d8-a0c8-5529607d119e-istio-token\") pod \"907144cd-9c12-45d8-a0c8-5529607d119e\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") "
Nov 03 18:52:07.882645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.882528    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-workload-certs\") pod \"907144cd-9c12-45d8-a0c8-5529607d119e\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") "
Nov 03 18:52:07.882645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.882578    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/907144cd-9c12-45d8-a0c8-5529607d119e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907144cd-9c12-45d8-a0c8-5529607d119e-istiod-ca-cert\") pod \"907144cd-9c12-45d8-a0c8-5529607d119e\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") "
Nov 03 18:52:07.882645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.882593    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-workload-socket" pod "907144cd-9c12-45d8-a0c8-5529607d119e" (UID: "907144cd-9c12-45d8-a0c8-5529607d119e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:52:07.882645 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.882620    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/907144cd-9c12-45d8-a0c8-5529607d119e-kube-api-access-nh96s\" (UniqueName: \"kubernetes.io/projected/907144cd-9c12-45d8-a0c8-5529607d119e-kube-api-access-nh96s\") pod \"907144cd-9c12-45d8-a0c8-5529607d119e\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") "
Nov 03 18:52:07.882953 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.882659    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-istio-data" pod "907144cd-9c12-45d8-a0c8-5529607d119e" (UID: "907144cd-9c12-45d8-a0c8-5529607d119e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:52:07.882953 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.882730    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-credential-socket\") pod \"907144cd-9c12-45d8-a0c8-5529607d119e\" (UID: \"907144cd-9c12-45d8-a0c8-5529607d119e\") "
Nov 03 18:52:07.882953 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.882830    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-workload-certs" pod "907144cd-9c12-45d8-a0c8-5529607d119e" (UID: "907144cd-9c12-45d8-a0c8-5529607d119e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:52:07.883117 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.882962    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/907144cd-9c12-45d8-a0c8-5529607d119e-istiod-ca-cert" pod "907144cd-9c12-45d8-a0c8-5529607d119e" (UID: "907144cd-9c12-45d8-a0c8-5529607d119e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:52:07.883117 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.883091    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-credential-socket" pod "907144cd-9c12-45d8-a0c8-5529607d119e" (UID: "907144cd-9c12-45d8-a0c8-5529607d119e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:52:07.883271 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.883251    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:52:07.883316 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.883278    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:52:07.883316 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.883294    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:52:07.883316 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.883308    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907144cd-9c12-45d8-a0c8-5529607d119e-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:52:07.883490 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.883323    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12b4c264-b6d4-486c-b2e1-30714f0742d5-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:52:07.883490 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.883337    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12b4c264-b6d4-486c-b2e1-30714f0742d5-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:52:07.883490 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.883370    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:52:07.883490 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.883387    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkcmk\" (UniqueName: \"kubernetes.io/projected/12b4c264-b6d4-486c-b2e1-30714f0742d5-kube-api-access-rkcmk\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:52:07.884548 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.884523    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/907144cd-9c12-45d8-a0c8-5529607d119e-istio-podinfo" pod "907144cd-9c12-45d8-a0c8-5529607d119e" (UID: "907144cd-9c12-45d8-a0c8-5529607d119e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:52:07.884706 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.884685    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907144cd-9c12-45d8-a0c8-5529607d119e-kube-api-access-nh96s" pod "907144cd-9c12-45d8-a0c8-5529607d119e" (UID: "907144cd-9c12-45d8-a0c8-5529607d119e"). InnerVolumeSpecName "kube-api-access-nh96s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:52:07.884976 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.884954    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-istio-envoy" pod "907144cd-9c12-45d8-a0c8-5529607d119e" (UID: "907144cd-9c12-45d8-a0c8-5529607d119e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:52:07.884976 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.884964    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907144cd-9c12-45d8-a0c8-5529607d119e-istio-token" pod "907144cd-9c12-45d8-a0c8-5529607d119e" (UID: "907144cd-9c12-45d8-a0c8-5529607d119e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:52:07.984828 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.984784    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907144cd-9c12-45d8-a0c8-5529607d119e-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:52:07.984828 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.984829    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/907144cd-9c12-45d8-a0c8-5529607d119e-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:52:07.985038 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.984846    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh96s\" (UniqueName: \"kubernetes.io/projected/907144cd-9c12-45d8-a0c8-5529607d119e-kube-api-access-nh96s\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:52:07.985038 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:07.984861    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907144cd-9c12-45d8-a0c8-5529607d119e-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:52:08.553910 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:08.553876    6059 generic.go:356] "Generic (PLEG): container finished" podID="907144cd-9c12-45d8-a0c8-5529607d119e" containerID="d2bbb5f4a4899f2fc509073dbc8b2751adb4da2127734e9f0d55eabe34a5d518" exitCode=0
Nov 03 18:52:08.554120 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:08.553946    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"
Nov 03 18:52:08.554120 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:08.553956    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b" event={"ID":"907144cd-9c12-45d8-a0c8-5529607d119e","Type":"ContainerDied","Data":"d2bbb5f4a4899f2fc509073dbc8b2751adb4da2127734e9f0d55eabe34a5d518"}
Nov 03 18:52:08.554120 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:08.554016    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b" event={"ID":"907144cd-9c12-45d8-a0c8-5529607d119e","Type":"ContainerDied","Data":"e3ef705f1424ed21b19184b40ced19b1f028ca34338b97e90e973c5c5c528a95"}
Nov 03 18:52:08.554120 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:08.554029    6059 scope.go:122] "RemoveContainer" containerID="d2bbb5f4a4899f2fc509073dbc8b2751adb4da2127734e9f0d55eabe34a5d518"
Nov 03 18:52:08.555554 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:08.555527    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn" event={"ID":"12b4c264-b6d4-486c-b2e1-30714f0742d5","Type":"ContainerDied","Data":"79432c9a730c7ed4409410bf2a9ad7b586100de484bbeb50962fd26f083574fc"}
Nov 03 18:52:08.555664 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:08.555560    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn"
Nov 03 18:52:08.555664 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:08.555577    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79432c9a730c7ed4409410bf2a9ad7b586100de484bbeb50962fd26f083574fc"
Nov 03 18:52:08.565801 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:08.565776    6059 scope.go:122] "RemoveContainer" containerID="6e6348351ede0e49367edce747a41cbf99f13dc7144839c7c5a26b9dc4f818b9"
Nov 03 18:52:08.575117 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:08.575099    6059 scope.go:122] "RemoveContainer" containerID="d2bbb5f4a4899f2fc509073dbc8b2751adb4da2127734e9f0d55eabe34a5d518"
Nov 03 18:52:08.575421 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:52:08.575399    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2bbb5f4a4899f2fc509073dbc8b2751adb4da2127734e9f0d55eabe34a5d518\": container with ID starting with d2bbb5f4a4899f2fc509073dbc8b2751adb4da2127734e9f0d55eabe34a5d518 not found: ID does not exist" containerID="d2bbb5f4a4899f2fc509073dbc8b2751adb4da2127734e9f0d55eabe34a5d518"
Nov 03 18:52:08.575514 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:08.575435    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2bbb5f4a4899f2fc509073dbc8b2751adb4da2127734e9f0d55eabe34a5d518"} err="failed to get container status \"d2bbb5f4a4899f2fc509073dbc8b2751adb4da2127734e9f0d55eabe34a5d518\": rpc error: code = NotFound desc = could not find container \"d2bbb5f4a4899f2fc509073dbc8b2751adb4da2127734e9f0d55eabe34a5d518\": container with ID starting with d2bbb5f4a4899f2fc509073dbc8b2751adb4da2127734e9f0d55eabe34a5d518 not found: ID does not exist"
Nov 03 18:52:08.575514 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:08.575462    6059 scope.go:122] "RemoveContainer" containerID="6e6348351ede0e49367edce747a41cbf99f13dc7144839c7c5a26b9dc4f818b9"
Nov 03 18:52:08.575791 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:52:08.575767    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e6348351ede0e49367edce747a41cbf99f13dc7144839c7c5a26b9dc4f818b9\": container with ID starting with 6e6348351ede0e49367edce747a41cbf99f13dc7144839c7c5a26b9dc4f818b9 not found: ID does not exist" containerID="6e6348351ede0e49367edce747a41cbf99f13dc7144839c7c5a26b9dc4f818b9"
Nov 03 18:52:08.575847 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:08.575802    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6348351ede0e49367edce747a41cbf99f13dc7144839c7c5a26b9dc4f818b9"} err="failed to get container status \"6e6348351ede0e49367edce747a41cbf99f13dc7144839c7c5a26b9dc4f818b9\": rpc error: code = NotFound desc = could not find container \"6e6348351ede0e49367edce747a41cbf99f13dc7144839c7c5a26b9dc4f818b9\": container with ID starting with 6e6348351ede0e49367edce747a41cbf99f13dc7144839c7c5a26b9dc4f818b9 not found: ID does not exist"
Nov 03 18:52:08.759411 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:08.759376    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"]
Nov 03 18:52:08.763596 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:08.763565    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-27a21b-openshift-default-c6b66878f-rr26b"]
Nov 03 18:52:09.337340 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:09.337297    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="907144cd-9c12-45d8-a0c8-5529607d119e" path="/var/lib/kubelet/pods/907144cd-9c12-45d8-a0c8-5529607d119e/volumes"
Nov 03 18:52:14.050452 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:14.050419    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb"]
Nov 03 18:52:14.097654 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:14.097622    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-glfvb"]
Nov 03 18:52:15.335455 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:15.335419    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fce4902-178e-4c23-88d0-1d7190733721" path="/var/lib/kubelet/pods/1fce4902-178e-4c23-88d0-1d7190733721/volumes"
Nov 03 18:52:34.049798 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:34.049721    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9jbg9"]
Nov 03 18:52:34.097051 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:34.097013    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9jbg9"]
Nov 03 18:52:35.335136 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:35.335100    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e90136c7-08d4-4ab1-88b2-a2f2a2dd3595" path="/var/lib/kubelet/pods/e90136c7-08d4-4ab1-88b2-a2f2a2dd3595/volumes"
Nov 03 18:52:36.862184 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:52:36.862139    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:52:40.203503 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:40.203475    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:52:40.221618 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:40.221588    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:52:40.226220 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:40.226202    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:52:40.242499 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:40.242474    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:52:44.158632 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:44.158599    6059 scope.go:122] "RemoveContainer" containerID="ded52d7ca40a16d90612a554a5b5ce8e15f218940e72cc64b52485d1fb1698da"
Nov 03 18:52:44.167501 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:44.167480    6059 scope.go:122] "RemoveContainer" containerID="e829fc3fb23c8bbddfecb93803ad1af2b2fbab78b6b1c7376ad1b739df0e488c"
Nov 03 18:52:52.047202 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:52.047169    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqkgt"]
Nov 03 18:52:52.094203 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:52.094173    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pqkgt"]
Nov 03 18:52:53.334902 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:52:53.334865    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a" path="/var/lib/kubelet/pods/5013240a-fb8b-48c8-8ff9-6cf0e33c9a4a/volumes"
Nov 03 18:53:03.270844 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:03.270803    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:53:03.452527 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:03.452489    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:53:06.862764 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:53:06.862723    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:53:36.860374 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:53:36.860326    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:53:44.212598 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:44.212569    6059 scope.go:122] "RemoveContainer" containerID="93ab09bfc57910b7046d4f8d1652a6f70909b88f5f73a02e013b69cc09c7d791"
Nov 03 18:53:44.224521 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:44.224496    6059 scope.go:122] "RemoveContainer" containerID="3a5ebaf5835ce76e506950c4b51ddb3d9a039b4a2b86285163aeab5a0d08f8d2"
Nov 03 18:53:44.232554 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:44.232522    6059 scope.go:122] "RemoveContainer" containerID="d4e5fff7f08bde94cc56a4ba0fcb88ba06c6e80271f78e4b77e0f6f9a3ab1bc6"
Nov 03 18:53:54.516487 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.516450    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-f2smj"]
Nov 03 18:53:54.516939 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.516922    6059 cpu_manager.go:414] "containerMap: removing container" podUID="907144cd-9c12-45d8-a0c8-5529607d119e" containerName="istio-proxy"
Nov 03 18:53:54.516991 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.516940    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="907144cd-9c12-45d8-a0c8-5529607d119e" containerName="istio-proxy"
Nov 03 18:53:54.516991 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.516960    6059 cpu_manager.go:414] "containerMap: removing container" podUID="907144cd-9c12-45d8-a0c8-5529607d119e" containerName="istio-proxy"
Nov 03 18:53:54.516991 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.516966    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="907144cd-9c12-45d8-a0c8-5529607d119e" containerName="istio-proxy"
Nov 03 18:53:54.516991 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.516976    6059 cpu_manager.go:414] "containerMap: removing container" podUID="12b4c264-b6d4-486c-b2e1-30714f0742d5" containerName="revoke-keys"
Nov 03 18:53:54.516991 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.516982    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b4c264-b6d4-486c-b2e1-30714f0742d5" containerName="revoke-keys"
Nov 03 18:53:54.517151 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.517055    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="907144cd-9c12-45d8-a0c8-5529607d119e" containerName="istio-proxy"
Nov 03 18:53:54.517151 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.517063    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="907144cd-9c12-45d8-a0c8-5529607d119e" containerName="istio-proxy"
Nov 03 18:53:54.517151 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.517069    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b4c264-b6d4-486c-b2e1-30714f0742d5" containerName="revoke-keys"
Nov 03 18:53:54.522273 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.522252    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-f2smj"
Nov 03 18:53:54.575367 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.575328    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-f2smj"]
Nov 03 18:53:54.598130 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.598101    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/117b3fb5-968b-46a5-9c3c-c6155c68ccf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-f2smj\" (UID: \"117b3fb5-968b-46a5-9c3c-c6155c68ccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-f2smj"
Nov 03 18:53:54.598301 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.598284    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/117b3fb5-968b-46a5-9c3c-c6155c68ccf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-f2smj\" (UID: \"117b3fb5-968b-46a5-9c3c-c6155c68ccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-f2smj"
Nov 03 18:53:54.598381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.598310    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s884k\" (UniqueName: \"kubernetes.io/projected/117b3fb5-968b-46a5-9c3c-c6155c68ccf5-kube-api-access-s884k\") pod \"maas-api-revoke-sub-d576e68b-f2smj\" (UID: \"117b3fb5-968b-46a5-9c3c-c6155c68ccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-f2smj"
Nov 03 18:53:54.619820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.619798    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:53:54.619969 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.619842    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:53:54.619969 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.619920    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:53:54.619969 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.619939    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:53:54.699251 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.699222    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/117b3fb5-968b-46a5-9c3c-c6155c68ccf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-f2smj\" (UID: \"117b3fb5-968b-46a5-9c3c-c6155c68ccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-f2smj"
Nov 03 18:53:54.699406 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.699312    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/117b3fb5-968b-46a5-9c3c-c6155c68ccf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-f2smj\" (UID: \"117b3fb5-968b-46a5-9c3c-c6155c68ccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-f2smj"
Nov 03 18:53:54.699406 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.699331    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s884k\" (UniqueName: \"kubernetes.io/projected/117b3fb5-968b-46a5-9c3c-c6155c68ccf5-kube-api-access-s884k\") pod \"maas-api-revoke-sub-d576e68b-f2smj\" (UID: \"117b3fb5-968b-46a5-9c3c-c6155c68ccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-f2smj"
Nov 03 18:53:54.699913 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.699891    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/117b3fb5-968b-46a5-9c3c-c6155c68ccf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-f2smj\" (UID: \"117b3fb5-968b-46a5-9c3c-c6155c68ccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-f2smj"
Nov 03 18:53:54.701644 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.701623    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/117b3fb5-968b-46a5-9c3c-c6155c68ccf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-f2smj\" (UID: \"117b3fb5-968b-46a5-9c3c-c6155c68ccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-f2smj"
Nov 03 18:53:54.758134 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.758098    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s884k\" (UniqueName: \"kubernetes.io/projected/117b3fb5-968b-46a5-9c3c-c6155c68ccf5-kube-api-access-s884k\") pod \"maas-api-revoke-sub-d576e68b-f2smj\" (UID: \"117b3fb5-968b-46a5-9c3c-c6155c68ccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-f2smj"
Nov 03 18:53:54.835280 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:54.835200    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-f2smj"
Nov 03 18:53:55.202518 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:55.202490    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-f2smj"]
Nov 03 18:53:55.204945 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:53:55.204916    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117b3fb5_968b_46a5_9c3c_c6155c68ccf5.slice/crio-89301d34173f225e93bdcc1ef7b0b6e7be4d2ef83ca0f70ca1b19b6533e4d226 WatchSource:0}: Error finding container 89301d34173f225e93bdcc1ef7b0b6e7be4d2ef83ca0f70ca1b19b6533e4d226: Status 404 returned error can't find the container with id 89301d34173f225e93bdcc1ef7b0b6e7be4d2ef83ca0f70ca1b19b6533e4d226
Nov 03 18:53:55.233458 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:55.233427    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-f2smj" event={"ID":"117b3fb5-968b-46a5-9c3c-c6155c68ccf5","Type":"ContainerStarted","Data":"89301d34173f225e93bdcc1ef7b0b6e7be4d2ef83ca0f70ca1b19b6533e4d226"}
Nov 03 18:53:56.240482 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:56.240446    6059 generic.go:356] "Generic (PLEG): container finished" podID="117b3fb5-968b-46a5-9c3c-c6155c68ccf5" containerID="30f408ec57329c081d0d745a585ba4cd9b4fbc9402ab584a6b4a7dae55712f5d" exitCode=0
Nov 03 18:53:56.240898 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:56.240541    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-f2smj" event={"ID":"117b3fb5-968b-46a5-9c3c-c6155c68ccf5","Type":"ContainerDied","Data":"30f408ec57329c081d0d745a585ba4cd9b4fbc9402ab584a6b4a7dae55712f5d"}
Nov 03 18:53:57.384598 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:57.384570    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-f2smj"
Nov 03 18:53:57.428554 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:57.428525    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/117b3fb5-968b-46a5-9c3c-c6155c68ccf5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/117b3fb5-968b-46a5-9c3c-c6155c68ccf5-maas-api-openshift-ca\") pod \"117b3fb5-968b-46a5-9c3c-c6155c68ccf5\" (UID: \"117b3fb5-968b-46a5-9c3c-c6155c68ccf5\") "
Nov 03 18:53:57.428707 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:57.428561    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/117b3fb5-968b-46a5-9c3c-c6155c68ccf5-kube-api-access-s884k\" (UniqueName: \"kubernetes.io/projected/117b3fb5-968b-46a5-9c3c-c6155c68ccf5-kube-api-access-s884k\") pod \"117b3fb5-968b-46a5-9c3c-c6155c68ccf5\" (UID: \"117b3fb5-968b-46a5-9c3c-c6155c68ccf5\") "
Nov 03 18:53:57.428707 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:57.428662    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/117b3fb5-968b-46a5-9c3c-c6155c68ccf5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/117b3fb5-968b-46a5-9c3c-c6155c68ccf5-maas-api-xks-ca\") pod \"117b3fb5-968b-46a5-9c3c-c6155c68ccf5\" (UID: \"117b3fb5-968b-46a5-9c3c-c6155c68ccf5\") "
Nov 03 18:53:57.428872 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:57.428850    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/117b3fb5-968b-46a5-9c3c-c6155c68ccf5-maas-api-openshift-ca" pod "117b3fb5-968b-46a5-9c3c-c6155c68ccf5" (UID: "117b3fb5-968b-46a5-9c3c-c6155c68ccf5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:53:57.429061 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:57.429040    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/117b3fb5-968b-46a5-9c3c-c6155c68ccf5-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:53:57.430830 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:57.430801    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117b3fb5-968b-46a5-9c3c-c6155c68ccf5-kube-api-access-s884k" pod "117b3fb5-968b-46a5-9c3c-c6155c68ccf5" (UID: "117b3fb5-968b-46a5-9c3c-c6155c68ccf5"). InnerVolumeSpecName "kube-api-access-s884k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:53:57.430929 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:57.430865    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/117b3fb5-968b-46a5-9c3c-c6155c68ccf5-maas-api-xks-ca" pod "117b3fb5-968b-46a5-9c3c-c6155c68ccf5" (UID: "117b3fb5-968b-46a5-9c3c-c6155c68ccf5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:53:57.530421 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:57.530329    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s884k\" (UniqueName: \"kubernetes.io/projected/117b3fb5-968b-46a5-9c3c-c6155c68ccf5-kube-api-access-s884k\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:53:57.530421 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:57.530392    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/117b3fb5-968b-46a5-9c3c-c6155c68ccf5-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:53:58.253576 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:58.253540    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-f2smj"
Nov 03 18:53:58.253763 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:58.253577    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-f2smj" event={"ID":"117b3fb5-968b-46a5-9c3c-c6155c68ccf5","Type":"ContainerDied","Data":"89301d34173f225e93bdcc1ef7b0b6e7be4d2ef83ca0f70ca1b19b6533e4d226"}
Nov 03 18:53:58.253763 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:53:58.253625    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89301d34173f225e93bdcc1ef7b0b6e7be4d2ef83ca0f70ca1b19b6533e4d226"
Nov 03 18:54:03.676611 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:03.676574    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:54:06.861282 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:54:06.861244    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:54:08.056382 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:08.048146    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d"]
Nov 03 18:54:08.098382 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:08.098012    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-c5h4d"]
Nov 03 18:54:09.386063 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:09.386025    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce9d50b-a49c-41e2-9111-33b9faf3c5ff" path="/var/lib/kubelet/pods/5ce9d50b-a49c-41e2-9111-33b9faf3c5ff/volumes"
Nov 03 18:54:14.293386 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:14.292425    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9jpj5"]
Nov 03 18:54:14.293386 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:14.293375    6059 cpu_manager.go:414] "containerMap: removing container" podUID="117b3fb5-968b-46a5-9c3c-c6155c68ccf5" containerName="revoke-keys"
Nov 03 18:54:14.294080 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:14.293397    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117b3fb5-968b-46a5-9c3c-c6155c68ccf5" containerName="revoke-keys"
Nov 03 18:54:14.294080 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:14.293587    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="117b3fb5-968b-46a5-9c3c-c6155c68ccf5" containerName="revoke-keys"
Nov 03 18:54:14.305657 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:14.305623    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9jpj5"
Nov 03 18:54:14.358225 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:14.358183    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9jpj5"]
Nov 03 18:54:14.404749 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:14.404715    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:54:14.405112 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:14.405092    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:54:14.405917 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:14.405896    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:54:14.406689 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:14.406664    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4zvz\" (UniqueName: \"kubernetes.io/projected/9233790f-7d17-42c8-9d8d-5c3841b4af01-kube-api-access-w4zvz\") pod \"maas-api-revoke-sub-d2bc93bc-9jpj5\" (UID: \"9233790f-7d17-42c8-9d8d-5c3841b4af01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9jpj5"
Nov 03 18:54:14.406854 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:14.406835    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9233790f-7d17-42c8-9d8d-5c3841b4af01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9jpj5\" (UID: \"9233790f-7d17-42c8-9d8d-5c3841b4af01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9jpj5"
Nov 03 18:54:14.407013 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:14.406996    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9233790f-7d17-42c8-9d8d-5c3841b4af01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9jpj5\" (UID: \"9233790f-7d17-42c8-9d8d-5c3841b4af01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9jpj5"
Nov 03 18:54:14.407729 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:14.407706    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:54:14.523379 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:14.522876    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4zvz\" (UniqueName: \"kubernetes.io/projected/9233790f-7d17-42c8-9d8d-5c3841b4af01-kube-api-access-w4zvz\") pod \"maas-api-revoke-sub-d2bc93bc-9jpj5\" (UID: \"9233790f-7d17-42c8-9d8d-5c3841b4af01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9jpj5"
Nov 03 18:54:14.523379 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:14.522937    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9233790f-7d17-42c8-9d8d-5c3841b4af01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9jpj5\" (UID: \"9233790f-7d17-42c8-9d8d-5c3841b4af01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9jpj5"
Nov 03 18:54:14.523379 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:14.522995    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9233790f-7d17-42c8-9d8d-5c3841b4af01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9jpj5\" (UID: \"9233790f-7d17-42c8-9d8d-5c3841b4af01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9jpj5"
Nov 03 18:54:14.524383 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:14.523940    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9233790f-7d17-42c8-9d8d-5c3841b4af01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9jpj5\" (UID: \"9233790f-7d17-42c8-9d8d-5c3841b4af01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9jpj5"
Nov 03 18:54:14.601351 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:14.601216    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9233790f-7d17-42c8-9d8d-5c3841b4af01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9jpj5\" (UID: \"9233790f-7d17-42c8-9d8d-5c3841b4af01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9jpj5"
Nov 03 18:54:14.632381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:14.619081    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4zvz\" (UniqueName: \"kubernetes.io/projected/9233790f-7d17-42c8-9d8d-5c3841b4af01-kube-api-access-w4zvz\") pod \"maas-api-revoke-sub-d2bc93bc-9jpj5\" (UID: \"9233790f-7d17-42c8-9d8d-5c3841b4af01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9jpj5"
Nov 03 18:54:14.636390 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:14.635555    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9jpj5"
Nov 03 18:54:14.723092 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:14.723019    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:54:15.663497 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:15.663467    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9jpj5"]
Nov 03 18:54:15.693780 ip-10-0-143-117 kubenswrapper[6059]: W1103 18:54:15.693730    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9233790f_7d17_42c8_9d8d_5c3841b4af01.slice/crio-037520ca879cf29866bf4c5d3bb9cbc947395ae5e9ae404f9eff44763a0bc903 WatchSource:0}: Error finding container 037520ca879cf29866bf4c5d3bb9cbc947395ae5e9ae404f9eff44763a0bc903: Status 404 returned error can't find the container with id 037520ca879cf29866bf4c5d3bb9cbc947395ae5e9ae404f9eff44763a0bc903
Nov 03 18:54:16.820288 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:16.820244    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9jpj5" event={"ID":"9233790f-7d17-42c8-9d8d-5c3841b4af01","Type":"ContainerStarted","Data":"22aa00a42740cb2059ba338ff1e26a3f217b0146bce62ee31845c5b77269ebe3"}
Nov 03 18:54:16.820974 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:16.820948    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9jpj5" event={"ID":"9233790f-7d17-42c8-9d8d-5c3841b4af01","Type":"ContainerStarted","Data":"037520ca879cf29866bf4c5d3bb9cbc947395ae5e9ae404f9eff44763a0bc903"}
Nov 03 18:54:17.056765 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:17.054460    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zp9n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:54:17.056765 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:17.054530    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7" podUID="60b66cb6-2bce-4c02-aa7d-7eb25f29f237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:54:17.079643 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:17.079533    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xpshn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:54:17.079643 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:17.079610    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn" podUID="d9e3abf7-c5e1-4414-858e-5f84f6a88670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:54:17.827448 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:17.827416    6059 generic.go:356] "Generic (PLEG): container finished" podID="9233790f-7d17-42c8-9d8d-5c3841b4af01" containerID="22aa00a42740cb2059ba338ff1e26a3f217b0146bce62ee31845c5b77269ebe3" exitCode=0
Nov 03 18:54:17.827860 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:17.827512    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9jpj5" event={"ID":"9233790f-7d17-42c8-9d8d-5c3841b4af01","Type":"ContainerDied","Data":"22aa00a42740cb2059ba338ff1e26a3f217b0146bce62ee31845c5b77269ebe3"}
Nov 03 18:54:18.045891 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:18.045861    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bsq4t"]
Nov 03 18:54:18.048387 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:18.048349    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bsq4t"]
Nov 03 18:54:18.967545 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:18.967520    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9jpj5"
Nov 03 18:54:19.037124 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:19.037093    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9233790f-7d17-42c8-9d8d-5c3841b4af01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9233790f-7d17-42c8-9d8d-5c3841b4af01-maas-api-xks-ca\") pod \"9233790f-7d17-42c8-9d8d-5c3841b4af01\" (UID: \"9233790f-7d17-42c8-9d8d-5c3841b4af01\") "
Nov 03 18:54:19.037285 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:19.037241    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9233790f-7d17-42c8-9d8d-5c3841b4af01-kube-api-access-w4zvz\" (UniqueName: \"kubernetes.io/projected/9233790f-7d17-42c8-9d8d-5c3841b4af01-kube-api-access-w4zvz\") pod \"9233790f-7d17-42c8-9d8d-5c3841b4af01\" (UID: \"9233790f-7d17-42c8-9d8d-5c3841b4af01\") "
Nov 03 18:54:19.037368 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:19.037333    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9233790f-7d17-42c8-9d8d-5c3841b4af01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9233790f-7d17-42c8-9d8d-5c3841b4af01-maas-api-openshift-ca\") pod \"9233790f-7d17-42c8-9d8d-5c3841b4af01\" (UID: \"9233790f-7d17-42c8-9d8d-5c3841b4af01\") "
Nov 03 18:54:19.037729 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:19.037702    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9233790f-7d17-42c8-9d8d-5c3841b4af01-maas-api-openshift-ca" pod "9233790f-7d17-42c8-9d8d-5c3841b4af01" (UID: "9233790f-7d17-42c8-9d8d-5c3841b4af01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:54:19.039219 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:19.039194    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9233790f-7d17-42c8-9d8d-5c3841b4af01-maas-api-xks-ca" pod "9233790f-7d17-42c8-9d8d-5c3841b4af01" (UID: "9233790f-7d17-42c8-9d8d-5c3841b4af01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:54:19.039333 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:19.039310    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9233790f-7d17-42c8-9d8d-5c3841b4af01-kube-api-access-w4zvz" pod "9233790f-7d17-42c8-9d8d-5c3841b4af01" (UID: "9233790f-7d17-42c8-9d8d-5c3841b4af01"). InnerVolumeSpecName "kube-api-access-w4zvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:54:19.138662 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:19.138579    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9233790f-7d17-42c8-9d8d-5c3841b4af01-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:54:19.138662 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:19.138611    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9233790f-7d17-42c8-9d8d-5c3841b4af01-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:54:19.138662 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:19.138622    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4zvz\" (UniqueName: \"kubernetes.io/projected/9233790f-7d17-42c8-9d8d-5c3841b4af01-kube-api-access-w4zvz\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:54:19.335289 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:19.335257    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9" path="/var/lib/kubelet/pods/c7ff47a7-5539-4ebf-94c3-fa8daec1a1b9/volumes"
Nov 03 18:54:19.840129 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:19.840097    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9jpj5"
Nov 03 18:54:19.840129 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:19.840117    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9jpj5" event={"ID":"9233790f-7d17-42c8-9d8d-5c3841b4af01","Type":"ContainerDied","Data":"037520ca879cf29866bf4c5d3bb9cbc947395ae5e9ae404f9eff44763a0bc903"}
Nov 03 18:54:19.840387 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:19.840176    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="037520ca879cf29866bf4c5d3bb9cbc947395ae5e9ae404f9eff44763a0bc903"
Nov 03 18:54:25.524252 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:25.524212    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:54:27.886008 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:27.885973    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:54:28.063241 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:28.063206    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:54:33.634820 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:33.634780    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:54:33.823800 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:33.823748    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:54:36.861058 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:54:36.861019    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:54:40.758383 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:40.758105    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn"]
Nov 03 18:54:40.758956 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:40.758879    6059 cpu_manager.go:414] "containerMap: removing container" podUID="9233790f-7d17-42c8-9d8d-5c3841b4af01" containerName="revoke-keys"
Nov 03 18:54:40.758956 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:40.758903    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9233790f-7d17-42c8-9d8d-5c3841b4af01" containerName="revoke-keys"
Nov 03 18:54:40.759102 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:40.759000    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9233790f-7d17-42c8-9d8d-5c3841b4af01" containerName="revoke-keys"
Nov 03 18:54:40.783381 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:40.780719    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn"
Nov 03 18:54:40.820589 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:40.820559    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn"]
Nov 03 18:54:40.934380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:40.930542    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:54:40.934380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:40.931227    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:54:40.934380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:40.931292    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:54:40.934380 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:40.931416    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 18:54:41.056435 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:41.040349    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2d4186d-qnxtn\" (UID: \"75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn"
Nov 03 18:54:41.056435 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:41.040494    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2d4186d-qnxtn\" (UID: \"75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn"
Nov 03 18:54:41.056435 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:41.040581    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rd9x\" (UniqueName: \"kubernetes.io/projected/75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651-kube-api-access-4rd9x\") pod \"maas-api-revoke-sub-f2d4186d-qnxtn\" (UID: \"75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn"
Nov 03 18:54:41.144389 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:41.143124    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rd9x\" (UniqueName: \"kubernetes.io/projected/75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651-kube-api-access-4rd9x\") pod \"maas-api-revoke-sub-f2d4186d-qnxtn\" (UID: \"75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn"
Nov 03 18:54:41.144389 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:41.143282    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2d4186d-qnxtn\" (UID: \"75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn"
Nov 03 18:54:41.144389 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:41.143342    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2d4186d-qnxtn\" (UID: \"75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn"
Nov 03 18:54:41.144722 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:41.144653    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2d4186d-qnxtn\" (UID: \"75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn"
Nov 03 18:54:41.181014 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:41.180973    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2d4186d-qnxtn\" (UID: \"75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn"
Nov 03 18:54:41.226651 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:41.226619    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rd9x\" (UniqueName: \"kubernetes.io/projected/75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651-kube-api-access-4rd9x\") pod \"maas-api-revoke-sub-f2d4186d-qnxtn\" (UID: \"75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn"
Nov 03 18:54:41.417724 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:41.417604    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn"
Nov 03 18:54:42.075010 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:42.074897    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn"]
Nov 03 18:54:43.035795 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:43.035760    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn" event={"ID":"75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651","Type":"ContainerStarted","Data":"bf8fd4e5ed88f8bd63b615b77079a5a1ca43cd95b5fe51f85de19dafc1628430"}
Nov 03 18:54:43.035991 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:43.035813    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn" event={"ID":"75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651","Type":"ContainerStarted","Data":"dede42059b8089049286e9c629297457c53a87d87dc42da28a3218aacc2926a2"}
Nov 03 18:54:43.182234 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:43.182177    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn" podStartSLOduration=3.182165861 podStartE2EDuration="3.182165861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:54:43.180663886 +0000 UTC m=+5823.978376668" watchObservedRunningTime="2026-11-03 18:54:43.182165861 +0000 UTC m=+5823.979878639"
Nov 03 18:54:44.045393 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:44.045333    6059 generic.go:356] "Generic (PLEG): container finished" podID="75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651" containerID="bf8fd4e5ed88f8bd63b615b77079a5a1ca43cd95b5fe51f85de19dafc1628430" exitCode=0
Nov 03 18:54:44.045393 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:44.045386    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn" event={"ID":"75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651","Type":"ContainerDied","Data":"bf8fd4e5ed88f8bd63b615b77079a5a1ca43cd95b5fe51f85de19dafc1628430"}
Nov 03 18:54:44.261224 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:44.261192    6059 scope.go:122] "RemoveContainer" containerID="a275bd2ae7a4e940f577ed55c6296fa0ec97d8114b34d30478b053a8d585dc8e"
Nov 03 18:54:44.270228 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:44.270197    6059 scope.go:122] "RemoveContainer" containerID="b1f5796476a02f7d962bc79728d947d917025e065a0df67f58c0dbf72148d8b7"
Nov 03 18:54:44.278336 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:44.278315    6059 scope.go:122] "RemoveContainer" containerID="a68730cf06d9778bfd8d3aafda78e49f3d1519891b130da6ec9c1445d79a40d8"
Nov 03 18:54:45.187585 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:45.187557    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn"
Nov 03 18:54:45.247510 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:45.247478    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651-maas-api-openshift-ca\") pod \"75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651\" (UID: \"75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651\") "
Nov 03 18:54:45.247738 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:45.247572    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651-maas-api-xks-ca\") pod \"75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651\" (UID: \"75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651\") "
Nov 03 18:54:45.247738 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:45.247639    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651-kube-api-access-4rd9x\" (UniqueName: \"kubernetes.io/projected/75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651-kube-api-access-4rd9x\") pod \"75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651\" (UID: \"75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651\") "
Nov 03 18:54:45.247905 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:45.247809    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651-maas-api-openshift-ca" pod "75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651" (UID: "75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:54:45.248030 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:45.248013    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:54:45.249748 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:45.249717    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651-maas-api-xks-ca" pod "75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651" (UID: "75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:54:45.249845 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:45.249755    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651-kube-api-access-4rd9x" pod "75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651" (UID: "75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651"). InnerVolumeSpecName "kube-api-access-4rd9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:54:45.348732 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:45.348694    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:54:45.348732 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:45.348724    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rd9x\" (UniqueName: \"kubernetes.io/projected/75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651-kube-api-access-4rd9x\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:54:46.059419 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:46.059389    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn"
Nov 03 18:54:46.059419 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:46.059404    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn" event={"ID":"75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651","Type":"ContainerDied","Data":"dede42059b8089049286e9c629297457c53a87d87dc42da28a3218aacc2926a2"}
Nov 03 18:54:46.059656 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:46.059460    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dede42059b8089049286e9c629297457c53a87d87dc42da28a3218aacc2926a2"
Nov 03 18:54:51.426864 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:51.426829    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:54:57.523431 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:54:57.523394    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:55:03.961991 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:55:03.961950    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:55:06.860591 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:55:06.860549    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:55:36.860765 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:55:36.860683    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:55:42.937732 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:55:42.937696    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:55:43.111754 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:55:43.111712    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:55:44.306999 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:55:44.306969    6059 scope.go:122] "RemoveContainer" containerID="684d73f6d6a6a3bbe2e4a10fb36932f370b3aca84c85b2b33e6b189d8f371419"
Nov 03 18:55:44.315758 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:55:44.315736    6059 scope.go:122] "RemoveContainer" containerID="26c8ed6d56198395233fa0c78b58499bd268c1bb88a5557b87d3575ef2ed157a"
Nov 03 18:55:44.327114 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:55:44.327087    6059 scope.go:122] "RemoveContainer" containerID="67d8e3301db178dfe350156b3fbb049adcc187f30ef1fc27d6815b05882a1cc8"
Nov 03 18:56:04.325210 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:56:04.325171    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:56:06.859991 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:56:06.859951    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:56:11.047913 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:56:11.047878    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls"]
Nov 03 18:56:11.095088 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:56:11.095056    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-013a378c-g9hls"]
Nov 03 18:56:11.334889 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:56:11.334804    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e799c8c-1dea-418d-baa4-73c655cfaa00" path="/var/lib/kubelet/pods/7e799c8c-1dea-418d-baa4-73c655cfaa00/volumes"
Nov 03 18:56:21.914809 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:56:21.914770    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:56:22.104655 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:56:22.104625    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:56:22.943521 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:56:22.943483    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:56:36.862425 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:56:36.862388    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:56:49.153805 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:56:49.153766    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:56:55.040426 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:56:55.040386    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:57:04.660118 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:57:04.660084    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:57:06.861024 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:57:06.860983    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:57:08.046287 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:57:08.046254    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn"]
Nov 03 18:57:08.093552 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:57:08.093518    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mn7cn"]
Nov 03 18:57:09.336266 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:57:09.336231    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12b4c264-b6d4-486c-b2e1-30714f0742d5" path="/var/lib/kubelet/pods/12b4c264-b6d4-486c-b2e1-30714f0742d5/volumes"
Nov 03 18:57:36.861831 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:57:36.861780    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:57:40.243762 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:57:40.243733    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:57:40.261799 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:57:40.261768    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:57:40.266840 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:57:40.266819    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 18:57:40.283065 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:57:40.283040    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 18:57:44.366894 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:57:44.366860    6059 scope.go:122] "RemoveContainer" containerID="873771e53fc11890ab934110ea6a1d89209fc08b538bd43756f251525db744d7"
Nov 03 18:58:05.009440 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:58:05.009405    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:06.860718 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:58:06.860677    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:58:19.636479 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:58:19.636398    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:19.825767 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:58:19.825737    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:36.398188 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:58:36.398147    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:36.860592 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:58:36.860551    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:58:42.332828 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:58:42.332772    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:44.391322 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:58:44.391288    6059 scope.go:122] "RemoveContainer" containerID="30ff0a4c035610200b757afbd0e92d246aad7f46a850eb8ca906acf4205869ff"
Nov 03 18:58:58.059494 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:58:58.059459    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-f2smj"]
Nov 03 18:58:58.106744 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:58:58.106710    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-f2smj"]
Nov 03 18:58:59.334934 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:58:59.334898    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="117b3fb5-968b-46a5-9c3c-c6155c68ccf5" path="/var/lib/kubelet/pods/117b3fb5-968b-46a5-9c3c-c6155c68ccf5/volumes"
Nov 03 18:59:05.411799 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:59:05.411764    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:06.862901 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:59:06.862868    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:59:20.050019 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:59:20.049985    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9jpj5"]
Nov 03 18:59:20.097410 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:59:20.097377    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9jpj5"]
Nov 03 18:59:21.334521 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:59:21.334482    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9233790f-7d17-42c8-9d8d-5c3841b4af01" path="/var/lib/kubelet/pods/9233790f-7d17-42c8-9d8d-5c3841b4af01/volumes"
Nov 03 18:59:36.859931 ip-10-0-143-117 kubenswrapper[6059]: E1103 18:59:36.859892    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 18:59:46.045544 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:59:46.045511    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn"]
Nov 03 18:59:46.092205 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:59:46.092175    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2d4186d-qnxtn"]
Nov 03 18:59:47.334702 ip-10-0-143-117 kubenswrapper[6059]: I1103 18:59:47.334616    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651" path="/var/lib/kubelet/pods/75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651/volumes"
Nov 03 19:00:00.175948 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:00.175911    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4"]
Nov 03 19:00:00.176381 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:00.176348    6059 cpu_manager.go:414] "containerMap: removing container" podUID="75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651" containerName="revoke-keys"
Nov 03 19:00:00.176381 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:00.176376    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651" containerName="revoke-keys"
Nov 03 19:00:00.176491 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:00.176480    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="75dd6f56-c2e4-4e94-aafe-e9f7aa8b5651" containerName="revoke-keys"
Nov 03 19:00:00.179809 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:00.179788    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4"
Nov 03 19:00:00.230179 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:00.230149    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:00:00.277396 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:00.277366    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:00:00.277396 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:00.277366    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:00:00.279501 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:00.279479    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4"]
Nov 03 19:00:00.297505 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:00.297475    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zkp5\" (UniqueName: \"kubernetes.io/projected/b588cba4-4e07-4d1e-af1c-013b416aba9e-kube-api-access-7zkp5\") pod \"maas-api-key-cleanup-29895540-4xmv4\" (UID: \"b588cba4-4e07-4d1e-af1c-013b416aba9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4"
Nov 03 19:00:00.297641 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:00.297548    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b588cba4-4e07-4d1e-af1c-013b416aba9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895540-4xmv4\" (UID: \"b588cba4-4e07-4d1e-af1c-013b416aba9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4"
Nov 03 19:00:00.398558 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:00.398529    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b588cba4-4e07-4d1e-af1c-013b416aba9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895540-4xmv4\" (UID: \"b588cba4-4e07-4d1e-af1c-013b416aba9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4"
Nov 03 19:00:00.398718 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:00.398639    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zkp5\" (UniqueName: \"kubernetes.io/projected/b588cba4-4e07-4d1e-af1c-013b416aba9e-kube-api-access-7zkp5\") pod \"maas-api-key-cleanup-29895540-4xmv4\" (UID: \"b588cba4-4e07-4d1e-af1c-013b416aba9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4"
Nov 03 19:00:00.399211 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:00.399192    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b588cba4-4e07-4d1e-af1c-013b416aba9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895540-4xmv4\" (UID: \"b588cba4-4e07-4d1e-af1c-013b416aba9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4"
Nov 03 19:00:00.457601 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:00.457572    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zkp5\" (UniqueName: \"kubernetes.io/projected/b588cba4-4e07-4d1e-af1c-013b416aba9e-kube-api-access-7zkp5\") pod \"maas-api-key-cleanup-29895540-4xmv4\" (UID: \"b588cba4-4e07-4d1e-af1c-013b416aba9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4"
Nov 03 19:00:00.492773 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:00.492747    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4"
Nov 03 19:00:00.865158 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:00.865132    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4"]
Nov 03 19:00:00.867948 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:00:00.867918    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb588cba4_4e07_4d1e_af1c_013b416aba9e.slice/crio-655e46305ceb4a6697ec2319effb0affd5702980b5aae23a0a8f0e62d70f78c2 WatchSource:0}: Error finding container 655e46305ceb4a6697ec2319effb0affd5702980b5aae23a0a8f0e62d70f78c2: Status 404 returned error can't find the container with id 655e46305ceb4a6697ec2319effb0affd5702980b5aae23a0a8f0e62d70f78c2
Nov 03 19:00:01.767677 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:01.767641    6059 generic.go:356] "Generic (PLEG): container finished" podID="b588cba4-4e07-4d1e-af1c-013b416aba9e" containerID="c647431c83a69f283b25e86db348ef6a85ee5f76e30528c9b5a3c3a9f994b656" exitCode=60
Nov 03 19:00:01.768088 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:01.767721    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4" event={"ID":"b588cba4-4e07-4d1e-af1c-013b416aba9e","Type":"ContainerDied","Data":"c647431c83a69f283b25e86db348ef6a85ee5f76e30528c9b5a3c3a9f994b656"}
Nov 03 19:00:01.768088 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:01.767770    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4" event={"ID":"b588cba4-4e07-4d1e-af1c-013b416aba9e","Type":"ContainerStarted","Data":"655e46305ceb4a6697ec2319effb0affd5702980b5aae23a0a8f0e62d70f78c2"}
Nov 03 19:00:01.768088 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:01.767940    6059 scope.go:122] "RemoveContainer" containerID="c647431c83a69f283b25e86db348ef6a85ee5f76e30528c9b5a3c3a9f994b656"
Nov 03 19:00:02.775522 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:02.775487    6059 generic.go:356] "Generic (PLEG): container finished" podID="b588cba4-4e07-4d1e-af1c-013b416aba9e" containerID="42c5fa1984ad41d917f59357c91824dd6ea780952366d0c014e50170587aee54" exitCode=60
Nov 03 19:00:02.775942 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:02.775534    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4" event={"ID":"b588cba4-4e07-4d1e-af1c-013b416aba9e","Type":"ContainerDied","Data":"42c5fa1984ad41d917f59357c91824dd6ea780952366d0c014e50170587aee54"}
Nov 03 19:00:02.775942 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:02.775583    6059 scope.go:122] "RemoveContainer" containerID="c647431c83a69f283b25e86db348ef6a85ee5f76e30528c9b5a3c3a9f994b656"
Nov 03 19:00:02.775942 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:02.775840    6059 scope.go:122] "RemoveContainer" containerID="42c5fa1984ad41d917f59357c91824dd6ea780952366d0c014e50170587aee54"
Nov 03 19:00:02.776124 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:00:02.776101    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895540-4xmv4_odh-ai-gateway-infra(b588cba4-4e07-4d1e-af1c-013b416aba9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4" podUID="b588cba4-4e07-4d1e-af1c-013b416aba9e"
Nov 03 19:00:03.782786 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:03.782758    6059 scope.go:122] "RemoveContainer" containerID="42c5fa1984ad41d917f59357c91824dd6ea780952366d0c014e50170587aee54"
Nov 03 19:00:03.783177 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:00:03.782989    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895540-4xmv4_odh-ai-gateway-infra(b588cba4-4e07-4d1e-af1c-013b416aba9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4" podUID="b588cba4-4e07-4d1e-af1c-013b416aba9e"
Nov 03 19:00:05.698378 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:05.698326    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:00:06.860260 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:00:06.860219    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:00:07.179575 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:07.179491    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:00:14.329566 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:14.329535    6059 scope.go:122] "RemoveContainer" containerID="42c5fa1984ad41d917f59357c91824dd6ea780952366d0c014e50170587aee54"
Nov 03 19:00:14.842566 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:14.842531    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4" event={"ID":"b588cba4-4e07-4d1e-af1c-013b416aba9e","Type":"ContainerStarted","Data":"045dc7a4d4a49eaeca0ba7ac43ca47c378852cfdaf805fd96669ccfa1c3f8283"}
Nov 03 19:00:14.955347 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:14.955251    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4" podStartSLOduration=14.955238765 podStartE2EDuration="14.955238765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:14.954660681 +0000 UTC m=+6155.752373462" watchObservedRunningTime="2026-11-03 19:00:14.955238765 +0000 UTC m=+6155.752951549"
Nov 03 19:00:15.849431 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:15.849395    6059 generic.go:356] "Generic (PLEG): container finished" podID="b588cba4-4e07-4d1e-af1c-013b416aba9e" containerID="045dc7a4d4a49eaeca0ba7ac43ca47c378852cfdaf805fd96669ccfa1c3f8283" exitCode=60
Nov 03 19:00:15.849910 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:15.849466    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4" event={"ID":"b588cba4-4e07-4d1e-af1c-013b416aba9e","Type":"ContainerDied","Data":"045dc7a4d4a49eaeca0ba7ac43ca47c378852cfdaf805fd96669ccfa1c3f8283"}
Nov 03 19:00:15.849910 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:15.849526    6059 scope.go:122] "RemoveContainer" containerID="42c5fa1984ad41d917f59357c91824dd6ea780952366d0c014e50170587aee54"
Nov 03 19:00:15.849910 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:15.849812    6059 scope.go:122] "RemoveContainer" containerID="045dc7a4d4a49eaeca0ba7ac43ca47c378852cfdaf805fd96669ccfa1c3f8283"
Nov 03 19:00:15.850086 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:00:15.850058    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895540-4xmv4_odh-ai-gateway-infra(b588cba4-4e07-4d1e-af1c-013b416aba9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4" podUID="b588cba4-4e07-4d1e-af1c-013b416aba9e"
Nov 03 19:00:15.977201 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:15.977164    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4"]
Nov 03 19:00:16.998620 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:16.998593    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4"
Nov 03 19:00:17.168956 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:17.168841    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b588cba4-4e07-4d1e-af1c-013b416aba9e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b588cba4-4e07-4d1e-af1c-013b416aba9e-maas-api-service-ca\") pod \"b588cba4-4e07-4d1e-af1c-013b416aba9e\" (UID: \"b588cba4-4e07-4d1e-af1c-013b416aba9e\") "
Nov 03 19:00:17.169173 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:17.169047    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b588cba4-4e07-4d1e-af1c-013b416aba9e-kube-api-access-7zkp5\" (UniqueName: \"kubernetes.io/projected/b588cba4-4e07-4d1e-af1c-013b416aba9e-kube-api-access-7zkp5\") pod \"b588cba4-4e07-4d1e-af1c-013b416aba9e\" (UID: \"b588cba4-4e07-4d1e-af1c-013b416aba9e\") "
Nov 03 19:00:17.169263 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:17.169188    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b588cba4-4e07-4d1e-af1c-013b416aba9e-maas-api-service-ca" pod "b588cba4-4e07-4d1e-af1c-013b416aba9e" (UID: "b588cba4-4e07-4d1e-af1c-013b416aba9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:00:17.169393 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:17.169349    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b588cba4-4e07-4d1e-af1c-013b416aba9e-maas-api-service-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:17.171052 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:17.171027    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b588cba4-4e07-4d1e-af1c-013b416aba9e-kube-api-access-7zkp5" pod "b588cba4-4e07-4d1e-af1c-013b416aba9e" (UID: "b588cba4-4e07-4d1e-af1c-013b416aba9e"). InnerVolumeSpecName "kube-api-access-7zkp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:00:17.270909 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:17.270877    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zkp5\" (UniqueName: \"kubernetes.io/projected/b588cba4-4e07-4d1e-af1c-013b416aba9e-kube-api-access-7zkp5\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:17.719479 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:17.719444    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:00:17.725477 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:17.725446    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"]
Nov 03 19:00:17.725657 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:17.725634    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" podUID="2159e743-a61b-4d76-9f78-9be36e825a43" containerName="istio-proxy" containerID="cri-o://a534099f4dffc13956221f54dfeab90e33cc6c87c30f1bbc8f67ee27d2c90ba1" gracePeriod=30
Nov 03 19:00:17.773094 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:17.773058    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"]
Nov 03 19:00:17.773374 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:17.773331    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn" podUID="d9e3abf7-c5e1-4414-858e-5f84f6a88670" containerName="istio-proxy" containerID="cri-o://0366d3184102a5202ba42d897e05e9449e49c80f0ee47f0dc5642c67db4b9488" gracePeriod=30
Nov 03 19:00:17.774919 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:17.774893    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"]
Nov 03 19:00:17.775093 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:17.775072    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7" podUID="60b66cb6-2bce-4c02-aa7d-7eb25f29f237" containerName="istio-proxy" containerID="cri-o://1fe6f7c31c90141bd1c0146db77a5e6a28c3f16968b2cd1cf074798481f20a93" gracePeriod=30
Nov 03 19:00:17.820348 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:17.820314    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"]
Nov 03 19:00:17.820542 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:17.820522    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" podUID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" containerName="istio-proxy" containerID="cri-o://7b05d327a1d2a420fd123ff5f8ea128b8306222325363ab777c69fe2bca1eb9f" gracePeriod=30
Nov 03 19:00:17.863655 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:17.863607    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4" event={"ID":"b588cba4-4e07-4d1e-af1c-013b416aba9e","Type":"ContainerDied","Data":"655e46305ceb4a6697ec2319effb0affd5702980b5aae23a0a8f0e62d70f78c2"}
Nov 03 19:00:17.863655 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:17.863639    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4"
Nov 03 19:00:17.863900 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:17.863693    6059 scope.go:122] "RemoveContainer" containerID="045dc7a4d4a49eaeca0ba7ac43ca47c378852cfdaf805fd96669ccfa1c3f8283"
Nov 03 19:00:18.121295 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:18.121198    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4"]
Nov 03 19:00:18.127084 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:18.127059    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895540-4xmv4"]
Nov 03 19:00:19.335232 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:19.335197    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b588cba4-4e07-4d1e-af1c-013b416aba9e" path="/var/lib/kubelet/pods/b588cba4-4e07-4d1e-af1c-013b416aba9e/volumes"
Nov 03 19:00:21.121280 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:21.121247    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-l2xpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:00:21.121724 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:21.121298    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" podUID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:00:21.231033 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:21.230990    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-gxxkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:00:21.231222 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:21.231060    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" podUID="2159e743-a61b-4d76-9f78-9be36e825a43" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:00:22.892303 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:22.892271    6059 generic.go:356] "Generic (PLEG): container finished" podID="60b66cb6-2bce-4c02-aa7d-7eb25f29f237" containerID="1fe6f7c31c90141bd1c0146db77a5e6a28c3f16968b2cd1cf074798481f20a93" exitCode=0
Nov 03 19:00:22.892761 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:22.892340    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7" event={"ID":"60b66cb6-2bce-4c02-aa7d-7eb25f29f237","Type":"ContainerDied","Data":"1fe6f7c31c90141bd1c0146db77a5e6a28c3f16968b2cd1cf074798481f20a93"}
Nov 03 19:00:22.894415 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:22.894387    6059 generic.go:356] "Generic (PLEG): container finished" podID="d9e3abf7-c5e1-4414-858e-5f84f6a88670" containerID="0366d3184102a5202ba42d897e05e9449e49c80f0ee47f0dc5642c67db4b9488" exitCode=0
Nov 03 19:00:22.894539 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:22.894463    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn" event={"ID":"d9e3abf7-c5e1-4414-858e-5f84f6a88670","Type":"ContainerDied","Data":"0366d3184102a5202ba42d897e05e9449e49c80f0ee47f0dc5642c67db4b9488"}
Nov 03 19:00:22.897107 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:22.897084    6059 generic.go:356] "Generic (PLEG): container finished" podID="2159e743-a61b-4d76-9f78-9be36e825a43" containerID="a534099f4dffc13956221f54dfeab90e33cc6c87c30f1bbc8f67ee27d2c90ba1" exitCode=0
Nov 03 19:00:22.897210 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:22.897158    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" event={"ID":"2159e743-a61b-4d76-9f78-9be36e825a43","Type":"ContainerDied","Data":"a534099f4dffc13956221f54dfeab90e33cc6c87c30f1bbc8f67ee27d2c90ba1"}
Nov 03 19:00:23.058706 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.058680    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 19:00:23.090456 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.090433    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 19:00:23.142801 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.142776    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 19:00:23.152036 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.152015    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 19:00:23.231037 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231008    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-workload-certs\") pod \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") "
Nov 03 19:00:23.231228 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231065    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-workload-certs\") pod \"2159e743-a61b-4d76-9f78-9be36e825a43\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") "
Nov 03 19:00:23.231228 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231101    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-podinfo\") pod \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") "
Nov 03 19:00:23.231228 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231131    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-envoy\") pod \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") "
Nov 03 19:00:23.231228 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231158    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2159e743-a61b-4d76-9f78-9be36e825a43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2159e743-a61b-4d76-9f78-9be36e825a43-istio-podinfo\") pod \"2159e743-a61b-4d76-9f78-9be36e825a43\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") "
Nov 03 19:00:23.231228 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231199    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-data\") pod \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") "
Nov 03 19:00:23.231568 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231346    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-workload-socket\") pod \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") "
Nov 03 19:00:23.231568 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231406    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istiod-ca-cert\") pod \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") "
Nov 03 19:00:23.231568 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231452    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-credential-socket\") pod \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") "
Nov 03 19:00:23.231568 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231451    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-workload-certs" pod "2159e743-a61b-4d76-9f78-9be36e825a43" (UID: "2159e743-a61b-4d76-9f78-9be36e825a43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:00:23.231568 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231499    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-token\" (UniqueName: \"kubernetes.io/projected/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-token\") pod \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") "
Nov 03 19:00:23.231568 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231525    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2159e743-a61b-4d76-9f78-9be36e825a43-istio-token\" (UniqueName: \"kubernetes.io/projected/2159e743-a61b-4d76-9f78-9be36e825a43-istio-token\") pod \"2159e743-a61b-4d76-9f78-9be36e825a43\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") "
Nov 03 19:00:23.231568 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231553    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-workload-socket\") pod \"2159e743-a61b-4d76-9f78-9be36e825a43\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") "
Nov 03 19:00:23.231965 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231574    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2159e743-a61b-4d76-9f78-9be36e825a43-kube-api-access-wsbjh\" (UniqueName: \"kubernetes.io/projected/2159e743-a61b-4d76-9f78-9be36e825a43-kube-api-access-wsbjh\") pod \"2159e743-a61b-4d76-9f78-9be36e825a43\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") "
Nov 03 19:00:23.231965 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231602    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-istio-data\") pod \"2159e743-a61b-4d76-9f78-9be36e825a43\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") "
Nov 03 19:00:23.231965 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231628    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e3abf7-c5e1-4414-858e-5f84f6a88670-kube-api-access-8s4jj\" (UniqueName: \"kubernetes.io/projected/d9e3abf7-c5e1-4414-858e-5f84f6a88670-kube-api-access-8s4jj\") pod \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\" (UID: \"d9e3abf7-c5e1-4414-858e-5f84f6a88670\") "
Nov 03 19:00:23.231965 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231660    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-istio-envoy\") pod \"2159e743-a61b-4d76-9f78-9be36e825a43\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") "
Nov 03 19:00:23.231965 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231687    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-credential-socket\") pod \"2159e743-a61b-4d76-9f78-9be36e825a43\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") "
Nov 03 19:00:23.231965 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231684    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-workload-socket" pod "d9e3abf7-c5e1-4414-858e-5f84f6a88670" (UID: "d9e3abf7-c5e1-4414-858e-5f84f6a88670"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:00:23.231965 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231731    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2159e743-a61b-4d76-9f78-9be36e825a43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2159e743-a61b-4d76-9f78-9be36e825a43-istiod-ca-cert\") pod \"2159e743-a61b-4d76-9f78-9be36e825a43\" (UID: \"2159e743-a61b-4d76-9f78-9be36e825a43\") "
Nov 03 19:00:23.231965 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231805    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istiod-ca-cert" pod "d9e3abf7-c5e1-4414-858e-5f84f6a88670" (UID: "d9e3abf7-c5e1-4414-858e-5f84f6a88670"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:00:23.231965 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231806    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-data" pod "d9e3abf7-c5e1-4414-858e-5f84f6a88670" (UID: "d9e3abf7-c5e1-4414-858e-5f84f6a88670"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:00:23.231965 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231917    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-workload-socket" pod "2159e743-a61b-4d76-9f78-9be36e825a43" (UID: "2159e743-a61b-4d76-9f78-9be36e825a43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:00:23.232584 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.232193    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-credential-socket" pod "d9e3abf7-c5e1-4414-858e-5f84f6a88670" (UID: "d9e3abf7-c5e1-4414-858e-5f84f6a88670"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:00:23.232584 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.232248    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.232584 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.232271    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.232584 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.232287    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.232584 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.232300    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.232584 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.232313    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.232584 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.232248    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2159e743-a61b-4d76-9f78-9be36e825a43-istiod-ca-cert" pod "2159e743-a61b-4d76-9f78-9be36e825a43" (UID: "2159e743-a61b-4d76-9f78-9be36e825a43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:00:23.232584 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.231350    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-workload-certs" pod "d9e3abf7-c5e1-4414-858e-5f84f6a88670" (UID: "d9e3abf7-c5e1-4414-858e-5f84f6a88670"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:00:23.233034 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.232681    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-credential-socket" pod "2159e743-a61b-4d76-9f78-9be36e825a43" (UID: "2159e743-a61b-4d76-9f78-9be36e825a43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:00:23.233294 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.233270    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-istio-data" pod "2159e743-a61b-4d76-9f78-9be36e825a43" (UID: "2159e743-a61b-4d76-9f78-9be36e825a43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:00:23.234181 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.234154    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-podinfo" pod "d9e3abf7-c5e1-4414-858e-5f84f6a88670" (UID: "d9e3abf7-c5e1-4414-858e-5f84f6a88670"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:00:23.234338 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.234313    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2159e743-a61b-4d76-9f78-9be36e825a43-istio-podinfo" pod "2159e743-a61b-4d76-9f78-9be36e825a43" (UID: "2159e743-a61b-4d76-9f78-9be36e825a43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:00:23.234455 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.234440    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-envoy" pod "d9e3abf7-c5e1-4414-858e-5f84f6a88670" (UID: "d9e3abf7-c5e1-4414-858e-5f84f6a88670"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:00:23.234539 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.234519    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-token" pod "d9e3abf7-c5e1-4414-858e-5f84f6a88670" (UID: "d9e3abf7-c5e1-4414-858e-5f84f6a88670"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:00:23.234663 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.234637    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e3abf7-c5e1-4414-858e-5f84f6a88670-kube-api-access-8s4jj" pod "d9e3abf7-c5e1-4414-858e-5f84f6a88670" (UID: "d9e3abf7-c5e1-4414-858e-5f84f6a88670"). InnerVolumeSpecName "kube-api-access-8s4jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:00:23.234774 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.234681    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2159e743-a61b-4d76-9f78-9be36e825a43-kube-api-access-wsbjh" pod "2159e743-a61b-4d76-9f78-9be36e825a43" (UID: "2159e743-a61b-4d76-9f78-9be36e825a43"). InnerVolumeSpecName "kube-api-access-wsbjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:00:23.235101 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.235084    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2159e743-a61b-4d76-9f78-9be36e825a43-istio-token" pod "2159e743-a61b-4d76-9f78-9be36e825a43" (UID: "2159e743-a61b-4d76-9f78-9be36e825a43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:00:23.235269 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.235255    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-istio-envoy" pod "2159e743-a61b-4d76-9f78-9be36e825a43" (UID: "2159e743-a61b-4d76-9f78-9be36e825a43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.333229    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-workload-certs\") pod \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") "
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.333282    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istiod-ca-cert\") pod \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") "
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.333317    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-credential-socket\") pod \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") "
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.333371    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb79e59-86dc-47d1-830f-49e542a5b8f3-kube-api-access-srnp4\" (UniqueName: \"kubernetes.io/projected/5eb79e59-86dc-47d1-830f-49e542a5b8f3-kube-api-access-srnp4\") pod \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") "
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.333418    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-workload-certs\") pod \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") "
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.333451    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-token\" (UniqueName: \"kubernetes.io/projected/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-token\") pod \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") "
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.333532    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-podinfo\") pod \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") "
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.333575    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-workload-socket\") pod \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") "
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.333599    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-data\") pod \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") "
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.333622    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-podinfo\") pod \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") "
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.333642    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-workload-certs" pod "5eb79e59-86dc-47d1-830f-49e542a5b8f3" (UID: "5eb79e59-86dc-47d1-830f-49e542a5b8f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.333650    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istiod-ca-cert\") pod \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") "
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.333721    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-kube-api-access-bcj2t\" (UniqueName: \"kubernetes.io/projected/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-kube-api-access-bcj2t\") pod \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") "
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.333760    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-token\" (UniqueName: \"kubernetes.io/projected/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-token\") pod \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") "
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.333770    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-credential-socket" pod "5eb79e59-86dc-47d1-830f-49e542a5b8f3" (UID: "5eb79e59-86dc-47d1-830f-49e542a5b8f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.333789    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-envoy\") pod \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") "
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.333892    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-credential-socket\") pod \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") "
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.333928    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-workload-socket\") pod \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") "
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.333953    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istiod-ca-cert" pod "5eb79e59-86dc-47d1-830f-49e542a5b8f3" (UID: "5eb79e59-86dc-47d1-830f-49e542a5b8f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.333980    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-envoy\") pod \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\" (UID: \"5eb79e59-86dc-47d1-830f-49e542a5b8f3\") "
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.334018    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-data\") pod \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\" (UID: \"60b66cb6-2bce-4c02-aa7d-7eb25f29f237\") "
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.334456    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.334479    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.334496    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.334511    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.334527    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2159e743-a61b-4d76-9f78-9be36e825a43-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.334541    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.334555    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e3abf7-c5e1-4414-858e-5f84f6a88670-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.334569    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2159e743-a61b-4d76-9f78-9be36e825a43-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.334586    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsbjh\" (UniqueName: \"kubernetes.io/projected/2159e743-a61b-4d76-9f78-9be36e825a43-kube-api-access-wsbjh\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.334604    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.334618    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s4jj\" (UniqueName: \"kubernetes.io/projected/d9e3abf7-c5e1-4414-858e-5f84f6a88670-kube-api-access-8s4jj\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.334635    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.334651    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2159e743-a61b-4d76-9f78-9be36e825a43-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.334666    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2159e743-a61b-4d76-9f78-9be36e825a43-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.334822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.334681    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e3abf7-c5e1-4414-858e-5f84f6a88670-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.339866 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.334903    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-data" pod "60b66cb6-2bce-4c02-aa7d-7eb25f29f237" (UID: "60b66cb6-2bce-4c02-aa7d-7eb25f29f237"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:00:23.339866 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.335516    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istiod-ca-cert" pod "60b66cb6-2bce-4c02-aa7d-7eb25f29f237" (UID: "60b66cb6-2bce-4c02-aa7d-7eb25f29f237"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:00:23.339866 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.336320    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-envoy" pod "60b66cb6-2bce-4c02-aa7d-7eb25f29f237" (UID: "60b66cb6-2bce-4c02-aa7d-7eb25f29f237"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:00:23.339866 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.336467    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-workload-socket" pod "5eb79e59-86dc-47d1-830f-49e542a5b8f3" (UID: "5eb79e59-86dc-47d1-830f-49e542a5b8f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:00:23.339866 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.336601    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-data" pod "5eb79e59-86dc-47d1-830f-49e542a5b8f3" (UID: "5eb79e59-86dc-47d1-830f-49e542a5b8f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:00:23.339866 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.336481    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-workload-socket" pod "60b66cb6-2bce-4c02-aa7d-7eb25f29f237" (UID: "60b66cb6-2bce-4c02-aa7d-7eb25f29f237"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:00:23.339866 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.336743    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-credential-socket" pod "60b66cb6-2bce-4c02-aa7d-7eb25f29f237" (UID: "60b66cb6-2bce-4c02-aa7d-7eb25f29f237"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:00:23.339866 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.337111    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb79e59-86dc-47d1-830f-49e542a5b8f3-kube-api-access-srnp4" pod "5eb79e59-86dc-47d1-830f-49e542a5b8f3" (UID: "5eb79e59-86dc-47d1-830f-49e542a5b8f3"). InnerVolumeSpecName "kube-api-access-srnp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:00:23.339866 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.337145    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-workload-certs" pod "60b66cb6-2bce-4c02-aa7d-7eb25f29f237" (UID: "60b66cb6-2bce-4c02-aa7d-7eb25f29f237"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:00:23.339866 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.338977    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-token" pod "60b66cb6-2bce-4c02-aa7d-7eb25f29f237" (UID: "60b66cb6-2bce-4c02-aa7d-7eb25f29f237"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:00:23.340554 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.340531    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-podinfo" pod "5eb79e59-86dc-47d1-830f-49e542a5b8f3" (UID: "5eb79e59-86dc-47d1-830f-49e542a5b8f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:00:23.343520 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.343494    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-token" pod "5eb79e59-86dc-47d1-830f-49e542a5b8f3" (UID: "5eb79e59-86dc-47d1-830f-49e542a5b8f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:00:23.343778 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.343736    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-podinfo" pod "60b66cb6-2bce-4c02-aa7d-7eb25f29f237" (UID: "60b66cb6-2bce-4c02-aa7d-7eb25f29f237"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:00:23.346739 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.346710    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-envoy" pod "5eb79e59-86dc-47d1-830f-49e542a5b8f3" (UID: "5eb79e59-86dc-47d1-830f-49e542a5b8f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:00:23.346850 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.346765    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-kube-api-access-bcj2t" pod "60b66cb6-2bce-4c02-aa7d-7eb25f29f237" (UID: "60b66cb6-2bce-4c02-aa7d-7eb25f29f237"). InnerVolumeSpecName "kube-api-access-bcj2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:00:23.435741 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.435710    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.435741 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.435737    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.435741 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.435748    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.435997 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.435758    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.435997 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.435772    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.435997 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.435783    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcj2t\" (UniqueName: \"kubernetes.io/projected/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-kube-api-access-bcj2t\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.435997 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.435792    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.435997 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.435801    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.435997 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.435809    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.435997 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.435818    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.435997 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.435827    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.435997 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.435837    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.435997 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.435850    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.435997 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.435860    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srnp4\" (UniqueName: \"kubernetes.io/projected/5eb79e59-86dc-47d1-830f-49e542a5b8f3-kube-api-access-srnp4\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.435997 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.435871    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60b66cb6-2bce-4c02-aa7d-7eb25f29f237-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.435997 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.435879    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb79e59-86dc-47d1-830f-49e542a5b8f3-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.904380 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.904332    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"
Nov 03 19:00:23.904380 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.904347    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7" event={"ID":"60b66cb6-2bce-4c02-aa7d-7eb25f29f237","Type":"ContainerDied","Data":"76282666b7f0611700c96cebfff43395bcfc0cd5e4b75558854a4fc8b41db81d"}
Nov 03 19:00:23.904877 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.904437    6059 scope.go:122] "RemoveContainer" containerID="1fe6f7c31c90141bd1c0146db77a5e6a28c3f16968b2cd1cf074798481f20a93"
Nov 03 19:00:23.905804 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.905791    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"
Nov 03 19:00:23.905898 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.905828    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn" event={"ID":"d9e3abf7-c5e1-4414-858e-5f84f6a88670","Type":"ContainerDied","Data":"ff090bfc8d193186df986f99e3606860050f4f323eb5475de0b1ece095f02b29"}
Nov 03 19:00:23.907397 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.907348    6059 generic.go:356] "Generic (PLEG): container finished" podID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" containerID="7b05d327a1d2a420fd123ff5f8ea128b8306222325363ab777c69fe2bca1eb9f" exitCode=0
Nov 03 19:00:23.907505 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.907413    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"
Nov 03 19:00:23.907505 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.907447    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" event={"ID":"5eb79e59-86dc-47d1-830f-49e542a5b8f3","Type":"ContainerDied","Data":"7b05d327a1d2a420fd123ff5f8ea128b8306222325363ab777c69fe2bca1eb9f"}
Nov 03 19:00:23.907505 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.907483    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl" event={"ID":"5eb79e59-86dc-47d1-830f-49e542a5b8f3","Type":"ContainerDied","Data":"9f5328318917f5308948233557571fdb2dcfa895ed7667aef175000885f26fb5"}
Nov 03 19:00:23.913385 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.913340    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk" event={"ID":"2159e743-a61b-4d76-9f78-9be36e825a43","Type":"ContainerDied","Data":"46390c01d533ba6fe5ef111dc75407a8b0e76ec9b76e711ff3ec12f1e1753197"}
Nov 03 19:00:23.913473 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.913418    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"
Nov 03 19:00:23.915755 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.915731    6059 scope.go:122] "RemoveContainer" containerID="0366d3184102a5202ba42d897e05e9449e49c80f0ee47f0dc5642c67db4b9488"
Nov 03 19:00:23.922273 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.922247    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:00:23.925215 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.925199    6059 scope.go:122] "RemoveContainer" containerID="7b05d327a1d2a420fd123ff5f8ea128b8306222325363ab777c69fe2bca1eb9f"
Nov 03 19:00:23.932932 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.932916    6059 scope.go:122] "RemoveContainer" containerID="7b05d327a1d2a420fd123ff5f8ea128b8306222325363ab777c69fe2bca1eb9f"
Nov 03 19:00:23.933179 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:00:23.933158    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b05d327a1d2a420fd123ff5f8ea128b8306222325363ab777c69fe2bca1eb9f\": container with ID starting with 7b05d327a1d2a420fd123ff5f8ea128b8306222325363ab777c69fe2bca1eb9f not found: ID does not exist" containerID="7b05d327a1d2a420fd123ff5f8ea128b8306222325363ab777c69fe2bca1eb9f"
Nov 03 19:00:23.933236 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.933185    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b05d327a1d2a420fd123ff5f8ea128b8306222325363ab777c69fe2bca1eb9f"} err="failed to get container status \"7b05d327a1d2a420fd123ff5f8ea128b8306222325363ab777c69fe2bca1eb9f\": rpc error: code = NotFound desc = could not find container \"7b05d327a1d2a420fd123ff5f8ea128b8306222325363ab777c69fe2bca1eb9f\": container with ID starting with 7b05d327a1d2a420fd123ff5f8ea128b8306222325363ab777c69fe2bca1eb9f not found: ID does not exist"
Nov 03 19:00:23.933236 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:23.933207    6059 scope.go:122] "RemoveContainer" containerID="a534099f4dffc13956221f54dfeab90e33cc6c87c30f1bbc8f67ee27d2c90ba1"
Nov 03 19:00:24.062063 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:24.061983    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"]
Nov 03 19:00:24.109455 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:24.109420    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zp9n7"]
Nov 03 19:00:24.172264 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:24.172235    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"]
Nov 03 19:00:24.178852 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:24.178815    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xpshn"]
Nov 03 19:00:24.286904 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:24.286871    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"]
Nov 03 19:00:24.292155 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:24.292128    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-l2xpl"]
Nov 03 19:00:24.398028 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:24.397950    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"]
Nov 03 19:00:24.402332 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:24.402308    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-gxxkk"]
Nov 03 19:00:25.335625 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:25.335594    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2159e743-a61b-4d76-9f78-9be36e825a43" path="/var/lib/kubelet/pods/2159e743-a61b-4d76-9f78-9be36e825a43/volumes"
Nov 03 19:00:25.336089 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:25.336074    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" path="/var/lib/kubelet/pods/5eb79e59-86dc-47d1-830f-49e542a5b8f3/volumes"
Nov 03 19:00:25.336493 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:25.336480    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60b66cb6-2bce-4c02-aa7d-7eb25f29f237" path="/var/lib/kubelet/pods/60b66cb6-2bce-4c02-aa7d-7eb25f29f237/volumes"
Nov 03 19:00:25.337160 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:25.337146    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9e3abf7-c5e1-4414-858e-5f84f6a88670" path="/var/lib/kubelet/pods/d9e3abf7-c5e1-4414-858e-5f84f6a88670/volumes"
Nov 03 19:00:29.958914 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:29.958878    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:00:36.859752 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:00:36.859709    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:00:44.421239 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:44.421197    6059 scope.go:122] "RemoveContainer" containerID="bf8fd4e5ed88f8bd63b615b77079a5a1ca43cd95b5fe51f85de19dafc1628430"
Nov 03 19:00:44.430409 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:44.430390    6059 scope.go:122] "RemoveContainer" containerID="22aa00a42740cb2059ba338ff1e26a3f217b0146bce62ee31845c5b77269ebe3"
Nov 03 19:00:44.438635 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:00:44.438613    6059 scope.go:122] "RemoveContainer" containerID="30f408ec57329c081d0d745a585ba4cd9b4fbc9402ab584a6b4a7dae55712f5d"
Nov 03 19:01:06.060850 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:06.060815    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:06.860448 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:01:06.860407    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:01:36.860418 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:01:36.860314    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:01:39.061167 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:39.061129    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:39.238991 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:39.238954    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:54.399841 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:54.399804    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:54.579394 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:54.579336    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:56.004005 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:56.003968    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:56.242675 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:56.242643    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:58.180495 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.180456    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj"]
Nov 03 19:01:58.180887 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.180856    6059 cpu_manager.go:414] "containerMap: removing container" podUID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" containerName="istio-proxy"
Nov 03 19:01:58.180887 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.180868    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" containerName="istio-proxy"
Nov 03 19:01:58.180887 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.180883    6059 cpu_manager.go:414] "containerMap: removing container" podUID="d9e3abf7-c5e1-4414-858e-5f84f6a88670" containerName="istio-proxy"
Nov 03 19:01:58.180887 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.180888    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9e3abf7-c5e1-4414-858e-5f84f6a88670" containerName="istio-proxy"
Nov 03 19:01:58.181031 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.180904    6059 cpu_manager.go:414] "containerMap: removing container" podUID="60b66cb6-2bce-4c02-aa7d-7eb25f29f237" containerName="istio-proxy"
Nov 03 19:01:58.181031 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.180910    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b66cb6-2bce-4c02-aa7d-7eb25f29f237" containerName="istio-proxy"
Nov 03 19:01:58.181031 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.180918    6059 cpu_manager.go:414] "containerMap: removing container" podUID="b588cba4-4e07-4d1e-af1c-013b416aba9e" containerName="cleanup"
Nov 03 19:01:58.181031 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.180922    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b588cba4-4e07-4d1e-af1c-013b416aba9e" containerName="cleanup"
Nov 03 19:01:58.181031 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.180941    6059 cpu_manager.go:414] "containerMap: removing container" podUID="2159e743-a61b-4d76-9f78-9be36e825a43" containerName="istio-proxy"
Nov 03 19:01:58.181031 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.180946    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2159e743-a61b-4d76-9f78-9be36e825a43" containerName="istio-proxy"
Nov 03 19:01:58.181031 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.180960    6059 cpu_manager.go:414] "containerMap: removing container" podUID="b588cba4-4e07-4d1e-af1c-013b416aba9e" containerName="cleanup"
Nov 03 19:01:58.181031 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.180965    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b588cba4-4e07-4d1e-af1c-013b416aba9e" containerName="cleanup"
Nov 03 19:01:58.181031 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.181019    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b588cba4-4e07-4d1e-af1c-013b416aba9e" containerName="cleanup"
Nov 03 19:01:58.181031 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.181027    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb79e59-86dc-47d1-830f-49e542a5b8f3" containerName="istio-proxy"
Nov 03 19:01:58.181031 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.181035    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9e3abf7-c5e1-4414-858e-5f84f6a88670" containerName="istio-proxy"
Nov 03 19:01:58.181414 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.181042    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b66cb6-2bce-4c02-aa7d-7eb25f29f237" containerName="istio-proxy"
Nov 03 19:01:58.181414 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.181051    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2159e743-a61b-4d76-9f78-9be36e825a43" containerName="istio-proxy"
Nov 03 19:01:58.181414 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.181056    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b588cba4-4e07-4d1e-af1c-013b416aba9e" containerName="cleanup"
Nov 03 19:01:58.185423 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.185403    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj"
Nov 03 19:01:58.235181 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.235159    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:01:58.235311 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.235257    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:01:58.242826 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.242807    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f331127f-493e-45a6-a834-747c701fab2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6cb1d90-n2qqj\" (UID: \"f331127f-493e-45a6-a834-747c701fab2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj"
Nov 03 19:01:58.242912 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.242857    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhft7\" (UniqueName: \"kubernetes.io/projected/f331127f-493e-45a6-a834-747c701fab2c-kube-api-access-dhft7\") pod \"maas-api-revoke-sub-d6cb1d90-n2qqj\" (UID: \"f331127f-493e-45a6-a834-747c701fab2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj"
Nov 03 19:01:58.242981 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.242962    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f331127f-493e-45a6-a834-747c701fab2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6cb1d90-n2qqj\" (UID: \"f331127f-493e-45a6-a834-747c701fab2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj"
Nov 03 19:01:58.282591 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.282567    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:01:58.282591 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.282586    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:01:58.284330 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.284308    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj"]
Nov 03 19:01:58.343998 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.343970    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f331127f-493e-45a6-a834-747c701fab2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6cb1d90-n2qqj\" (UID: \"f331127f-493e-45a6-a834-747c701fab2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj"
Nov 03 19:01:58.344169 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.344068    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f331127f-493e-45a6-a834-747c701fab2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6cb1d90-n2qqj\" (UID: \"f331127f-493e-45a6-a834-747c701fab2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj"
Nov 03 19:01:58.344169 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.344126    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhft7\" (UniqueName: \"kubernetes.io/projected/f331127f-493e-45a6-a834-747c701fab2c-kube-api-access-dhft7\") pod \"maas-api-revoke-sub-d6cb1d90-n2qqj\" (UID: \"f331127f-493e-45a6-a834-747c701fab2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj"
Nov 03 19:01:58.344752 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.344730    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f331127f-493e-45a6-a834-747c701fab2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6cb1d90-n2qqj\" (UID: \"f331127f-493e-45a6-a834-747c701fab2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj"
Nov 03 19:01:58.346559 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.346540    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f331127f-493e-45a6-a834-747c701fab2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6cb1d90-n2qqj\" (UID: \"f331127f-493e-45a6-a834-747c701fab2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj"
Nov 03 19:01:58.402654 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.402621    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhft7\" (UniqueName: \"kubernetes.io/projected/f331127f-493e-45a6-a834-747c701fab2c-kube-api-access-dhft7\") pod \"maas-api-revoke-sub-d6cb1d90-n2qqj\" (UID: \"f331127f-493e-45a6-a834-747c701fab2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj"
Nov 03 19:01:58.499337 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.499303    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj"
Nov 03 19:01:58.865915 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:58.865890    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj"]
Nov 03 19:01:58.868298 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:01:58.868267    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf331127f_493e_45a6_a834_747c701fab2c.slice/crio-7d9f0027af7b99f39888edd934d13f675fb655d0b1726c8356e23b416e773304 WatchSource:0}: Error finding container 7d9f0027af7b99f39888edd934d13f675fb655d0b1726c8356e23b416e773304: Status 404 returned error can't find the container with id 7d9f0027af7b99f39888edd934d13f675fb655d0b1726c8356e23b416e773304
Nov 03 19:01:59.390709 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:59.390617    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj" event={"ID":"f331127f-493e-45a6-a834-747c701fab2c","Type":"ContainerStarted","Data":"cf056963ff8fd4b3bfa39ee7f97a69e1ad5a1afaa19329e9d502cabb34c75950"}
Nov 03 19:01:59.390709 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:59.390671    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj" event={"ID":"f331127f-493e-45a6-a834-747c701fab2c","Type":"ContainerStarted","Data":"7d9f0027af7b99f39888edd934d13f675fb655d0b1726c8356e23b416e773304"}
Nov 03 19:01:59.501183 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:01:59.501129    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj" podStartSLOduration=1.501113533 podStartE2EDuration="1.501113533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:59.500895874 +0000 UTC m=+6260.298608645" watchObservedRunningTime="2026-11-03 19:01:59.501113533 +0000 UTC m=+6260.298826313"
Nov 03 19:02:00.397480 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:00.397445    6059 generic.go:356] "Generic (PLEG): container finished" podID="f331127f-493e-45a6-a834-747c701fab2c" containerID="cf056963ff8fd4b3bfa39ee7f97a69e1ad5a1afaa19329e9d502cabb34c75950" exitCode=0
Nov 03 19:02:00.397907 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:00.397527    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj" event={"ID":"f331127f-493e-45a6-a834-747c701fab2c","Type":"ContainerDied","Data":"cf056963ff8fd4b3bfa39ee7f97a69e1ad5a1afaa19329e9d502cabb34c75950"}
Nov 03 19:02:01.532226 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:01.532199    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj"
Nov 03 19:02:01.582239 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:01.582207    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f331127f-493e-45a6-a834-747c701fab2c-kube-api-access-dhft7\" (UniqueName: \"kubernetes.io/projected/f331127f-493e-45a6-a834-747c701fab2c-kube-api-access-dhft7\") pod \"f331127f-493e-45a6-a834-747c701fab2c\" (UID: \"f331127f-493e-45a6-a834-747c701fab2c\") "
Nov 03 19:02:01.582454 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:01.582248    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f331127f-493e-45a6-a834-747c701fab2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f331127f-493e-45a6-a834-747c701fab2c-maas-api-openshift-ca\") pod \"f331127f-493e-45a6-a834-747c701fab2c\" (UID: \"f331127f-493e-45a6-a834-747c701fab2c\") "
Nov 03 19:02:01.582454 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:01.582400    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f331127f-493e-45a6-a834-747c701fab2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f331127f-493e-45a6-a834-747c701fab2c-maas-api-xks-ca\") pod \"f331127f-493e-45a6-a834-747c701fab2c\" (UID: \"f331127f-493e-45a6-a834-747c701fab2c\") "
Nov 03 19:02:01.582664 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:01.582637    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f331127f-493e-45a6-a834-747c701fab2c-maas-api-openshift-ca" pod "f331127f-493e-45a6-a834-747c701fab2c" (UID: "f331127f-493e-45a6-a834-747c701fab2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:02:01.582760 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:01.582745    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f331127f-493e-45a6-a834-747c701fab2c-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:01.584399 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:01.584379    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f331127f-493e-45a6-a834-747c701fab2c-maas-api-xks-ca" pod "f331127f-493e-45a6-a834-747c701fab2c" (UID: "f331127f-493e-45a6-a834-747c701fab2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:02:01.584499 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:01.584399    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f331127f-493e-45a6-a834-747c701fab2c-kube-api-access-dhft7" pod "f331127f-493e-45a6-a834-747c701fab2c" (UID: "f331127f-493e-45a6-a834-747c701fab2c"). InnerVolumeSpecName "kube-api-access-dhft7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:02:01.683784 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:01.683680    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f331127f-493e-45a6-a834-747c701fab2c-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:01.683784 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:01.683717    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhft7\" (UniqueName: \"kubernetes.io/projected/f331127f-493e-45a6-a834-747c701fab2c-kube-api-access-dhft7\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:02.408722 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:02.408688    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj"
Nov 03 19:02:02.408913 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:02.408694    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj" event={"ID":"f331127f-493e-45a6-a834-747c701fab2c","Type":"ContainerDied","Data":"7d9f0027af7b99f39888edd934d13f675fb655d0b1726c8356e23b416e773304"}
Nov 03 19:02:02.408913 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:02.408814    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d9f0027af7b99f39888edd934d13f675fb655d0b1726c8356e23b416e773304"
Nov 03 19:02:06.403513 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:06.403467    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:02:06.860188 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:02:06.860146    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:02:08.831155 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:08.831074    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:02:15.060990 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:15.060955    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:02:15.549347 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:15.549312    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:02:33.781903 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.781860    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"]
Nov 03 19:02:33.782315 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.782286    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f331127f-493e-45a6-a834-747c701fab2c" containerName="revoke-keys"
Nov 03 19:02:33.782315 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.782299    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f331127f-493e-45a6-a834-747c701fab2c" containerName="revoke-keys"
Nov 03 19:02:33.782418 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.782328    6059 cpu_manager.go:414] "containerMap: removing container" podUID="b588cba4-4e07-4d1e-af1c-013b416aba9e" containerName="cleanup"
Nov 03 19:02:33.782418 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.782334    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b588cba4-4e07-4d1e-af1c-013b416aba9e" containerName="cleanup"
Nov 03 19:02:33.782418 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.782405    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b588cba4-4e07-4d1e-af1c-013b416aba9e" containerName="cleanup"
Nov 03 19:02:33.782418 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.782416    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f331127f-493e-45a6-a834-747c701fab2c" containerName="revoke-keys"
Nov 03 19:02:33.785689 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.785668    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.829424 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.829386    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"]
Nov 03 19:02:33.833006 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.832989    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.847268 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.847239    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-t4h2g\""
Nov 03 19:02:33.886261 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.886233    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"]
Nov 03 19:02:33.888654 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.888626    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24c1306-3d52-4b39-af5d-5b769ec03a1a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.888811 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.888664    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c17b0b8a-a3a3-4824-acc3-ce83fb644937-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.888811 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.888697    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c17b0b8a-a3a3-4824-acc3-ce83fb644937-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.888811 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.888790    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24c1306-3d52-4b39-af5d-5b769ec03a1a-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.888972 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.888868    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c17b0b8a-a3a3-4824-acc3-ce83fb644937-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.888972 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.888910    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24c1306-3d52-4b39-af5d-5b769ec03a1a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.888972 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.888952    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c17b0b8a-a3a3-4824-acc3-ce83fb644937-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.889104 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.888996    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c1306-3d52-4b39-af5d-5b769ec03a1a-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.889104 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.889047    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c1306-3d52-4b39-af5d-5b769ec03a1a-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.889104 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.889068    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c17b0b8a-a3a3-4824-acc3-ce83fb644937-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.889104 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.889084    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c17b0b8a-a3a3-4824-acc3-ce83fb644937-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.889259 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.889149    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c17b0b8a-a3a3-4824-acc3-ce83fb644937-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.889259 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.889176    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxnpj\" (UniqueName: \"kubernetes.io/projected/c24c1306-3d52-4b39-af5d-5b769ec03a1a-kube-api-access-bxnpj\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.889259 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.889214    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c24c1306-3d52-4b39-af5d-5b769ec03a1a-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.889259 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.889250    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24c1306-3d52-4b39-af5d-5b769ec03a1a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.889497 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.889336    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c17b0b8a-a3a3-4824-acc3-ce83fb644937-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.889497 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.889399    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4dlv\" (UniqueName: \"kubernetes.io/projected/c17b0b8a-a3a3-4824-acc3-ce83fb644937-kube-api-access-v4dlv\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.889497 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.889436    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24c1306-3d52-4b39-af5d-5b769ec03a1a-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.933701 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.933669    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"]
Nov 03 19:02:33.990145 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.990104    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c1306-3d52-4b39-af5d-5b769ec03a1a-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.990435 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.990152    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c1306-3d52-4b39-af5d-5b769ec03a1a-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.990435 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.990186    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c17b0b8a-a3a3-4824-acc3-ce83fb644937-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.990435 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.990207    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c17b0b8a-a3a3-4824-acc3-ce83fb644937-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.990435 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.990225    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c17b0b8a-a3a3-4824-acc3-ce83fb644937-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.990435 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.990248    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxnpj\" (UniqueName: \"kubernetes.io/projected/c24c1306-3d52-4b39-af5d-5b769ec03a1a-kube-api-access-bxnpj\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.990435 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.990292    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c24c1306-3d52-4b39-af5d-5b769ec03a1a-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.990435 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.990319    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24c1306-3d52-4b39-af5d-5b769ec03a1a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.990435 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.990404    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c17b0b8a-a3a3-4824-acc3-ce83fb644937-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.990435 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.990433    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4dlv\" (UniqueName: \"kubernetes.io/projected/c17b0b8a-a3a3-4824-acc3-ce83fb644937-kube-api-access-v4dlv\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.990946 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.990467    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24c1306-3d52-4b39-af5d-5b769ec03a1a-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.990946 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.990515    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24c1306-3d52-4b39-af5d-5b769ec03a1a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.990946 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.990541    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c17b0b8a-a3a3-4824-acc3-ce83fb644937-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.990946 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.990573    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c17b0b8a-a3a3-4824-acc3-ce83fb644937-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.990946 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.990604    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24c1306-3d52-4b39-af5d-5b769ec03a1a-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.990946 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.990613    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c1306-3d52-4b39-af5d-5b769ec03a1a-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.990946 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.990645    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c17b0b8a-a3a3-4824-acc3-ce83fb644937-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.990946 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.990651    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c17b0b8a-a3a3-4824-acc3-ce83fb644937-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.990946 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.990770    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24c1306-3d52-4b39-af5d-5b769ec03a1a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.990946 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.990815    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c17b0b8a-a3a3-4824-acc3-ce83fb644937-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.990946 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.990897    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c1306-3d52-4b39-af5d-5b769ec03a1a-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.991911 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.991004    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c17b0b8a-a3a3-4824-acc3-ce83fb644937-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.991911 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.991018    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24c1306-3d52-4b39-af5d-5b769ec03a1a-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.991911 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.991412    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c17b0b8a-a3a3-4824-acc3-ce83fb644937-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.991911 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.991477    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c17b0b8a-a3a3-4824-acc3-ce83fb644937-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.991911 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.991898    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c17b0b8a-a3a3-4824-acc3-ce83fb644937-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.992219 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.992041    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24c1306-3d52-4b39-af5d-5b769ec03a1a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.992325 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.992299    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24c1306-3d52-4b39-af5d-5b769ec03a1a-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.993027 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.993001    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c17b0b8a-a3a3-4824-acc3-ce83fb644937-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.993027 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.993023    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24c1306-3d52-4b39-af5d-5b769ec03a1a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:33.993226 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.993207    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c17b0b8a-a3a3-4824-acc3-ce83fb644937-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:33.993886 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:33.993865    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24c1306-3d52-4b39-af5d-5b769ec03a1a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:34.049603 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:34.049528    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c24c1306-3d52-4b39-af5d-5b769ec03a1a-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:34.049779 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:34.049753    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c17b0b8a-a3a3-4824-acc3-ce83fb644937-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:34.049827 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:34.049788    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4dlv\" (UniqueName: \"kubernetes.io/projected/c17b0b8a-a3a3-4824-acc3-ce83fb644937-kube-api-access-v4dlv\") pod \"maas-default-gateway-openshift-default-58c6585447-bwg8p\" (UID: \"c17b0b8a-a3a3-4824-acc3-ce83fb644937\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:34.049865 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:34.049826    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxnpj\" (UniqueName: \"kubernetes.io/projected/c24c1306-3d52-4b39-af5d-5b769ec03a1a-kube-api-access-bxnpj\") pod \"maas-default-gateway-openshift-default-58c6585447-xkkj4\" (UID: \"c24c1306-3d52-4b39-af5d-5b769ec03a1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:34.098681 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:34.098641    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:34.146089 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:34.146041    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:34.516580 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:34.516545    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"]
Nov 03 19:02:34.551628 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:02:34.551596    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17b0b8a_a3a3_4824_acc3_ce83fb644937.slice/crio-3f4128293ebff1e3bb8525406d0eac79e5524b05f94e563dce3bf7a0fe761638 WatchSource:0}: Error finding container 3f4128293ebff1e3bb8525406d0eac79e5524b05f94e563dce3bf7a0fe761638: Status 404 returned error can't find the container with id 3f4128293ebff1e3bb8525406d0eac79e5524b05f94e563dce3bf7a0fe761638
Nov 03 19:02:34.553638 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:34.553607    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:02:34.553757 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:34.553684    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:02:34.553757 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:34.553728    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:02:34.566651 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:34.566630    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p" event={"ID":"c17b0b8a-a3a3-4824-acc3-ce83fb644937","Type":"ContainerStarted","Data":"3f4128293ebff1e3bb8525406d0eac79e5524b05f94e563dce3bf7a0fe761638"}
Nov 03 19:02:34.567300 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:34.567248    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"]
Nov 03 19:02:34.570486 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:02:34.570462    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24c1306_3d52_4b39_af5d_5b769ec03a1a.slice/crio-70e3dd3d0d6bb0d61cf44fcc88dc5bebe21dfcbff3514916084367b5c2b9e06d WatchSource:0}: Error finding container 70e3dd3d0d6bb0d61cf44fcc88dc5bebe21dfcbff3514916084367b5c2b9e06d: Status 404 returned error can't find the container with id 70e3dd3d0d6bb0d61cf44fcc88dc5bebe21dfcbff3514916084367b5c2b9e06d
Nov 03 19:02:34.573086 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:34.573051    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:02:34.573169 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:34.573134    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:02:34.573220 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:34.573177    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:02:35.585417 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:35.585350    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4" event={"ID":"c24c1306-3d52-4b39-af5d-5b769ec03a1a","Type":"ContainerStarted","Data":"22474b7534315d43aff0b68e41d86b4a29231e0742b6062c0355ab97163e1909"}
Nov 03 19:02:35.585910 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:35.585432    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4" event={"ID":"c24c1306-3d52-4b39-af5d-5b769ec03a1a","Type":"ContainerStarted","Data":"70e3dd3d0d6bb0d61cf44fcc88dc5bebe21dfcbff3514916084367b5c2b9e06d"}
Nov 03 19:02:35.587542 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:35.587515    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p" event={"ID":"c17b0b8a-a3a3-4824-acc3-ce83fb644937","Type":"ContainerStarted","Data":"29b6ee12db5e27f5bba861fee20ea8136ed5a2bbd8b7fcd4776dc95c2e8874ad"}
Nov 03 19:02:35.802822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:35.802767    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4" podStartSLOduration=2.80275323 podStartE2EDuration="2.80275323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:35.754884456 +0000 UTC m=+6296.552597237" watchObservedRunningTime="2026-11-03 19:02:35.80275323 +0000 UTC m=+6296.600466011"
Nov 03 19:02:35.872887 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:35.872743    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p" podStartSLOduration=2.87272151 podStartE2EDuration="2.87272151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:35.870955554 +0000 UTC m=+6296.668668345" watchObservedRunningTime="2026-11-03 19:02:35.87272151 +0000 UTC m=+6296.670434293"
Nov 03 19:02:36.099393 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:36.099341    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:36.100969 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:36.100946    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bwg8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" start-of-body=
Nov 03 19:02:36.101073 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:36.100989    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p" podUID="c17b0b8a-a3a3-4824-acc3-ce83fb644937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused"
Nov 03 19:02:36.152436 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:36.152007    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:36.152436 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:36.152168    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xkkj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body=
Nov 03 19:02:36.152436 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:36.152213    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4" podUID="c24c1306-3d52-4b39-af5d-5b769ec03a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused"
Nov 03 19:02:36.900548 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:02:36.900508    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:02:37.099850 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:37.099816    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bwg8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" start-of-body=
Nov 03 19:02:37.100029 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:37.099867    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p" podUID="c17b0b8a-a3a3-4824-acc3-ce83fb644937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused"
Nov 03 19:02:37.150379 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:37.146791    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xkkj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body=
Nov 03 19:02:37.150379 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:37.146877    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4" podUID="c24c1306-3d52-4b39-af5d-5b769ec03a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused"
Nov 03 19:02:37.208232 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:37.208199    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:02:38.099070 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:38.099025    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bwg8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" start-of-body=
Nov 03 19:02:38.099535 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:38.099451    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p" podUID="c17b0b8a-a3a3-4824-acc3-ce83fb644937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused"
Nov 03 19:02:38.146874 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:38.146831    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xkkj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body=
Nov 03 19:02:38.147176 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:38.147145    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4" podUID="c24c1306-3d52-4b39-af5d-5b769ec03a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused"
Nov 03 19:02:39.099454 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:39.099420    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bwg8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" start-of-body=
Nov 03 19:02:39.099879 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:39.099490    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p" podUID="c17b0b8a-a3a3-4824-acc3-ce83fb644937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused"
Nov 03 19:02:39.151513 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:39.151471    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xkkj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body=
Nov 03 19:02:39.151835 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:39.151805    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4" podUID="c24c1306-3d52-4b39-af5d-5b769ec03a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused"
Nov 03 19:02:40.099783 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:40.099733    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bwg8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" start-of-body=
Nov 03 19:02:40.100270 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:40.099805    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p" podUID="c17b0b8a-a3a3-4824-acc3-ce83fb644937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused"
Nov 03 19:02:40.163380 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:40.162914    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xkkj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body=
Nov 03 19:02:40.163380 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:40.162975    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4" podUID="c24c1306-3d52-4b39-af5d-5b769ec03a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused"
Nov 03 19:02:40.295018 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:40.294981    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 19:02:40.304079 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:40.304051    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 19:02:40.311202 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:40.311178    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 19:02:40.324922 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:40.324899    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 19:02:41.099459 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:41.099415    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bwg8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" start-of-body=
Nov 03 19:02:41.099697 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:41.099472    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p" podUID="c17b0b8a-a3a3-4824-acc3-ce83fb644937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused"
Nov 03 19:02:41.146612 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:41.146566    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xkkj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body=
Nov 03 19:02:41.147118 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:41.146641    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4" podUID="c24c1306-3d52-4b39-af5d-5b769ec03a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused"
Nov 03 19:02:42.099196 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:42.099159    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bwg8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused" start-of-body=
Nov 03 19:02:42.099433 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:42.099212    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p" podUID="c17b0b8a-a3a3-4824-acc3-ce83fb644937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": dial tcp 10.133.1.70:15021: connect: connection refused"
Nov 03 19:02:42.146500 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:42.146466    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xkkj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body=
Nov 03 19:02:42.146718 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:42.146519    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4" podUID="c24c1306-3d52-4b39-af5d-5b769ec03a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused"
Nov 03 19:02:44.140607 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:44.140557    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bwg8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:02:44.141144 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:44.140641    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p" podUID="c17b0b8a-a3a3-4824-acc3-ce83fb644937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:44.188113 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:44.188082    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:44.188434 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:44.188408    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xkkj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:02:44.188576 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:44.188474    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4" podUID="c24c1306-3d52-4b39-af5d-5b769ec03a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:44.188717 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:44.188697    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:44.189631 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:44.189606    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p"
Nov 03 19:02:44.210498 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:44.210464    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:44.210809 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:44.210789    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:44.211703 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:44.211679    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4"
Nov 03 19:02:48.925573 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:48.925540    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"]
Nov 03 19:02:48.929604 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:48.929583    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:48.993193 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:48.993156    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc06524-297b-4b57-b0a2-4b2a92d4a00c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:48.993420 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:48.993292    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc06524-297b-4b57-b0a2-4b2a92d4a00c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:48.993420 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:48.993368    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc06524-297b-4b57-b0a2-4b2a92d4a00c-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:48.993420 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:48.993398    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc06524-297b-4b57-b0a2-4b2a92d4a00c-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:48.993588 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:48.993515    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc06524-297b-4b57-b0a2-4b2a92d4a00c-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:48.993588 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:48.993555    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc06524-297b-4b57-b0a2-4b2a92d4a00c-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:48.993588 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:48.993574    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb98v\" (UniqueName: \"kubernetes.io/projected/acc06524-297b-4b57-b0a2-4b2a92d4a00c-kube-api-access-lb98v\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:48.993813 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:48.993703    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc06524-297b-4b57-b0a2-4b2a92d4a00c-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:48.993813 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:48.993765    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc06524-297b-4b57-b0a2-4b2a92d4a00c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:49.095382 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.095325    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc06524-297b-4b57-b0a2-4b2a92d4a00c-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:49.095710 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.095684    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc06524-297b-4b57-b0a2-4b2a92d4a00c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:49.095861 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.095843    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc06524-297b-4b57-b0a2-4b2a92d4a00c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:49.096048 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.096031    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc06524-297b-4b57-b0a2-4b2a92d4a00c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:49.096189 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.096174    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc06524-297b-4b57-b0a2-4b2a92d4a00c-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:49.096310 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.096294    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc06524-297b-4b57-b0a2-4b2a92d4a00c-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:49.096471 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.096454    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc06524-297b-4b57-b0a2-4b2a92d4a00c-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:49.096600 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.096585    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc06524-297b-4b57-b0a2-4b2a92d4a00c-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:49.096718 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.096702    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb98v\" (UniqueName: \"kubernetes.io/projected/acc06524-297b-4b57-b0a2-4b2a92d4a00c-kube-api-access-lb98v\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:49.097628 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.097599    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc06524-297b-4b57-b0a2-4b2a92d4a00c-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:49.100994 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.100291    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc06524-297b-4b57-b0a2-4b2a92d4a00c-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:49.100994 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.100622    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc06524-297b-4b57-b0a2-4b2a92d4a00c-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:49.101169 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.100991    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc06524-297b-4b57-b0a2-4b2a92d4a00c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:49.101568 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.101483    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc06524-297b-4b57-b0a2-4b2a92d4a00c-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:49.101949 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.101845    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc06524-297b-4b57-b0a2-4b2a92d4a00c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:49.104230 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.104206    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc06524-297b-4b57-b0a2-4b2a92d4a00c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:49.127233 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.127201    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"]
Nov 03 19:02:49.155330 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.155299    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb98v\" (UniqueName: \"kubernetes.io/projected/acc06524-297b-4b57-b0a2-4b2a92d4a00c-kube-api-access-lb98v\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:49.158465 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.158440    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc06524-297b-4b57-b0a2-4b2a92d4a00c-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-c55ww\" (UID: \"acc06524-297b-4b57-b0a2-4b2a92d4a00c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:49.241840 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.241801    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:49.664159 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.664136    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"]
Nov 03 19:02:49.666475 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:02:49.666446    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc06524_297b_4b57_b0a2_4b2a92d4a00c.slice/crio-393b8c29127066eeefa5d9b0cddf914a52737fcd39945cff1214b753ad24e27e WatchSource:0}: Error finding container 393b8c29127066eeefa5d9b0cddf914a52737fcd39945cff1214b753ad24e27e: Status 404 returned error can't find the container with id 393b8c29127066eeefa5d9b0cddf914a52737fcd39945cff1214b753ad24e27e
Nov 03 19:02:49.668644 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.668612    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:02:49.668721 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.668677    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:02:49.668721 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:49.668707    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:02:50.673546 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:50.673505    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww" event={"ID":"acc06524-297b-4b57-b0a2-4b2a92d4a00c","Type":"ContainerStarted","Data":"8630895470dde42bbe8da6d14e576264485ba2026a6618a2035a588cda17c191"}
Nov 03 19:02:50.673938 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:50.673573    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww" event={"ID":"acc06524-297b-4b57-b0a2-4b2a92d4a00c","Type":"ContainerStarted","Data":"393b8c29127066eeefa5d9b0cddf914a52737fcd39945cff1214b753ad24e27e"}
Nov 03 19:02:51.252380 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:51.250200    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:51.252380 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:51.250325    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-c55ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Nov 03 19:02:51.252380 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:51.250402    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww" podUID="acc06524-297b-4b57-b0a2-4b2a92d4a00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Nov 03 19:02:52.253381 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:52.251589    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-c55ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Nov 03 19:02:52.253381 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:52.251648    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww" podUID="acc06524-297b-4b57-b0a2-4b2a92d4a00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Nov 03 19:02:53.247389 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:53.245770    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-c55ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Nov 03 19:02:53.247389 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:53.245836    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww" podUID="acc06524-297b-4b57-b0a2-4b2a92d4a00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Nov 03 19:02:54.249380 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:54.246895    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-c55ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Nov 03 19:02:54.249380 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:54.246957    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww" podUID="acc06524-297b-4b57-b0a2-4b2a92d4a00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Nov 03 19:02:55.254398 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:55.250813    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-c55ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Nov 03 19:02:55.254398 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:55.250876    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww" podUID="acc06524-297b-4b57-b0a2-4b2a92d4a00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Nov 03 19:02:56.242480 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:56.242443    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-c55ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Nov 03 19:02:56.242773 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:56.242502    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww" podUID="acc06524-297b-4b57-b0a2-4b2a92d4a00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Nov 03 19:02:57.242474 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:57.242346    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-c55ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Nov 03 19:02:57.242860 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:57.242503    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww" podUID="acc06524-297b-4b57-b0a2-4b2a92d4a00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Nov 03 19:02:59.106756 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:59.106723    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:59.107247 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:59.107095    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:59.107919 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:59.107897    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww"
Nov 03 19:02:59.270155 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:02:59.270102    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww" podStartSLOduration=11.270089755 podStartE2EDuration="11.270089755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:50.843296553 +0000 UTC m=+6311.641009334" watchObservedRunningTime="2026-11-03 19:02:59.270089755 +0000 UTC m=+6320.067802608"
Nov 03 19:03:06.741703 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:06.741669    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:03:06.860603 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:03:06.860563    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:03:09.093821 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:09.093783    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:03:36.862202 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:03:36.862157    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:03:39.737243 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:39.737203    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k"]
Nov 03 19:03:39.741258 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:39.741231    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k"
Nov 03 19:03:39.832928 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:39.832893    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:03:39.832928 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:39.832893    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:03:39.832928 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:39.832928    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:03:39.834819 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:39.834796    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k"]
Nov 03 19:03:39.839832 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:39.839814    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:03:39.842617 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:39.842588    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c4c69dc-cdcf-4854-b924-34202b02d0be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af40a079-f7g9k\" (UID: \"3c4c69dc-cdcf-4854-b924-34202b02d0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k"
Nov 03 19:03:39.842690 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:39.842635    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nmjf\" (UniqueName: \"kubernetes.io/projected/3c4c69dc-cdcf-4854-b924-34202b02d0be-kube-api-access-6nmjf\") pod \"maas-api-revoke-sub-af40a079-f7g9k\" (UID: \"3c4c69dc-cdcf-4854-b924-34202b02d0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k"
Nov 03 19:03:39.842823 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:39.842806    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c4c69dc-cdcf-4854-b924-34202b02d0be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af40a079-f7g9k\" (UID: \"3c4c69dc-cdcf-4854-b924-34202b02d0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k"
Nov 03 19:03:39.944514 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:39.944470    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c4c69dc-cdcf-4854-b924-34202b02d0be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af40a079-f7g9k\" (UID: \"3c4c69dc-cdcf-4854-b924-34202b02d0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k"
Nov 03 19:03:39.944707 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:39.944533    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c4c69dc-cdcf-4854-b924-34202b02d0be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af40a079-f7g9k\" (UID: \"3c4c69dc-cdcf-4854-b924-34202b02d0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k"
Nov 03 19:03:39.944707 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:39.944567    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nmjf\" (UniqueName: \"kubernetes.io/projected/3c4c69dc-cdcf-4854-b924-34202b02d0be-kube-api-access-6nmjf\") pod \"maas-api-revoke-sub-af40a079-f7g9k\" (UID: \"3c4c69dc-cdcf-4854-b924-34202b02d0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k"
Nov 03 19:03:39.945208 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:39.945182    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c4c69dc-cdcf-4854-b924-34202b02d0be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af40a079-f7g9k\" (UID: \"3c4c69dc-cdcf-4854-b924-34202b02d0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k"
Nov 03 19:03:39.946955 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:39.946936    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c4c69dc-cdcf-4854-b924-34202b02d0be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af40a079-f7g9k\" (UID: \"3c4c69dc-cdcf-4854-b924-34202b02d0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k"
Nov 03 19:03:40.003796 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:40.003713    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nmjf\" (UniqueName: \"kubernetes.io/projected/3c4c69dc-cdcf-4854-b924-34202b02d0be-kube-api-access-6nmjf\") pod \"maas-api-revoke-sub-af40a079-f7g9k\" (UID: \"3c4c69dc-cdcf-4854-b924-34202b02d0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k"
Nov 03 19:03:40.056057 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:40.056021    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k"
Nov 03 19:03:40.424558 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:40.424529    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k"]
Nov 03 19:03:40.427168 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:03:40.427128    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4c69dc_cdcf_4854_b924_34202b02d0be.slice/crio-459147db5f81248d021d20c31424b063f88bfb96ada12335ea069ee4366eab4c WatchSource:0}: Error finding container 459147db5f81248d021d20c31424b063f88bfb96ada12335ea069ee4366eab4c: Status 404 returned error can't find the container with id 459147db5f81248d021d20c31424b063f88bfb96ada12335ea069ee4366eab4c
Nov 03 19:03:40.948949 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:40.948908    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k" event={"ID":"3c4c69dc-cdcf-4854-b924-34202b02d0be","Type":"ContainerStarted","Data":"0866b1d7177428f8c88c33dfaa7d5f408052bd781cdafccb54be776a74a4009a"}
Nov 03 19:03:40.949416 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:40.948986    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k" event={"ID":"3c4c69dc-cdcf-4854-b924-34202b02d0be","Type":"ContainerStarted","Data":"459147db5f81248d021d20c31424b063f88bfb96ada12335ea069ee4366eab4c"}
Nov 03 19:03:41.078557 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:41.078448    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k" podStartSLOduration=2.078432311 podStartE2EDuration="2.078432311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:41.076473449 +0000 UTC m=+6361.874186232" watchObservedRunningTime="2026-11-03 19:03:41.078432311 +0000 UTC m=+6361.876145086"
Nov 03 19:03:41.956019 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:41.955982    6059 generic.go:356] "Generic (PLEG): container finished" podID="3c4c69dc-cdcf-4854-b924-34202b02d0be" containerID="0866b1d7177428f8c88c33dfaa7d5f408052bd781cdafccb54be776a74a4009a" exitCode=0
Nov 03 19:03:41.956533 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:41.956033    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k" event={"ID":"3c4c69dc-cdcf-4854-b924-34202b02d0be","Type":"ContainerDied","Data":"0866b1d7177428f8c88c33dfaa7d5f408052bd781cdafccb54be776a74a4009a"}
Nov 03 19:03:43.095385 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:43.095334    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k"
Nov 03 19:03:43.179205 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:43.179166    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c4c69dc-cdcf-4854-b924-34202b02d0be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c4c69dc-cdcf-4854-b924-34202b02d0be-maas-api-openshift-ca\") pod \"3c4c69dc-cdcf-4854-b924-34202b02d0be\" (UID: \"3c4c69dc-cdcf-4854-b924-34202b02d0be\") "
Nov 03 19:03:43.179446 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:43.179253    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4c69dc-cdcf-4854-b924-34202b02d0be-kube-api-access-6nmjf\" (UniqueName: \"kubernetes.io/projected/3c4c69dc-cdcf-4854-b924-34202b02d0be-kube-api-access-6nmjf\") pod \"3c4c69dc-cdcf-4854-b924-34202b02d0be\" (UID: \"3c4c69dc-cdcf-4854-b924-34202b02d0be\") "
Nov 03 19:03:43.179446 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:43.179303    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c4c69dc-cdcf-4854-b924-34202b02d0be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c4c69dc-cdcf-4854-b924-34202b02d0be-maas-api-xks-ca\") pod \"3c4c69dc-cdcf-4854-b924-34202b02d0be\" (UID: \"3c4c69dc-cdcf-4854-b924-34202b02d0be\") "
Nov 03 19:03:43.179605 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:43.179567    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c4c69dc-cdcf-4854-b924-34202b02d0be-maas-api-openshift-ca" pod "3c4c69dc-cdcf-4854-b924-34202b02d0be" (UID: "3c4c69dc-cdcf-4854-b924-34202b02d0be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:03:43.181435 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:43.181398    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c4c69dc-cdcf-4854-b924-34202b02d0be-maas-api-xks-ca" pod "3c4c69dc-cdcf-4854-b924-34202b02d0be" (UID: "3c4c69dc-cdcf-4854-b924-34202b02d0be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:03:43.181545 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:43.181460    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4c69dc-cdcf-4854-b924-34202b02d0be-kube-api-access-6nmjf" pod "3c4c69dc-cdcf-4854-b924-34202b02d0be" (UID: "3c4c69dc-cdcf-4854-b924-34202b02d0be"). InnerVolumeSpecName "kube-api-access-6nmjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:03:43.280180 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:43.280092    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c4c69dc-cdcf-4854-b924-34202b02d0be-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:03:43.280180 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:43.280123    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nmjf\" (UniqueName: \"kubernetes.io/projected/3c4c69dc-cdcf-4854-b924-34202b02d0be-kube-api-access-6nmjf\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:03:43.280180 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:43.280134    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c4c69dc-cdcf-4854-b924-34202b02d0be-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:03:43.967138 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:43.967092    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k" event={"ID":"3c4c69dc-cdcf-4854-b924-34202b02d0be","Type":"ContainerDied","Data":"459147db5f81248d021d20c31424b063f88bfb96ada12335ea069ee4366eab4c"}
Nov 03 19:03:43.967318 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:43.967174    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="459147db5f81248d021d20c31424b063f88bfb96ada12335ea069ee4366eab4c"
Nov 03 19:03:43.967318 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:43.967110    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k"
Nov 03 19:03:50.578037 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:50.578001    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:03:56.646467 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:03:56.646429    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:04:06.861759 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:04:06.861721    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:04:07.261094 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:07.261048    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:04:07.436490 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:07.436458    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:04:19.708891 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.708803    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"]
Nov 03 19:04:19.709400 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.709257    6059 cpu_manager.go:414] "containerMap: removing container" podUID="3c4c69dc-cdcf-4854-b924-34202b02d0be" containerName="revoke-keys"
Nov 03 19:04:19.709400 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.709269    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c4c69dc-cdcf-4854-b924-34202b02d0be" containerName="revoke-keys"
Nov 03 19:04:19.709400 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.709374    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c4c69dc-cdcf-4854-b924-34202b02d0be" containerName="revoke-keys"
Nov 03 19:04:19.713221 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.713199    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.880467 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.880419    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.880668 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.880489    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.880668 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.880571    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.880668 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.880607    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.880865 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.880721    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.880865 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.880784    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.880865 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.880849    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztxh6\" (UniqueName: \"kubernetes.io/projected/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-kube-api-access-ztxh6\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.881022 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.880897    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.881022 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.881004    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.915414 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.915376    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"]
Nov 03 19:04:19.982100 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.982008    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.982100 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.982065    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.982100 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.982087    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.982430 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.982106    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.982430 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.982134    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.982430 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.982162    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.982430 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.982211    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztxh6\" (UniqueName: \"kubernetes.io/projected/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-kube-api-access-ztxh6\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.982430 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.982253    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.982430 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.982317    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.982770 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.982712    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.982770 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.982753    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.982860 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.982787    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.982860 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.982834    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.982988 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.982968    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.984592 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.984564    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:19.984885 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:19.984866    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:20.040980 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:20.040940    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:20.041153 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:20.041058    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztxh6\" (UniqueName: \"kubernetes.io/projected/3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3-kube-api-access-ztxh6\") pod \"maas-default-gateway-openshift-default-58c6585447-csr7q\" (UID: \"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:20.327428 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:20.327303    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:20.755168 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:20.755133    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"]
Nov 03 19:04:20.756795 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:04:20.756760    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5d6ca4_de05_401f_a8ce_5e1c02a6dcc3.slice/crio-2d892b5df83454b46717b7a8e5b3526d47f1635edfbcb5da83e3cb94eacaf649 WatchSource:0}: Error finding container 2d892b5df83454b46717b7a8e5b3526d47f1635edfbcb5da83e3cb94eacaf649: Status 404 returned error can't find the container with id 2d892b5df83454b46717b7a8e5b3526d47f1635edfbcb5da83e3cb94eacaf649
Nov 03 19:04:20.758883 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:20.758852    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:04:20.758957 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:20.758917    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:04:20.758957 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:20.758947    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:04:21.198981 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:21.198891    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q" event={"ID":"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3","Type":"ContainerStarted","Data":"3d4a93995bbb9f7feaad20661a37ab0224b8c76c5f15724f803aa950d26cb1f9"}
Nov 03 19:04:21.198981 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:21.198956    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q" event={"ID":"3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3","Type":"ContainerStarted","Data":"2d892b5df83454b46717b7a8e5b3526d47f1635edfbcb5da83e3cb94eacaf649"}
Nov 03 19:04:21.344379 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:21.342926    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-csr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body=
Nov 03 19:04:21.344379 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:21.342990    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q" podUID="3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused"
Nov 03 19:04:21.357021 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:21.356978    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:21.411643 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:21.411580    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q" podStartSLOduration=2.411564997 podStartE2EDuration="2.411564997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:04:21.363230829 +0000 UTC m=+6402.160943610" watchObservedRunningTime="2026-11-03 19:04:21.411564997 +0000 UTC m=+6402.209277837"
Nov 03 19:04:22.328016 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:22.327965    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-csr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body=
Nov 03 19:04:22.328571 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:22.328040    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q" podUID="3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused"
Nov 03 19:04:23.328495 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:23.328448    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-csr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body=
Nov 03 19:04:23.328893 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:23.328521    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q" podUID="3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused"
Nov 03 19:04:24.328465 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:24.328418    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-csr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body=
Nov 03 19:04:24.328690 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:24.328480    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q" podUID="3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused"
Nov 03 19:04:25.328759 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:25.328722    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-csr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body=
Nov 03 19:04:25.329180 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:25.328787    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q" podUID="3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused"
Nov 03 19:04:26.327929 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:26.327895    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-csr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body=
Nov 03 19:04:26.328090 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:26.327947    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q" podUID="3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused"
Nov 03 19:04:27.327751 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:27.327717    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-csr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body=
Nov 03 19:04:27.328274 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:27.327785    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q" podUID="3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused"
Nov 03 19:04:28.327851 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:28.327811    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-csr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body=
Nov 03 19:04:28.328342 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:28.327881    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q" podUID="3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused"
Nov 03 19:04:30.102999 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:30.102958    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:30.103515 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:30.103400    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:30.104471 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:04:30.104449    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q"
Nov 03 19:04:36.861009 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:04:36.860961    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:05:02.011193 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:02.011155    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:02.192300 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:02.192262    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:06.862104 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:05:06.862061    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:05:07.468703 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:07.468666    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:21.170352 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.170314    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:21.702372 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.702324    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"]
Nov 03 19:05:21.706327 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.706300    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.768815 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.768781    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-88025f84-openshift-default-dockercfg-cgqkr\""
Nov 03 19:05:21.820296 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.820260    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"]
Nov 03 19:05:21.845711 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.845678    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:21.871085 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.871046    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istiod-ca-cert\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.871561 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.871099    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vd42\" (UniqueName: \"kubernetes.io/projected/dc09fa62-ea2b-4983-af9c-f7d318a908e4-kube-api-access-7vd42\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.871561 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.871147    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-workload-socket\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.871561 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.871284    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-podinfo\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.871561 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.871321    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-token\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.871561 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.871351    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-workload-certs\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.871561 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.871405    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-credential-socket\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.871561 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.871434    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-data\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.871561 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.871463    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-envoy\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.972318 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.972276    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istiod-ca-cert\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.972318 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.972320    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vd42\" (UniqueName: \"kubernetes.io/projected/dc09fa62-ea2b-4983-af9c-f7d318a908e4-kube-api-access-7vd42\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.972650 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.972349    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-workload-socket\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.972650 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.972419    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-podinfo\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.972650 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.972444    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-token\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.972650 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.972472    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-workload-certs\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.972650 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.972515    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-credential-socket\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.972650 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.972555    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-data\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.972650 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.972601    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-envoy\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.973096 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.973069    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-data\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.973204 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.973147    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-credential-socket\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.973204 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.973191    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istiod-ca-cert\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.973317 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.973208    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-workload-certs\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.973317 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.973285    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-workload-socket\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.974978 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.974956    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-envoy\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:21.975128 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:21.975110    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-podinfo\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:22.031661 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:22.031628    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-token\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:22.031822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:22.031695    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vd42\" (UniqueName: \"kubernetes.io/projected/dc09fa62-ea2b-4983-af9c-f7d318a908e4-kube-api-access-7vd42\") pod \"e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") " pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:22.222823 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:22.222737    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:22.319490 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:22.319453    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:22.776317 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:22.776131    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"]
Nov 03 19:05:22.779008 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:05:22.778976    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc09fa62_ea2b_4983_af9c_f7d318a908e4.slice/crio-521885a1be5889f9f38776eb1164b541b18535f947deeccd3c6577c3bdf2ce1b WatchSource:0}: Error finding container 521885a1be5889f9f38776eb1164b541b18535f947deeccd3c6577c3bdf2ce1b: Status 404 returned error can't find the container with id 521885a1be5889f9f38776eb1164b541b18535f947deeccd3c6577c3bdf2ce1b
Nov 03 19:05:22.781181 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:22.781150    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:05:22.781277 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:22.781218    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:05:22.781277 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:22.781246    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:05:23.545583 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:23.545543    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk" event={"ID":"dc09fa62-ea2b-4983-af9c-f7d318a908e4","Type":"ContainerStarted","Data":"5a42c22faf4c34933e40ea822374d666db75b68ec94489f87572c8a738e3c0d6"}
Nov 03 19:05:23.546019 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:23.545606    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk" event={"ID":"dc09fa62-ea2b-4983-af9c-f7d318a908e4","Type":"ContainerStarted","Data":"521885a1be5889f9f38776eb1164b541b18535f947deeccd3c6577c3bdf2ce1b"}
Nov 03 19:05:23.709543 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:23.709481    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk" podStartSLOduration=2.709465861 podStartE2EDuration="2.709465861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:23.707850887 +0000 UTC m=+6464.505563691" watchObservedRunningTime="2026-11-03 19:05:23.709465861 +0000 UTC m=+6464.507178641"
Nov 03 19:05:24.320323 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:24.320286    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:24.325335 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:24.325297    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:24.550802 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:24.550766    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:24.551772 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:24.551754    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:26.670249 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:26.670211    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:26.885604 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:26.885567    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:27.061595 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:27.061562    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:30.000467 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.000430    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:30.176996 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.176963    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:30.373079 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.372994    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr"]
Nov 03 19:05:30.376998 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.376972    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr"
Nov 03 19:05:30.426766 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.426730    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-88025f84\""
Nov 03 19:05:30.462423 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.462380    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65641dd6-6289-455b-be56-40c85b340edf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr\" (UID: \"65641dd6-6289-455b-be56-40c85b340edf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr"
Nov 03 19:05:30.462605 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.462472    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fx2h\" (UniqueName: \"kubernetes.io/projected/65641dd6-6289-455b-be56-40c85b340edf-kube-api-access-6fx2h\") pod \"payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr\" (UID: \"65641dd6-6289-455b-be56-40c85b340edf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr"
Nov 03 19:05:30.481536 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.481488    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-88025f84-dockercfg-kpth8\""
Nov 03 19:05:30.484387 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.481936    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr"]
Nov 03 19:05:30.537467 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.537433    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb"]
Nov 03 19:05:30.541638 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.541613    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb"
Nov 03 19:05:30.563372 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.563329    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c34bd238-0f1b-4bb5-af7d-539c417dd6b6-plugins-config-volume\") pod \"payload-processing-e2e-mt-88025f84-745cbc88b5-89thb\" (UID: \"c34bd238-0f1b-4bb5-af7d-539c417dd6b6\") " pod="openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb"
Nov 03 19:05:30.563537 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.563393    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fx2h\" (UniqueName: \"kubernetes.io/projected/65641dd6-6289-455b-be56-40c85b340edf-kube-api-access-6fx2h\") pod \"payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr\" (UID: \"65641dd6-6289-455b-be56-40c85b340edf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr"
Nov 03 19:05:30.563537 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.563473    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjkn6\" (UniqueName: \"kubernetes.io/projected/c34bd238-0f1b-4bb5-af7d-539c417dd6b6-kube-api-access-cjkn6\") pod \"payload-processing-e2e-mt-88025f84-745cbc88b5-89thb\" (UID: \"c34bd238-0f1b-4bb5-af7d-539c417dd6b6\") " pod="openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb"
Nov 03 19:05:30.563621 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.563606    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65641dd6-6289-455b-be56-40c85b340edf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr\" (UID: \"65641dd6-6289-455b-be56-40c85b340edf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr"
Nov 03 19:05:30.564291 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.564275    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65641dd6-6289-455b-be56-40c85b340edf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr\" (UID: \"65641dd6-6289-455b-be56-40c85b340edf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr"
Nov 03 19:05:30.596209 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.596168    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb"]
Nov 03 19:05:30.622080 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.622032    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fx2h\" (UniqueName: \"kubernetes.io/projected/65641dd6-6289-455b-be56-40c85b340edf-kube-api-access-6fx2h\") pod \"payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr\" (UID: \"65641dd6-6289-455b-be56-40c85b340edf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr"
Nov 03 19:05:30.665207 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.665102    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c34bd238-0f1b-4bb5-af7d-539c417dd6b6-plugins-config-volume\") pod \"payload-processing-e2e-mt-88025f84-745cbc88b5-89thb\" (UID: \"c34bd238-0f1b-4bb5-af7d-539c417dd6b6\") " pod="openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb"
Nov 03 19:05:30.665207 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.665167    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjkn6\" (UniqueName: \"kubernetes.io/projected/c34bd238-0f1b-4bb5-af7d-539c417dd6b6-kube-api-access-cjkn6\") pod \"payload-processing-e2e-mt-88025f84-745cbc88b5-89thb\" (UID: \"c34bd238-0f1b-4bb5-af7d-539c417dd6b6\") " pod="openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb"
Nov 03 19:05:30.665883 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.665860    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c34bd238-0f1b-4bb5-af7d-539c417dd6b6-plugins-config-volume\") pod \"payload-processing-e2e-mt-88025f84-745cbc88b5-89thb\" (UID: \"c34bd238-0f1b-4bb5-af7d-539c417dd6b6\") " pod="openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb"
Nov 03 19:05:30.691339 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.691301    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr"
Nov 03 19:05:30.723903 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.723870    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjkn6\" (UniqueName: \"kubernetes.io/projected/c34bd238-0f1b-4bb5-af7d-539c417dd6b6-kube-api-access-cjkn6\") pod \"payload-processing-e2e-mt-88025f84-745cbc88b5-89thb\" (UID: \"c34bd238-0f1b-4bb5-af7d-539c417dd6b6\") " pod="openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb"
Nov 03 19:05:30.854738 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:30.854700    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb"
Nov 03 19:05:31.063459 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:31.063430    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr"]
Nov 03 19:05:31.065585 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:05:31.065548    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65641dd6_6289_455b_be56_40c85b340edf.slice/crio-b16d923905228873d5d36383894dd95a6d1cf6dff000f26aa0f91e7ba5996a82 WatchSource:0}: Error finding container b16d923905228873d5d36383894dd95a6d1cf6dff000f26aa0f91e7ba5996a82: Status 404 returned error can't find the container with id b16d923905228873d5d36383894dd95a6d1cf6dff000f26aa0f91e7ba5996a82
Nov 03 19:05:31.224974 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:31.224927    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb"]
Nov 03 19:05:31.227474 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:05:31.227440    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34bd238_0f1b_4bb5_af7d_539c417dd6b6.slice/crio-9b8f5de12c0e816269d5b3267ab54c189387ee6e5eaf0a4f5bd64a1127797c4d WatchSource:0}: Error finding container 9b8f5de12c0e816269d5b3267ab54c189387ee6e5eaf0a4f5bd64a1127797c4d: Status 404 returned error can't find the container with id 9b8f5de12c0e816269d5b3267ab54c189387ee6e5eaf0a4f5bd64a1127797c4d
Nov 03 19:05:31.589267 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:31.589230    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb" event={"ID":"c34bd238-0f1b-4bb5-af7d-539c417dd6b6","Type":"ContainerStarted","Data":"c717b0d5c74f13ffe7350e042de7dd96993ce43a05ff79735628c8ff07b95352"}
Nov 03 19:05:31.589519 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:31.589294    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb" event={"ID":"c34bd238-0f1b-4bb5-af7d-539c417dd6b6","Type":"ContainerStarted","Data":"9b8f5de12c0e816269d5b3267ab54c189387ee6e5eaf0a4f5bd64a1127797c4d"}
Nov 03 19:05:31.589519 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:31.589321    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb"
Nov 03 19:05:31.590968 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:31.590938    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr" event={"ID":"65641dd6-6289-455b-be56-40c85b340edf","Type":"ContainerStarted","Data":"2774544f1d2f5a6eec42f1283537f33d6fd033d0a26511346dd3bdb2317cf589"}
Nov 03 19:05:31.591087 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:31.590992    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr" event={"ID":"65641dd6-6289-455b-be56-40c85b340edf","Type":"ContainerStarted","Data":"b16d923905228873d5d36383894dd95a6d1cf6dff000f26aa0f91e7ba5996a82"}
Nov 03 19:05:31.591087 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:31.591069    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr"
Nov 03 19:05:31.699846 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:31.699785    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb" podStartSLOduration=1.6997703830000002 podStartE2EDuration="1.699770383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:31.699003082 +0000 UTC m=+6472.496715864" watchObservedRunningTime="2026-11-03 19:05:31.699770383 +0000 UTC m=+6472.497483164"
Nov 03 19:05:31.810200 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:31.810149    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr" podStartSLOduration=1.810135291 podStartE2EDuration="1.810135291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:31.808405518 +0000 UTC m=+6472.606118299" watchObservedRunningTime="2026-11-03 19:05:31.810135291 +0000 UTC m=+6472.607848071"
Nov 03 19:05:33.636875 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:33.636833    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:33.814044 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:33.813983    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:33.991970 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:33.991930    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:36.861619 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:05:36.861577    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:05:38.697643 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:38.697589    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:42.599047 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:42.599017    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb"
Nov 03 19:05:42.599508 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:42.599085    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr"
Nov 03 19:05:43.812605 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:43.812572    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:48.798773 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:48.798717    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:51.432048 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:51.432008    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:51.709737 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:51.709704    6059 generic.go:356] "Generic (PLEG): container finished" podID="dc09fa62-ea2b-4983-af9c-f7d318a908e4" containerID="5a42c22faf4c34933e40ea822374d666db75b68ec94489f87572c8a738e3c0d6" exitCode=0
Nov 03 19:05:51.709948 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:51.709782    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk" event={"ID":"dc09fa62-ea2b-4983-af9c-f7d318a908e4","Type":"ContainerDied","Data":"5a42c22faf4c34933e40ea822374d666db75b68ec94489f87572c8a738e3c0d6"}
Nov 03 19:05:51.710249 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:51.710232    6059 scope.go:122] "RemoveContainer" containerID="5a42c22faf4c34933e40ea822374d666db75b68ec94489f87572c8a738e3c0d6"
Nov 03 19:05:51.711502 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:51.711476    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:05:51.711571 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:51.711545    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:05:51.711629 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:51.711579    6059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:05:52.318580 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:52.318539    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:52.499218 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:52.499185    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:52.717432 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:52.717395    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk" event={"ID":"dc09fa62-ea2b-4983-af9c-f7d318a908e4","Type":"ContainerStarted","Data":"73a21919d9c8d2ef2d78f63e691256c35196abc08e074f37b83440b7def2b80e"}
Nov 03 19:05:52.922095 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:52.922067    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:53.319670 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:53.319629    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:53.320864 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:53.320836    6059 patch_prober.go:29] interesting pod/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body=
Nov 03 19:05:53.321011 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:53.320895    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk" podUID="dc09fa62-ea2b-4983-af9c-f7d318a908e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused"
Nov 03 19:05:54.249382 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:54.239043    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:54.320093 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:54.320062    6059 patch_prober.go:29] interesting pod/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body=
Nov 03 19:05:54.320282 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:54.320131    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk" podUID="dc09fa62-ea2b-4983-af9c-f7d318a908e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused"
Nov 03 19:05:54.552539 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:54.552448    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:55.320206 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:55.320169    6059 patch_prober.go:29] interesting pod/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body=
Nov 03 19:05:55.320652 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:55.320239    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk" podUID="dc09fa62-ea2b-4983-af9c-f7d318a908e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused"
Nov 03 19:05:56.320824 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:56.320788    6059 patch_prober.go:29] interesting pod/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body=
Nov 03 19:05:56.321226 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:56.320844    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk" podUID="dc09fa62-ea2b-4983-af9c-f7d318a908e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused"
Nov 03 19:05:57.320171 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:57.320135    6059 patch_prober.go:29] interesting pod/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body=
Nov 03 19:05:57.320382 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:57.320187    6059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk" podUID="dc09fa62-ea2b-4983-af9c-f7d318a908e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused"
Nov 03 19:05:58.324475 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:58.324441    6059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:58.325513 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:58.325493    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:05:59.173617 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:59.173580    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:59.381897 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:05:59.381861    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:04.376197 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:06:04.376147    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:06.861581 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:06:06.861541    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:06:07.817305 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:06:07.817273    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:09.194724 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:06:09.194690    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:14.120463 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:06:14.120430    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:19.002761 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:06:19.002718    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:23.894034 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:06:23.893997    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:28.789976 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:06:28.789942    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:33.690102 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:06:33.690062    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:36.861416 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:06:36.861371    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:06:38.582321 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:06:38.582290    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:43.587577 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:06:43.587543    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:48.482077 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:06:48.482045    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:53.399192 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:06:53.399157    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:58.296570 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:06:58.296527    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:02.058305 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:02.058260    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj"]
Nov 03 19:07:02.105293 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:02.105254    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6cb1d90-n2qqj"]
Nov 03 19:07:03.213584 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:03.213547    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:03.334931 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:03.334895    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f331127f-493e-45a6-a834-747c701fab2c" path="/var/lib/kubelet/pods/f331127f-493e-45a6-a834-747c701fab2c/volumes"
Nov 03 19:07:03.503224 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:03.503105    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:03.685017 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:03.684985    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:06.861845 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:07:06.861804    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:07:08.094082 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:08.094045    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:08.271804 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:08.271763    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:08.452638 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:08.452546    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:12.999320 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:12.999286    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:17.923415 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:17.923324    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:22.889713 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:22.889677    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:23.732553 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:23.732517    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:26.560065 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:26.560032    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:27.762921 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:27.762889    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:29.842972 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:29.842939    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:30.101608 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:30.101519    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb"]
Nov 03 19:07:30.201085 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:30.201053    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr"]
Nov 03 19:07:30.465564 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:30.465523    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:30.646124 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:30.646093    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:32.326918 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:32.326888    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:35.102401 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:35.102322    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb" podUID="c34bd238-0f1b-4bb5-af7d-539c417dd6b6" containerName="payload-processing" containerID="cri-o://c717b0d5c74f13ffe7350e042de7dd96993ce43a05ff79735628c8ff07b95352" gracePeriod=25
Nov 03 19:07:35.201777 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:35.201722    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr" podUID="65641dd6-6289-455b-be56-40c85b340edf" containerName="payload-pre-processing" containerID="cri-o://2774544f1d2f5a6eec42f1283537f33d6fd033d0a26511346dd3bdb2317cf589" gracePeriod=25
Nov 03 19:07:35.307217 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:35.307182    6059 generic.go:356] "Generic (PLEG): container finished" podID="c34bd238-0f1b-4bb5-af7d-539c417dd6b6" containerID="c717b0d5c74f13ffe7350e042de7dd96993ce43a05ff79735628c8ff07b95352" exitCode=0
Nov 03 19:07:35.307436 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:35.307262    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb" event={"ID":"c34bd238-0f1b-4bb5-af7d-539c417dd6b6","Type":"ContainerDied","Data":"c717b0d5c74f13ffe7350e042de7dd96993ce43a05ff79735628c8ff07b95352"}
Nov 03 19:07:35.359659 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:35.359594    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb"
Nov 03 19:07:35.469140 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:35.469112    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr"
Nov 03 19:07:35.470801 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:35.470275    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c34bd238-0f1b-4bb5-af7d-539c417dd6b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c34bd238-0f1b-4bb5-af7d-539c417dd6b6-plugins-config-volume\") pod \"c34bd238-0f1b-4bb5-af7d-539c417dd6b6\" (UID: \"c34bd238-0f1b-4bb5-af7d-539c417dd6b6\") "
Nov 03 19:07:35.470801 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:35.470396    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34bd238-0f1b-4bb5-af7d-539c417dd6b6-kube-api-access-cjkn6\" (UniqueName: \"kubernetes.io/projected/c34bd238-0f1b-4bb5-af7d-539c417dd6b6-kube-api-access-cjkn6\") pod \"c34bd238-0f1b-4bb5-af7d-539c417dd6b6\" (UID: \"c34bd238-0f1b-4bb5-af7d-539c417dd6b6\") "
Nov 03 19:07:35.471269 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:35.471235    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c34bd238-0f1b-4bb5-af7d-539c417dd6b6-plugins-config-volume" pod "c34bd238-0f1b-4bb5-af7d-539c417dd6b6" (UID: "c34bd238-0f1b-4bb5-af7d-539c417dd6b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:07:35.473281 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:35.473253    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34bd238-0f1b-4bb5-af7d-539c417dd6b6-kube-api-access-cjkn6" pod "c34bd238-0f1b-4bb5-af7d-539c417dd6b6" (UID: "c34bd238-0f1b-4bb5-af7d-539c417dd6b6"). InnerVolumeSpecName "kube-api-access-cjkn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:07:35.475971 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:35.475939    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:35.571505 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:35.571472    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65641dd6-6289-455b-be56-40c85b340edf-kube-api-access-6fx2h\" (UniqueName: \"kubernetes.io/projected/65641dd6-6289-455b-be56-40c85b340edf-kube-api-access-6fx2h\") pod \"65641dd6-6289-455b-be56-40c85b340edf\" (UID: \"65641dd6-6289-455b-be56-40c85b340edf\") "
Nov 03 19:07:35.571694 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:35.571536    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65641dd6-6289-455b-be56-40c85b340edf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65641dd6-6289-455b-be56-40c85b340edf-plugins-config-volume\") pod \"65641dd6-6289-455b-be56-40c85b340edf\" (UID: \"65641dd6-6289-455b-be56-40c85b340edf\") "
Nov 03 19:07:35.571858 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:35.571845    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjkn6\" (UniqueName: \"kubernetes.io/projected/c34bd238-0f1b-4bb5-af7d-539c417dd6b6-kube-api-access-cjkn6\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:35.571895 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:35.571860    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c34bd238-0f1b-4bb5-af7d-539c417dd6b6-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:35.571936 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:35.571909    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65641dd6-6289-455b-be56-40c85b340edf-plugins-config-volume" pod "65641dd6-6289-455b-be56-40c85b340edf" (UID: "65641dd6-6289-455b-be56-40c85b340edf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:07:35.573649 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:35.573631    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65641dd6-6289-455b-be56-40c85b340edf-kube-api-access-6fx2h" pod "65641dd6-6289-455b-be56-40c85b340edf" (UID: "65641dd6-6289-455b-be56-40c85b340edf"). InnerVolumeSpecName "kube-api-access-6fx2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:07:35.672961 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:35.672875    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fx2h\" (UniqueName: \"kubernetes.io/projected/65641dd6-6289-455b-be56-40c85b340edf-kube-api-access-6fx2h\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:35.672961 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:35.672908    6059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65641dd6-6289-455b-be56-40c85b340edf-plugins-config-volume\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:36.313847 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:36.313804    6059 generic.go:356] "Generic (PLEG): container finished" podID="65641dd6-6289-455b-be56-40c85b340edf" containerID="2774544f1d2f5a6eec42f1283537f33d6fd033d0a26511346dd3bdb2317cf589" exitCode=0
Nov 03 19:07:36.313847 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:36.313849    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr"
Nov 03 19:07:36.314352 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:36.313878    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr" event={"ID":"65641dd6-6289-455b-be56-40c85b340edf","Type":"ContainerDied","Data":"2774544f1d2f5a6eec42f1283537f33d6fd033d0a26511346dd3bdb2317cf589"}
Nov 03 19:07:36.314352 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:36.313930    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr" event={"ID":"65641dd6-6289-455b-be56-40c85b340edf","Type":"ContainerDied","Data":"b16d923905228873d5d36383894dd95a6d1cf6dff000f26aa0f91e7ba5996a82"}
Nov 03 19:07:36.314352 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:36.313953    6059 scope.go:122] "RemoveContainer" containerID="2774544f1d2f5a6eec42f1283537f33d6fd033d0a26511346dd3bdb2317cf589"
Nov 03 19:07:36.315425 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:36.315403    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb" event={"ID":"c34bd238-0f1b-4bb5-af7d-539c417dd6b6","Type":"ContainerDied","Data":"9b8f5de12c0e816269d5b3267ab54c189387ee6e5eaf0a4f5bd64a1127797c4d"}
Nov 03 19:07:36.315515 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:36.315442    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb"
Nov 03 19:07:36.333813 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:36.333792    6059 scope.go:122] "RemoveContainer" containerID="2774544f1d2f5a6eec42f1283537f33d6fd033d0a26511346dd3bdb2317cf589"
Nov 03 19:07:36.334143 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:07:36.334123    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2774544f1d2f5a6eec42f1283537f33d6fd033d0a26511346dd3bdb2317cf589\": container with ID starting with 2774544f1d2f5a6eec42f1283537f33d6fd033d0a26511346dd3bdb2317cf589 not found: ID does not exist" containerID="2774544f1d2f5a6eec42f1283537f33d6fd033d0a26511346dd3bdb2317cf589"
Nov 03 19:07:36.334198 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:36.334151    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2774544f1d2f5a6eec42f1283537f33d6fd033d0a26511346dd3bdb2317cf589"} err="failed to get container status \"2774544f1d2f5a6eec42f1283537f33d6fd033d0a26511346dd3bdb2317cf589\": rpc error: code = NotFound desc = could not find container \"2774544f1d2f5a6eec42f1283537f33d6fd033d0a26511346dd3bdb2317cf589\": container with ID starting with 2774544f1d2f5a6eec42f1283537f33d6fd033d0a26511346dd3bdb2317cf589 not found: ID does not exist"
Nov 03 19:07:36.334198 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:36.334170    6059 scope.go:122] "RemoveContainer" containerID="c717b0d5c74f13ffe7350e042de7dd96993ce43a05ff79735628c8ff07b95352"
Nov 03 19:07:36.469801 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:36.469762    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr"]
Nov 03 19:07:36.517854 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:36.517816    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88025f84-79cb5b5b79-dsbpr"]
Nov 03 19:07:36.620909 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:36.620872    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb"]
Nov 03 19:07:36.625012 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:36.624978    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88025f84-745cbc88b5-89thb"]
Nov 03 19:07:36.862605 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:07:36.862511    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:07:37.334857 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:37.334826    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65641dd6-6289-455b-be56-40c85b340edf" path="/var/lib/kubelet/pods/65641dd6-6289-455b-be56-40c85b340edf/volumes"
Nov 03 19:07:37.335264 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:37.335221    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c34bd238-0f1b-4bb5-af7d-539c417dd6b6" path="/var/lib/kubelet/pods/c34bd238-0f1b-4bb5-af7d-539c417dd6b6/volumes"
Nov 03 19:07:40.337896 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:40.337861    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 19:07:40.349643 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:40.349606    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 19:07:40.361628 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:40.361597    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 19:07:40.370936 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:40.370909    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 19:07:41.611828 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:41.611795    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:41.803450 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:41.803414    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:43.206289 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:43.206242    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:43.811208 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:43.811167    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"]
Nov 03 19:07:43.811439 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:43.811380    6059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk" podUID="dc09fa62-ea2b-4983-af9c-f7d318a908e4" containerName="istio-proxy" containerID="cri-o://73a21919d9c8d2ef2d78f63e691256c35196abc08e074f37b83440b7def2b80e" gracePeriod=30
Nov 03 19:07:43.855221 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:43.855164    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:49.069703 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.069672    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:07:49.217598 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.217561    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-workload-certs\") pod \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") "
Nov 03 19:07:49.217800 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.217611    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-envoy\") pod \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") "
Nov 03 19:07:49.217800 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.217649    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-credential-socket\") pod \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") "
Nov 03 19:07:49.217800 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.217670    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-podinfo\") pod \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") "
Nov 03 19:07:49.217800 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.217736    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc09fa62-ea2b-4983-af9c-f7d318a908e4-kube-api-access-7vd42\" (UniqueName: \"kubernetes.io/projected/dc09fa62-ea2b-4983-af9c-f7d318a908e4-kube-api-access-7vd42\") pod \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") "
Nov 03 19:07:49.217800 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.217779    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-token\" (UniqueName: \"kubernetes.io/projected/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-token\") pod \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") "
Nov 03 19:07:49.218103 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.217812    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istiod-ca-cert\") pod \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") "
Nov 03 19:07:49.218103 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.217840    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-workload-socket\") pod \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") "
Nov 03 19:07:49.218103 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.217865    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-workload-certs" pod "dc09fa62-ea2b-4983-af9c-f7d318a908e4" (UID: "dc09fa62-ea2b-4983-af9c-f7d318a908e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:07:49.218103 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.217906    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-data\") pod \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\" (UID: \"dc09fa62-ea2b-4983-af9c-f7d318a908e4\") "
Nov 03 19:07:49.218103 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.217979    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-credential-socket" pod "dc09fa62-ea2b-4983-af9c-f7d318a908e4" (UID: "dc09fa62-ea2b-4983-af9c-f7d318a908e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:07:49.218446 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.218169    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-workload-socket" pod "dc09fa62-ea2b-4983-af9c-f7d318a908e4" (UID: "dc09fa62-ea2b-4983-af9c-f7d318a908e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:07:49.218446 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.218294    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-data" pod "dc09fa62-ea2b-4983-af9c-f7d318a908e4" (UID: "dc09fa62-ea2b-4983-af9c-f7d318a908e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:07:49.218446 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.218335    6059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-workload-certs\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:49.218446 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.218382    6059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-credential-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:49.218446 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.218399    6059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-workload-socket\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:49.218446 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.218336    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istiod-ca-cert" pod "dc09fa62-ea2b-4983-af9c-f7d318a908e4" (UID: "dc09fa62-ea2b-4983-af9c-f7d318a908e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:07:49.220153 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.220130    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-envoy" pod "dc09fa62-ea2b-4983-af9c-f7d318a908e4" (UID: "dc09fa62-ea2b-4983-af9c-f7d318a908e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:07:49.220264 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.220147    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-token" pod "dc09fa62-ea2b-4983-af9c-f7d318a908e4" (UID: "dc09fa62-ea2b-4983-af9c-f7d318a908e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:07:49.220334 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.220317    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-podinfo" pod "dc09fa62-ea2b-4983-af9c-f7d318a908e4" (UID: "dc09fa62-ea2b-4983-af9c-f7d318a908e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:07:49.220404 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.220335    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc09fa62-ea2b-4983-af9c-f7d318a908e4-kube-api-access-7vd42" pod "dc09fa62-ea2b-4983-af9c-f7d318a908e4" (UID: "dc09fa62-ea2b-4983-af9c-f7d318a908e4"). InnerVolumeSpecName "kube-api-access-7vd42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:07:49.319525 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.319484    6059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-data\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:49.319525 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.319516    6059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-envoy\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:49.319525 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.319526    6059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-podinfo\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:49.319525 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.319536    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vd42\" (UniqueName: \"kubernetes.io/projected/dc09fa62-ea2b-4983-af9c-f7d318a908e4-kube-api-access-7vd42\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:49.319835 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.319547    6059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istio-token\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:49.319835 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.319556    6059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc09fa62-ea2b-4983-af9c-f7d318a908e4-istiod-ca-cert\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:49.392052 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.392019    6059 generic.go:356] "Generic (PLEG): container finished" podID="dc09fa62-ea2b-4983-af9c-f7d318a908e4" containerID="73a21919d9c8d2ef2d78f63e691256c35196abc08e074f37b83440b7def2b80e" exitCode=0
Nov 03 19:07:49.392234 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.392065    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"
Nov 03 19:07:49.392234 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.392091    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk" event={"ID":"dc09fa62-ea2b-4983-af9c-f7d318a908e4","Type":"ContainerDied","Data":"73a21919d9c8d2ef2d78f63e691256c35196abc08e074f37b83440b7def2b80e"}
Nov 03 19:07:49.392234 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.392135    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk" event={"ID":"dc09fa62-ea2b-4983-af9c-f7d318a908e4","Type":"ContainerDied","Data":"521885a1be5889f9f38776eb1164b541b18535f947deeccd3c6577c3bdf2ce1b"}
Nov 03 19:07:49.392234 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.392157    6059 scope.go:122] "RemoveContainer" containerID="73a21919d9c8d2ef2d78f63e691256c35196abc08e074f37b83440b7def2b80e"
Nov 03 19:07:49.401545 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.401522    6059 scope.go:122] "RemoveContainer" containerID="5a42c22faf4c34933e40ea822374d666db75b68ec94489f87572c8a738e3c0d6"
Nov 03 19:07:49.410251 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.410224    6059 scope.go:122] "RemoveContainer" containerID="73a21919d9c8d2ef2d78f63e691256c35196abc08e074f37b83440b7def2b80e"
Nov 03 19:07:49.410687 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:07:49.410664    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a21919d9c8d2ef2d78f63e691256c35196abc08e074f37b83440b7def2b80e\": container with ID starting with 73a21919d9c8d2ef2d78f63e691256c35196abc08e074f37b83440b7def2b80e not found: ID does not exist" containerID="73a21919d9c8d2ef2d78f63e691256c35196abc08e074f37b83440b7def2b80e"
Nov 03 19:07:49.410757 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.410698    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a21919d9c8d2ef2d78f63e691256c35196abc08e074f37b83440b7def2b80e"} err="failed to get container status \"73a21919d9c8d2ef2d78f63e691256c35196abc08e074f37b83440b7def2b80e\": rpc error: code = NotFound desc = could not find container \"73a21919d9c8d2ef2d78f63e691256c35196abc08e074f37b83440b7def2b80e\": container with ID starting with 73a21919d9c8d2ef2d78f63e691256c35196abc08e074f37b83440b7def2b80e not found: ID does not exist"
Nov 03 19:07:49.410757 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.410717    6059 scope.go:122] "RemoveContainer" containerID="5a42c22faf4c34933e40ea822374d666db75b68ec94489f87572c8a738e3c0d6"
Nov 03 19:07:49.411035 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:07:49.411011    6059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a42c22faf4c34933e40ea822374d666db75b68ec94489f87572c8a738e3c0d6\": container with ID starting with 5a42c22faf4c34933e40ea822374d666db75b68ec94489f87572c8a738e3c0d6 not found: ID does not exist" containerID="5a42c22faf4c34933e40ea822374d666db75b68ec94489f87572c8a738e3c0d6"
Nov 03 19:07:49.411134 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.411038    6059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a42c22faf4c34933e40ea822374d666db75b68ec94489f87572c8a738e3c0d6"} err="failed to get container status \"5a42c22faf4c34933e40ea822374d666db75b68ec94489f87572c8a738e3c0d6\": rpc error: code = NotFound desc = could not find container \"5a42c22faf4c34933e40ea822374d666db75b68ec94489f87572c8a738e3c0d6\": container with ID starting with 5a42c22faf4c34933e40ea822374d666db75b68ec94489f87572c8a738e3c0d6 not found: ID does not exist"
Nov 03 19:07:49.548503 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.548415    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"]
Nov 03 19:07:49.552022 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:49.551994    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-88025f84-openshift-default-8679d6488f-jq7zk"]
Nov 03 19:07:51.335121 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:07:51.335079    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc09fa62-ea2b-4983-af9c-f7d318a908e4" path="/var/lib/kubelet/pods/dc09fa62-ea2b-4983-af9c-f7d318a908e4/volumes"
Nov 03 19:08:00.152530 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:00.152479    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-c55ww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:08:00.153158 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:00.153120    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww" podUID="acc06524-297b-4b57-b0a2-4b2a92d4a00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:00.237496 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:00.237454    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bwg8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:08:00.237792 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:00.237762    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p" podUID="c17b0b8a-a3a3-4824-acc3-ce83fb644937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:00.264514 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:00.264471    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xkkj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:08:00.264808 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:00.264779    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4" podUID="c24c1306-3d52-4b39-af5d-5b769ec03a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:01.146301 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:01.145646    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-csr7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:08:01.146301 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:01.145796    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q" podUID="3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:06.862327 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:08:06.862283    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:08:08.531156 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:08.531116    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:08:10.804934 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:10.804896    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3390c81f-gwzqj"]
Nov 03 19:08:10.805429 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:10.805345    6059 cpu_manager.go:414] "containerMap: removing container" podUID="65641dd6-6289-455b-be56-40c85b340edf" containerName="payload-pre-processing"
Nov 03 19:08:10.805429 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:10.805375    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65641dd6-6289-455b-be56-40c85b340edf" containerName="payload-pre-processing"
Nov 03 19:08:10.805429 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:10.805415    6059 cpu_manager.go:414] "containerMap: removing container" podUID="c34bd238-0f1b-4bb5-af7d-539c417dd6b6" containerName="payload-processing"
Nov 03 19:08:10.805429 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:10.805421    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34bd238-0f1b-4bb5-af7d-539c417dd6b6" containerName="payload-processing"
Nov 03 19:08:10.805429 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:10.805429    6059 cpu_manager.go:414] "containerMap: removing container" podUID="dc09fa62-ea2b-4983-af9c-f7d318a908e4" containerName="istio-proxy"
Nov 03 19:08:10.805429 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:10.805434    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc09fa62-ea2b-4983-af9c-f7d318a908e4" containerName="istio-proxy"
Nov 03 19:08:10.805703 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:10.805449    6059 cpu_manager.go:414] "containerMap: removing container" podUID="dc09fa62-ea2b-4983-af9c-f7d318a908e4" containerName="istio-proxy"
Nov 03 19:08:10.805703 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:10.805453    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc09fa62-ea2b-4983-af9c-f7d318a908e4" containerName="istio-proxy"
Nov 03 19:08:10.805703 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:10.805522    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34bd238-0f1b-4bb5-af7d-539c417dd6b6" containerName="payload-processing"
Nov 03 19:08:10.805703 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:10.805528    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc09fa62-ea2b-4983-af9c-f7d318a908e4" containerName="istio-proxy"
Nov 03 19:08:10.805703 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:10.805536    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc09fa62-ea2b-4983-af9c-f7d318a908e4" containerName="istio-proxy"
Nov 03 19:08:10.805703 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:10.805543    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="65641dd6-6289-455b-be56-40c85b340edf" containerName="payload-pre-processing"
Nov 03 19:08:10.808856 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:10.808834    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3390c81f-gwzqj"
Nov 03 19:08:10.860829 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:10.860794    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:08:10.861069 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:10.860877    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:08:10.908376 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:10.908315    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:08:10.908560 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:10.908443    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:08:10.910568 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:10.910541    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3390c81f-gwzqj"]
Nov 03 19:08:10.934455 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:10.934416    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b64a378-7575-4d35-acec-4bf6a3b41959-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3390c81f-gwzqj\" (UID: \"8b64a378-7575-4d35-acec-4bf6a3b41959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3390c81f-gwzqj"
Nov 03 19:08:10.934624 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:10.934486    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72dcg\" (UniqueName: \"kubernetes.io/projected/8b64a378-7575-4d35-acec-4bf6a3b41959-kube-api-access-72dcg\") pod \"maas-api-revoke-sub-3390c81f-gwzqj\" (UID: \"8b64a378-7575-4d35-acec-4bf6a3b41959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3390c81f-gwzqj"
Nov 03 19:08:10.934624 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:10.934589    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b64a378-7575-4d35-acec-4bf6a3b41959-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3390c81f-gwzqj\" (UID: \"8b64a378-7575-4d35-acec-4bf6a3b41959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3390c81f-gwzqj"
Nov 03 19:08:11.035666 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:11.035623    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72dcg\" (UniqueName: \"kubernetes.io/projected/8b64a378-7575-4d35-acec-4bf6a3b41959-kube-api-access-72dcg\") pod \"maas-api-revoke-sub-3390c81f-gwzqj\" (UID: \"8b64a378-7575-4d35-acec-4bf6a3b41959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3390c81f-gwzqj"
Nov 03 19:08:11.035887 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:11.035691    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b64a378-7575-4d35-acec-4bf6a3b41959-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3390c81f-gwzqj\" (UID: \"8b64a378-7575-4d35-acec-4bf6a3b41959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3390c81f-gwzqj"
Nov 03 19:08:11.035887 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:11.035827    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b64a378-7575-4d35-acec-4bf6a3b41959-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3390c81f-gwzqj\" (UID: \"8b64a378-7575-4d35-acec-4bf6a3b41959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3390c81f-gwzqj"
Nov 03 19:08:11.036645 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:11.036620    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b64a378-7575-4d35-acec-4bf6a3b41959-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3390c81f-gwzqj\" (UID: \"8b64a378-7575-4d35-acec-4bf6a3b41959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3390c81f-gwzqj"
Nov 03 19:08:11.038077 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:11.038055    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b64a378-7575-4d35-acec-4bf6a3b41959-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3390c81f-gwzqj\" (UID: \"8b64a378-7575-4d35-acec-4bf6a3b41959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3390c81f-gwzqj"
Nov 03 19:08:11.094728 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:11.094653    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72dcg\" (UniqueName: \"kubernetes.io/projected/8b64a378-7575-4d35-acec-4bf6a3b41959-kube-api-access-72dcg\") pod \"maas-api-revoke-sub-3390c81f-gwzqj\" (UID: \"8b64a378-7575-4d35-acec-4bf6a3b41959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3390c81f-gwzqj"
Nov 03 19:08:11.122191 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:11.122151    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3390c81f-gwzqj"
Nov 03 19:08:11.492460 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:11.492434    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3390c81f-gwzqj"]
Nov 03 19:08:11.494851 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:08:11.494824    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b64a378_7575_4d35_acec_4bf6a3b41959.slice/crio-a8d3a34ac87797021f879f5b051b41fbdee9caf012ffa165411f98da0e7bd716 WatchSource:0}: Error finding container a8d3a34ac87797021f879f5b051b41fbdee9caf012ffa165411f98da0e7bd716: Status 404 returned error can't find the container with id a8d3a34ac87797021f879f5b051b41fbdee9caf012ffa165411f98da0e7bd716
Nov 03 19:08:11.653017 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:11.652979    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3390c81f-gwzqj" event={"ID":"8b64a378-7575-4d35-acec-4bf6a3b41959","Type":"ContainerStarted","Data":"a8d3a34ac87797021f879f5b051b41fbdee9caf012ffa165411f98da0e7bd716"}
Nov 03 19:08:12.660901 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:12.660801    6059 generic.go:356] "Generic (PLEG): container finished" podID="8b64a378-7575-4d35-acec-4bf6a3b41959" containerID="eb554b31a669bad9dde860368f3da87f2c48576e3b176b956a28d19d906276e0" exitCode=0
Nov 03 19:08:12.660901 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:12.660844    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3390c81f-gwzqj" event={"ID":"8b64a378-7575-4d35-acec-4bf6a3b41959","Type":"ContainerDied","Data":"eb554b31a669bad9dde860368f3da87f2c48576e3b176b956a28d19d906276e0"}
Nov 03 19:08:13.807053 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:13.807018    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3390c81f-gwzqj"
Nov 03 19:08:13.969096 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:13.969059    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b64a378-7575-4d35-acec-4bf6a3b41959-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b64a378-7575-4d35-acec-4bf6a3b41959-maas-api-xks-ca\") pod \"8b64a378-7575-4d35-acec-4bf6a3b41959\" (UID: \"8b64a378-7575-4d35-acec-4bf6a3b41959\") "
Nov 03 19:08:13.969249 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:13.969191    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b64a378-7575-4d35-acec-4bf6a3b41959-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b64a378-7575-4d35-acec-4bf6a3b41959-maas-api-openshift-ca\") pod \"8b64a378-7575-4d35-acec-4bf6a3b41959\" (UID: \"8b64a378-7575-4d35-acec-4bf6a3b41959\") "
Nov 03 19:08:13.969315 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:13.969257    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b64a378-7575-4d35-acec-4bf6a3b41959-kube-api-access-72dcg\" (UniqueName: \"kubernetes.io/projected/8b64a378-7575-4d35-acec-4bf6a3b41959-kube-api-access-72dcg\") pod \"8b64a378-7575-4d35-acec-4bf6a3b41959\" (UID: \"8b64a378-7575-4d35-acec-4bf6a3b41959\") "
Nov 03 19:08:13.969598 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:13.969567    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b64a378-7575-4d35-acec-4bf6a3b41959-maas-api-openshift-ca" pod "8b64a378-7575-4d35-acec-4bf6a3b41959" (UID: "8b64a378-7575-4d35-acec-4bf6a3b41959"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:08:13.971265 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:13.971239    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b64a378-7575-4d35-acec-4bf6a3b41959-kube-api-access-72dcg" pod "8b64a378-7575-4d35-acec-4bf6a3b41959" (UID: "8b64a378-7575-4d35-acec-4bf6a3b41959"). InnerVolumeSpecName "kube-api-access-72dcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:08:13.971265 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:13.971256    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b64a378-7575-4d35-acec-4bf6a3b41959-maas-api-xks-ca" pod "8b64a378-7575-4d35-acec-4bf6a3b41959" (UID: "8b64a378-7575-4d35-acec-4bf6a3b41959"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:08:14.070891 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:14.070846    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b64a378-7575-4d35-acec-4bf6a3b41959-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:14.070891 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:14.070884    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72dcg\" (UniqueName: \"kubernetes.io/projected/8b64a378-7575-4d35-acec-4bf6a3b41959-kube-api-access-72dcg\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:14.070891 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:14.070898    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b64a378-7575-4d35-acec-4bf6a3b41959-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:14.672893 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:14.672858    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3390c81f-gwzqj" event={"ID":"8b64a378-7575-4d35-acec-4bf6a3b41959","Type":"ContainerDied","Data":"a8d3a34ac87797021f879f5b051b41fbdee9caf012ffa165411f98da0e7bd716"}
Nov 03 19:08:14.673084 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:14.672908    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8d3a34ac87797021f879f5b051b41fbdee9caf012ffa165411f98da0e7bd716"
Nov 03 19:08:14.673084 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:14.672871    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3390c81f-gwzqj"
Nov 03 19:08:21.558018 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:21.557985    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn"]
Nov 03 19:08:21.558486 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:21.558451    6059 cpu_manager.go:414] "containerMap: removing container" podUID="8b64a378-7575-4d35-acec-4bf6a3b41959" containerName="revoke-keys"
Nov 03 19:08:21.558486 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:21.558465    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b64a378-7575-4d35-acec-4bf6a3b41959" containerName="revoke-keys"
Nov 03 19:08:21.558577 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:21.558566    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b64a378-7575-4d35-acec-4bf6a3b41959" containerName="revoke-keys"
Nov 03 19:08:21.563319 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:21.563284    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn"
Nov 03 19:08:21.616637 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:21.616599    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn"]
Nov 03 19:08:21.648511 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:21.648475    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ae8d86b-4382-417f-9eb5-e2709576686f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccdf6d1d-95nqn\" (UID: \"9ae8d86b-4382-417f-9eb5-e2709576686f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn"
Nov 03 19:08:21.648695 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:21.648522    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ae8d86b-4382-417f-9eb5-e2709576686f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccdf6d1d-95nqn\" (UID: \"9ae8d86b-4382-417f-9eb5-e2709576686f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn"
Nov 03 19:08:21.648695 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:21.648686    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjnb2\" (UniqueName: \"kubernetes.io/projected/9ae8d86b-4382-417f-9eb5-e2709576686f-kube-api-access-qjnb2\") pod \"maas-api-revoke-sub-ccdf6d1d-95nqn\" (UID: \"9ae8d86b-4382-417f-9eb5-e2709576686f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn"
Nov 03 19:08:21.660998 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:21.660963    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:08:21.660998 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:21.660962    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:08:21.661212 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:21.660998    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:08:21.661212 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:21.661010    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:08:21.749632 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:21.749597    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ae8d86b-4382-417f-9eb5-e2709576686f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccdf6d1d-95nqn\" (UID: \"9ae8d86b-4382-417f-9eb5-e2709576686f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn"
Nov 03 19:08:21.749835 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:21.749652    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ae8d86b-4382-417f-9eb5-e2709576686f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccdf6d1d-95nqn\" (UID: \"9ae8d86b-4382-417f-9eb5-e2709576686f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn"
Nov 03 19:08:21.749835 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:21.749732    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjnb2\" (UniqueName: \"kubernetes.io/projected/9ae8d86b-4382-417f-9eb5-e2709576686f-kube-api-access-qjnb2\") pod \"maas-api-revoke-sub-ccdf6d1d-95nqn\" (UID: \"9ae8d86b-4382-417f-9eb5-e2709576686f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn"
Nov 03 19:08:21.750415 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:21.750391    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ae8d86b-4382-417f-9eb5-e2709576686f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccdf6d1d-95nqn\" (UID: \"9ae8d86b-4382-417f-9eb5-e2709576686f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn"
Nov 03 19:08:21.751988 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:21.751969    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ae8d86b-4382-417f-9eb5-e2709576686f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccdf6d1d-95nqn\" (UID: \"9ae8d86b-4382-417f-9eb5-e2709576686f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn"
Nov 03 19:08:21.808138 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:21.808062    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjnb2\" (UniqueName: \"kubernetes.io/projected/9ae8d86b-4382-417f-9eb5-e2709576686f-kube-api-access-qjnb2\") pod \"maas-api-revoke-sub-ccdf6d1d-95nqn\" (UID: \"9ae8d86b-4382-417f-9eb5-e2709576686f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn"
Nov 03 19:08:21.881059 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:21.881017    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn"
Nov 03 19:08:22.250623 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:22.250596    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn"]
Nov 03 19:08:22.253556 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:08:22.253513    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae8d86b_4382_417f_9eb5_e2709576686f.slice/crio-19669c76bf31fb4498d7c9d1b33b522d15f18287f3e9a4bc6e0bd5da53ed11be WatchSource:0}: Error finding container 19669c76bf31fb4498d7c9d1b33b522d15f18287f3e9a4bc6e0bd5da53ed11be: Status 404 returned error can't find the container with id 19669c76bf31fb4498d7c9d1b33b522d15f18287f3e9a4bc6e0bd5da53ed11be
Nov 03 19:08:22.715098 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:22.715059    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn" event={"ID":"9ae8d86b-4382-417f-9eb5-e2709576686f","Type":"ContainerStarted","Data":"df7c48786840ada42191193b6e1d166a61485f6455a8de1ecd23899d11d9916c"}
Nov 03 19:08:22.715539 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:22.715125    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn" event={"ID":"9ae8d86b-4382-417f-9eb5-e2709576686f","Type":"ContainerStarted","Data":"19669c76bf31fb4498d7c9d1b33b522d15f18287f3e9a4bc6e0bd5da53ed11be"}
Nov 03 19:08:22.827090 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:22.826976    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn" podStartSLOduration=1.826960871 podStartE2EDuration="1.826960871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:22.825238742 +0000 UTC m=+6643.622951523" watchObservedRunningTime="2026-11-03 19:08:22.826960871 +0000 UTC m=+6643.624673651"
Nov 03 19:08:23.722374 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:23.722321    6059 generic.go:356] "Generic (PLEG): container finished" podID="9ae8d86b-4382-417f-9eb5-e2709576686f" containerID="df7c48786840ada42191193b6e1d166a61485f6455a8de1ecd23899d11d9916c" exitCode=0
Nov 03 19:08:23.722794 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:23.722391    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn" event={"ID":"9ae8d86b-4382-417f-9eb5-e2709576686f","Type":"ContainerDied","Data":"df7c48786840ada42191193b6e1d166a61485f6455a8de1ecd23899d11d9916c"}
Nov 03 19:08:24.865683 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:24.865656    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn"
Nov 03 19:08:24.986874 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:24.986791    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ae8d86b-4382-417f-9eb5-e2709576686f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ae8d86b-4382-417f-9eb5-e2709576686f-maas-api-xks-ca\") pod \"9ae8d86b-4382-417f-9eb5-e2709576686f\" (UID: \"9ae8d86b-4382-417f-9eb5-e2709576686f\") "
Nov 03 19:08:24.986874 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:24.986846    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ae8d86b-4382-417f-9eb5-e2709576686f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ae8d86b-4382-417f-9eb5-e2709576686f-maas-api-openshift-ca\") pod \"9ae8d86b-4382-417f-9eb5-e2709576686f\" (UID: \"9ae8d86b-4382-417f-9eb5-e2709576686f\") "
Nov 03 19:08:24.987115 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:24.986906    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae8d86b-4382-417f-9eb5-e2709576686f-kube-api-access-qjnb2\" (UniqueName: \"kubernetes.io/projected/9ae8d86b-4382-417f-9eb5-e2709576686f-kube-api-access-qjnb2\") pod \"9ae8d86b-4382-417f-9eb5-e2709576686f\" (UID: \"9ae8d86b-4382-417f-9eb5-e2709576686f\") "
Nov 03 19:08:24.987243 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:24.987213    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae8d86b-4382-417f-9eb5-e2709576686f-maas-api-openshift-ca" pod "9ae8d86b-4382-417f-9eb5-e2709576686f" (UID: "9ae8d86b-4382-417f-9eb5-e2709576686f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:08:24.989135 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:24.989103    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae8d86b-4382-417f-9eb5-e2709576686f-kube-api-access-qjnb2" pod "9ae8d86b-4382-417f-9eb5-e2709576686f" (UID: "9ae8d86b-4382-417f-9eb5-e2709576686f"). InnerVolumeSpecName "kube-api-access-qjnb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:08:24.989290 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:24.989187    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ae8d86b-4382-417f-9eb5-e2709576686f-maas-api-xks-ca" pod "9ae8d86b-4382-417f-9eb5-e2709576686f" (UID: "9ae8d86b-4382-417f-9eb5-e2709576686f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:08:25.088723 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:25.088681    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ae8d86b-4382-417f-9eb5-e2709576686f-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:25.088723 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:25.088715    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ae8d86b-4382-417f-9eb5-e2709576686f-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:25.088723 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:25.088725    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjnb2\" (UniqueName: \"kubernetes.io/projected/9ae8d86b-4382-417f-9eb5-e2709576686f-kube-api-access-qjnb2\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:25.734766 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:25.734734    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn"
Nov 03 19:08:25.734932 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:25.734767    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn" event={"ID":"9ae8d86b-4382-417f-9eb5-e2709576686f","Type":"ContainerDied","Data":"19669c76bf31fb4498d7c9d1b33b522d15f18287f3e9a4bc6e0bd5da53ed11be"}
Nov 03 19:08:25.734932 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:25.734811    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19669c76bf31fb4498d7c9d1b33b522d15f18287f3e9a4bc6e0bd5da53ed11be"
Nov 03 19:08:36.866027 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:08:36.865977    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:08:44.080747 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:44.080696    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k"]
Nov 03 19:08:44.099030 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:44.098991    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af40a079-f7g9k"]
Nov 03 19:08:44.569942 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:44.569908    6059 scope.go:122] "RemoveContainer" containerID="cf056963ff8fd4b3bfa39ee7f97a69e1ad5a1afaa19329e9d502cabb34c75950"
Nov 03 19:08:45.180141 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:45.179542    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-c55ww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:08:45.180141 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:45.179635    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww" podUID="acc06524-297b-4b57-b0a2-4b2a92d4a00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:45.239522 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:45.239486    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bwg8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:08:45.239819 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:45.239790    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p" podUID="c17b0b8a-a3a3-4824-acc3-ce83fb644937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:45.260597 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:45.260465    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xkkj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:08:45.260597 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:45.260536    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4" podUID="c24c1306-3d52-4b39-af5d-5b769ec03a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:45.352209 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:45.348699    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c4c69dc-cdcf-4854-b924-34202b02d0be" path="/var/lib/kubelet/pods/3c4c69dc-cdcf-4854-b924-34202b02d0be/volumes"
Nov 03 19:08:55.710055 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:55.710019    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:08:55.895510 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:55.895460    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:08:57.964289 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:57.964249    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9925ccc3-htbcg"]
Nov 03 19:08:57.964773 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:57.964757    6059 cpu_manager.go:414] "containerMap: removing container" podUID="9ae8d86b-4382-417f-9eb5-e2709576686f" containerName="revoke-keys"
Nov 03 19:08:57.964820 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:57.964774    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ae8d86b-4382-417f-9eb5-e2709576686f" containerName="revoke-keys"
Nov 03 19:08:57.964861 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:57.964857    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ae8d86b-4382-417f-9eb5-e2709576686f" containerName="revoke-keys"
Nov 03 19:08:57.969853 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:57.969829    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9925ccc3-htbcg"
Nov 03 19:08:58.021797 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:58.021760    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9925ccc3-htbcg"]
Nov 03 19:08:58.066291 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:58.066255    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:08:58.066500 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:58.066294    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:08:58.066500 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:58.066310    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:08:58.066648 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:58.066633    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:08:58.146149 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:58.146108    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxfn9\" (UniqueName: \"kubernetes.io/projected/9249e174-6085-4a7d-84ae-fd8ef208ced0-kube-api-access-kxfn9\") pod \"maas-api-revoke-sub-9925ccc3-htbcg\" (UID: \"9249e174-6085-4a7d-84ae-fd8ef208ced0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9925ccc3-htbcg"
Nov 03 19:08:58.146336 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:58.146232    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9249e174-6085-4a7d-84ae-fd8ef208ced0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9925ccc3-htbcg\" (UID: \"9249e174-6085-4a7d-84ae-fd8ef208ced0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9925ccc3-htbcg"
Nov 03 19:08:58.146336 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:58.146321    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9249e174-6085-4a7d-84ae-fd8ef208ced0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9925ccc3-htbcg\" (UID: \"9249e174-6085-4a7d-84ae-fd8ef208ced0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9925ccc3-htbcg"
Nov 03 19:08:58.247514 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:58.247424    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9249e174-6085-4a7d-84ae-fd8ef208ced0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9925ccc3-htbcg\" (UID: \"9249e174-6085-4a7d-84ae-fd8ef208ced0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9925ccc3-htbcg"
Nov 03 19:08:58.247700 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:58.247541    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxfn9\" (UniqueName: \"kubernetes.io/projected/9249e174-6085-4a7d-84ae-fd8ef208ced0-kube-api-access-kxfn9\") pod \"maas-api-revoke-sub-9925ccc3-htbcg\" (UID: \"9249e174-6085-4a7d-84ae-fd8ef208ced0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9925ccc3-htbcg"
Nov 03 19:08:58.247700 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:58.247610    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9249e174-6085-4a7d-84ae-fd8ef208ced0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9925ccc3-htbcg\" (UID: \"9249e174-6085-4a7d-84ae-fd8ef208ced0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9925ccc3-htbcg"
Nov 03 19:08:58.248183 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:58.248155    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9249e174-6085-4a7d-84ae-fd8ef208ced0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9925ccc3-htbcg\" (UID: \"9249e174-6085-4a7d-84ae-fd8ef208ced0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9925ccc3-htbcg"
Nov 03 19:08:58.249928 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:58.249905    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9249e174-6085-4a7d-84ae-fd8ef208ced0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9925ccc3-htbcg\" (UID: \"9249e174-6085-4a7d-84ae-fd8ef208ced0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9925ccc3-htbcg"
Nov 03 19:08:58.305938 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:58.305903    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxfn9\" (UniqueName: \"kubernetes.io/projected/9249e174-6085-4a7d-84ae-fd8ef208ced0-kube-api-access-kxfn9\") pod \"maas-api-revoke-sub-9925ccc3-htbcg\" (UID: \"9249e174-6085-4a7d-84ae-fd8ef208ced0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9925ccc3-htbcg"
Nov 03 19:08:58.583219 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:58.583141    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9925ccc3-htbcg"
Nov 03 19:08:58.953488 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:58.953460    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9925ccc3-htbcg"]
Nov 03 19:08:58.956126 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:08:58.956096    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9249e174_6085_4a7d_84ae_fd8ef208ced0.slice/crio-fe0d5dcb27066ee8a6bb83591a1ab4fe0489d4309de1631cf746ff3fffe3cf3d WatchSource:0}: Error finding container fe0d5dcb27066ee8a6bb83591a1ab4fe0489d4309de1631cf746ff3fffe3cf3d: Status 404 returned error can't find the container with id fe0d5dcb27066ee8a6bb83591a1ab4fe0489d4309de1631cf746ff3fffe3cf3d
Nov 03 19:08:58.973576 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:58.973548    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9925ccc3-htbcg" event={"ID":"9249e174-6085-4a7d-84ae-fd8ef208ced0","Type":"ContainerStarted","Data":"fe0d5dcb27066ee8a6bb83591a1ab4fe0489d4309de1631cf746ff3fffe3cf3d"}
Nov 03 19:08:59.980981 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:59.980945    6059 generic.go:356] "Generic (PLEG): container finished" podID="9249e174-6085-4a7d-84ae-fd8ef208ced0" containerID="7af97f6993dfbdc05485c5553a80fcb17fb40c0a857216e2f22e52604bed4ab3" exitCode=0
Nov 03 19:08:59.981429 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:08:59.981038    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9925ccc3-htbcg" event={"ID":"9249e174-6085-4a7d-84ae-fd8ef208ced0","Type":"ContainerDied","Data":"7af97f6993dfbdc05485c5553a80fcb17fb40c0a857216e2f22e52604bed4ab3"}
Nov 03 19:09:01.126636 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:01.126610    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9925ccc3-htbcg"
Nov 03 19:09:01.174859 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:01.174823    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9249e174-6085-4a7d-84ae-fd8ef208ced0-kube-api-access-kxfn9\" (UniqueName: \"kubernetes.io/projected/9249e174-6085-4a7d-84ae-fd8ef208ced0-kube-api-access-kxfn9\") pod \"9249e174-6085-4a7d-84ae-fd8ef208ced0\" (UID: \"9249e174-6085-4a7d-84ae-fd8ef208ced0\") "
Nov 03 19:09:01.175024 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:01.174870    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9249e174-6085-4a7d-84ae-fd8ef208ced0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9249e174-6085-4a7d-84ae-fd8ef208ced0-maas-api-xks-ca\") pod \"9249e174-6085-4a7d-84ae-fd8ef208ced0\" (UID: \"9249e174-6085-4a7d-84ae-fd8ef208ced0\") "
Nov 03 19:09:01.175024 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:01.174975    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9249e174-6085-4a7d-84ae-fd8ef208ced0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9249e174-6085-4a7d-84ae-fd8ef208ced0-maas-api-openshift-ca\") pod \"9249e174-6085-4a7d-84ae-fd8ef208ced0\" (UID: \"9249e174-6085-4a7d-84ae-fd8ef208ced0\") "
Nov 03 19:09:01.175327 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:01.175302    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9249e174-6085-4a7d-84ae-fd8ef208ced0-maas-api-openshift-ca" pod "9249e174-6085-4a7d-84ae-fd8ef208ced0" (UID: "9249e174-6085-4a7d-84ae-fd8ef208ced0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:09:01.176935 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:01.176909    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9249e174-6085-4a7d-84ae-fd8ef208ced0-maas-api-xks-ca" pod "9249e174-6085-4a7d-84ae-fd8ef208ced0" (UID: "9249e174-6085-4a7d-84ae-fd8ef208ced0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:09:01.176935 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:01.176912    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9249e174-6085-4a7d-84ae-fd8ef208ced0-kube-api-access-kxfn9" pod "9249e174-6085-4a7d-84ae-fd8ef208ced0" (UID: "9249e174-6085-4a7d-84ae-fd8ef208ced0"). InnerVolumeSpecName "kube-api-access-kxfn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:09:01.276001 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:01.275915    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9249e174-6085-4a7d-84ae-fd8ef208ced0-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:01.276001 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:01.275948    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9249e174-6085-4a7d-84ae-fd8ef208ced0-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:01.276001 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:01.275959    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxfn9\" (UniqueName: \"kubernetes.io/projected/9249e174-6085-4a7d-84ae-fd8ef208ced0-kube-api-access-kxfn9\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:01.993579 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:01.993546    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9925ccc3-htbcg"
Nov 03 19:09:01.993808 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:01.993549    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9925ccc3-htbcg" event={"ID":"9249e174-6085-4a7d-84ae-fd8ef208ced0","Type":"ContainerDied","Data":"fe0d5dcb27066ee8a6bb83591a1ab4fe0489d4309de1631cf746ff3fffe3cf3d"}
Nov 03 19:09:01.993808 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:01.993676    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe0d5dcb27066ee8a6bb83591a1ab4fe0489d4309de1631cf746ff3fffe3cf3d"
Nov 03 19:09:06.861845 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:09:06.861800    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:09:08.917125 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:08.917076    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:16.565072 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:16.565033    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:36.861125 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:09:36.861085    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:09:44.287553 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:44.287516    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9"]
Nov 03 19:09:44.287983 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:44.287970    6059 cpu_manager.go:414] "containerMap: removing container" podUID="9249e174-6085-4a7d-84ae-fd8ef208ced0" containerName="revoke-keys"
Nov 03 19:09:44.288031 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:44.287984    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9249e174-6085-4a7d-84ae-fd8ef208ced0" containerName="revoke-keys"
Nov 03 19:09:44.288075 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:44.288063    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9249e174-6085-4a7d-84ae-fd8ef208ced0" containerName="revoke-keys"
Nov 03 19:09:44.291679 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:44.291656    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9"
Nov 03 19:09:44.343035 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:44.343008    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:09:44.343479 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:44.343435    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:09:44.390398 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:44.390365    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:09:44.390398 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:44.390393    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:09:44.392418 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:44.392397    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9"]
Nov 03 19:09:44.430719 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:44.430684    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2c5146-978a-4165-b722-6438daa73f55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-388b812d-llqp9\" (UID: \"4f2c5146-978a-4165-b722-6438daa73f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9"
Nov 03 19:09:44.430948 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:44.430796    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2c5146-978a-4165-b722-6438daa73f55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-388b812d-llqp9\" (UID: \"4f2c5146-978a-4165-b722-6438daa73f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9"
Nov 03 19:09:44.430948 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:44.430900    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z7wz\" (UniqueName: \"kubernetes.io/projected/4f2c5146-978a-4165-b722-6438daa73f55-kube-api-access-4z7wz\") pod \"maas-api-revoke-sub-388b812d-llqp9\" (UID: \"4f2c5146-978a-4165-b722-6438daa73f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9"
Nov 03 19:09:44.532043 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:44.532007    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2c5146-978a-4165-b722-6438daa73f55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-388b812d-llqp9\" (UID: \"4f2c5146-978a-4165-b722-6438daa73f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9"
Nov 03 19:09:44.532230 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:44.532053    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2c5146-978a-4165-b722-6438daa73f55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-388b812d-llqp9\" (UID: \"4f2c5146-978a-4165-b722-6438daa73f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9"
Nov 03 19:09:44.532230 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:44.532098    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z7wz\" (UniqueName: \"kubernetes.io/projected/4f2c5146-978a-4165-b722-6438daa73f55-kube-api-access-4z7wz\") pod \"maas-api-revoke-sub-388b812d-llqp9\" (UID: \"4f2c5146-978a-4165-b722-6438daa73f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9"
Nov 03 19:09:44.532680 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:44.532660    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2c5146-978a-4165-b722-6438daa73f55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-388b812d-llqp9\" (UID: \"4f2c5146-978a-4165-b722-6438daa73f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9"
Nov 03 19:09:44.534502 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:44.534478    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2c5146-978a-4165-b722-6438daa73f55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-388b812d-llqp9\" (UID: \"4f2c5146-978a-4165-b722-6438daa73f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9"
Nov 03 19:09:44.590388 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:44.590289    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z7wz\" (UniqueName: \"kubernetes.io/projected/4f2c5146-978a-4165-b722-6438daa73f55-kube-api-access-4z7wz\") pod \"maas-api-revoke-sub-388b812d-llqp9\" (UID: \"4f2c5146-978a-4165-b722-6438daa73f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9"
Nov 03 19:09:44.605136 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:44.605111    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9"
Nov 03 19:09:44.661973 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:44.661945    6059 scope.go:122] "RemoveContainer" containerID="0866b1d7177428f8c88c33dfaa7d5f408052bd781cdafccb54be776a74a4009a"
Nov 03 19:09:44.971408 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:44.971375    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9"]
Nov 03 19:09:44.974932 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:09:44.974898    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2c5146_978a_4165_b722_6438daa73f55.slice/crio-d7f5967a6579ee6ab833921f7364d2b2867fa80a31bbdcaa876e2eb35bc950a6 WatchSource:0}: Error finding container d7f5967a6579ee6ab833921f7364d2b2867fa80a31bbdcaa876e2eb35bc950a6: Status 404 returned error can't find the container with id d7f5967a6579ee6ab833921f7364d2b2867fa80a31bbdcaa876e2eb35bc950a6
Nov 03 19:09:45.391027 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:45.390982    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9" event={"ID":"4f2c5146-978a-4165-b722-6438daa73f55","Type":"ContainerStarted","Data":"671185766fd53d180dc815e53f683c348037f3a89997d6ab4e54b35466872368"}
Nov 03 19:09:45.391529 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:45.391058    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9" event={"ID":"4f2c5146-978a-4165-b722-6438daa73f55","Type":"ContainerStarted","Data":"d7f5967a6579ee6ab833921f7364d2b2867fa80a31bbdcaa876e2eb35bc950a6"}
Nov 03 19:09:45.502679 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:45.502575    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9" podStartSLOduration=1.50256022 podStartE2EDuration="1.50256022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:45.501207379 +0000 UTC m=+6726.298920161" watchObservedRunningTime="2026-11-03 19:09:45.50256022 +0000 UTC m=+6726.300273002"
Nov 03 19:09:46.397751 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:46.397713    6059 generic.go:356] "Generic (PLEG): container finished" podID="4f2c5146-978a-4165-b722-6438daa73f55" containerID="671185766fd53d180dc815e53f683c348037f3a89997d6ab4e54b35466872368" exitCode=0
Nov 03 19:09:46.398152 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:46.397748    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9" event={"ID":"4f2c5146-978a-4165-b722-6438daa73f55","Type":"ContainerDied","Data":"671185766fd53d180dc815e53f683c348037f3a89997d6ab4e54b35466872368"}
Nov 03 19:09:47.537451 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:47.537421    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9"
Nov 03 19:09:47.563902 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:47.563869    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f2c5146-978a-4165-b722-6438daa73f55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2c5146-978a-4165-b722-6438daa73f55-maas-api-openshift-ca\") pod \"4f2c5146-978a-4165-b722-6438daa73f55\" (UID: \"4f2c5146-978a-4165-b722-6438daa73f55\") "
Nov 03 19:09:47.564057 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:47.563923    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f2c5146-978a-4165-b722-6438daa73f55-kube-api-access-4z7wz\" (UniqueName: \"kubernetes.io/projected/4f2c5146-978a-4165-b722-6438daa73f55-kube-api-access-4z7wz\") pod \"4f2c5146-978a-4165-b722-6438daa73f55\" (UID: \"4f2c5146-978a-4165-b722-6438daa73f55\") "
Nov 03 19:09:47.564117 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:47.564094    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f2c5146-978a-4165-b722-6438daa73f55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2c5146-978a-4165-b722-6438daa73f55-maas-api-xks-ca\") pod \"4f2c5146-978a-4165-b722-6438daa73f55\" (UID: \"4f2c5146-978a-4165-b722-6438daa73f55\") "
Nov 03 19:09:47.564207 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:47.564183    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f2c5146-978a-4165-b722-6438daa73f55-maas-api-openshift-ca" pod "4f2c5146-978a-4165-b722-6438daa73f55" (UID: "4f2c5146-978a-4165-b722-6438daa73f55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:09:47.564454 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:47.564433    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2c5146-978a-4165-b722-6438daa73f55-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:47.565999 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:47.565976    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f2c5146-978a-4165-b722-6438daa73f55-maas-api-xks-ca" pod "4f2c5146-978a-4165-b722-6438daa73f55" (UID: "4f2c5146-978a-4165-b722-6438daa73f55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:09:47.566102 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:47.566045    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f2c5146-978a-4165-b722-6438daa73f55-kube-api-access-4z7wz" pod "4f2c5146-978a-4165-b722-6438daa73f55" (UID: "4f2c5146-978a-4165-b722-6438daa73f55"). InnerVolumeSpecName "kube-api-access-4z7wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:09:47.665664 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:47.665583    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z7wz\" (UniqueName: \"kubernetes.io/projected/4f2c5146-978a-4165-b722-6438daa73f55-kube-api-access-4z7wz\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:47.665664 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:47.665609    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2c5146-978a-4165-b722-6438daa73f55-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:48.410324 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:48.410292    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9"
Nov 03 19:09:48.410528 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:48.410325    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9" event={"ID":"4f2c5146-978a-4165-b722-6438daa73f55","Type":"ContainerDied","Data":"d7f5967a6579ee6ab833921f7364d2b2867fa80a31bbdcaa876e2eb35bc950a6"}
Nov 03 19:09:48.410528 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:48.410384    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7f5967a6579ee6ab833921f7364d2b2867fa80a31bbdcaa876e2eb35bc950a6"
Nov 03 19:09:52.505144 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:52.505107    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:55.087623 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:55.087584    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx"]
Nov 03 19:09:55.088264 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:55.088240    6059 cpu_manager.go:414] "containerMap: removing container" podUID="4f2c5146-978a-4165-b722-6438daa73f55" containerName="revoke-keys"
Nov 03 19:09:55.088376 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:55.088266    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f2c5146-978a-4165-b722-6438daa73f55" containerName="revoke-keys"
Nov 03 19:09:55.088449 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:55.088419    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f2c5146-978a-4165-b722-6438daa73f55" containerName="revoke-keys"
Nov 03 19:09:55.092285 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:55.092256    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx"
Nov 03 19:09:55.139893 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:55.139857    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/964676bd-8a6d-42a0-96bb-52b90bfa97bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d67d5fb-n8kgx\" (UID: \"964676bd-8a6d-42a0-96bb-52b90bfa97bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx"
Nov 03 19:09:55.140045 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:55.139975    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krqjn\" (UniqueName: \"kubernetes.io/projected/964676bd-8a6d-42a0-96bb-52b90bfa97bb-kube-api-access-krqjn\") pod \"maas-api-revoke-sub-3d67d5fb-n8kgx\" (UID: \"964676bd-8a6d-42a0-96bb-52b90bfa97bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx"
Nov 03 19:09:55.140045 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:55.140033    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/964676bd-8a6d-42a0-96bb-52b90bfa97bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d67d5fb-n8kgx\" (UID: \"964676bd-8a6d-42a0-96bb-52b90bfa97bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx"
Nov 03 19:09:55.191087 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:55.191040    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:09:55.191087 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:55.191070    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:09:55.191087 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:55.191058    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:09:55.191087 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:55.191040    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:09:55.193311 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:55.193284    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx"]
Nov 03 19:09:55.241602 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:55.241579    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/964676bd-8a6d-42a0-96bb-52b90bfa97bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d67d5fb-n8kgx\" (UID: \"964676bd-8a6d-42a0-96bb-52b90bfa97bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx"
Nov 03 19:09:55.241714 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:55.241652    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krqjn\" (UniqueName: \"kubernetes.io/projected/964676bd-8a6d-42a0-96bb-52b90bfa97bb-kube-api-access-krqjn\") pod \"maas-api-revoke-sub-3d67d5fb-n8kgx\" (UID: \"964676bd-8a6d-42a0-96bb-52b90bfa97bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx"
Nov 03 19:09:55.241714 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:55.241707    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/964676bd-8a6d-42a0-96bb-52b90bfa97bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d67d5fb-n8kgx\" (UID: \"964676bd-8a6d-42a0-96bb-52b90bfa97bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx"
Nov 03 19:09:55.242290 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:55.242271    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/964676bd-8a6d-42a0-96bb-52b90bfa97bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d67d5fb-n8kgx\" (UID: \"964676bd-8a6d-42a0-96bb-52b90bfa97bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx"
Nov 03 19:09:55.243856 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:55.243834    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/964676bd-8a6d-42a0-96bb-52b90bfa97bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d67d5fb-n8kgx\" (UID: \"964676bd-8a6d-42a0-96bb-52b90bfa97bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx"
Nov 03 19:09:55.300888 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:55.300849    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krqjn\" (UniqueName: \"kubernetes.io/projected/964676bd-8a6d-42a0-96bb-52b90bfa97bb-kube-api-access-krqjn\") pod \"maas-api-revoke-sub-3d67d5fb-n8kgx\" (UID: \"964676bd-8a6d-42a0-96bb-52b90bfa97bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx"
Nov 03 19:09:55.405943 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:55.405859    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx"
Nov 03 19:09:55.774351 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:55.774328    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx"]
Nov 03 19:09:55.776614 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:09:55.776583    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod964676bd_8a6d_42a0_96bb_52b90bfa97bb.slice/crio-af9711dae2edc86ce18f749e7c98daabf26a9793f3ccd221537ca0f90ec94a94 WatchSource:0}: Error finding container af9711dae2edc86ce18f749e7c98daabf26a9793f3ccd221537ca0f90ec94a94: Status 404 returned error can't find the container with id af9711dae2edc86ce18f749e7c98daabf26a9793f3ccd221537ca0f90ec94a94
Nov 03 19:09:56.454125 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:56.454034    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx" event={"ID":"964676bd-8a6d-42a0-96bb-52b90bfa97bb","Type":"ContainerStarted","Data":"615ab3c96c77207282eda0d374b46cb3b5ad08d8983a0c091765d2ae2e775b96"}
Nov 03 19:09:56.454125 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:56.454101    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx" event={"ID":"964676bd-8a6d-42a0-96bb-52b90bfa97bb","Type":"ContainerStarted","Data":"af9711dae2edc86ce18f749e7c98daabf26a9793f3ccd221537ca0f90ec94a94"}
Nov 03 19:09:56.565214 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:56.565165    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx" podStartSLOduration=1.565153011 podStartE2EDuration="1.565153011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:56.564739128 +0000 UTC m=+6737.362451909" watchObservedRunningTime="2026-11-03 19:09:56.565153011 +0000 UTC m=+6737.362865792"
Nov 03 19:09:57.461571 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:57.461528    6059 generic.go:356] "Generic (PLEG): container finished" podID="964676bd-8a6d-42a0-96bb-52b90bfa97bb" containerID="615ab3c96c77207282eda0d374b46cb3b5ad08d8983a0c091765d2ae2e775b96" exitCode=0
Nov 03 19:09:57.462020 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:57.461615    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx" event={"ID":"964676bd-8a6d-42a0-96bb-52b90bfa97bb","Type":"ContainerDied","Data":"615ab3c96c77207282eda0d374b46cb3b5ad08d8983a0c091765d2ae2e775b96"}
Nov 03 19:09:58.606477 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:58.606452    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx"
Nov 03 19:09:58.681743 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:58.681712    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/964676bd-8a6d-42a0-96bb-52b90bfa97bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/964676bd-8a6d-42a0-96bb-52b90bfa97bb-maas-api-xks-ca\") pod \"964676bd-8a6d-42a0-96bb-52b90bfa97bb\" (UID: \"964676bd-8a6d-42a0-96bb-52b90bfa97bb\") "
Nov 03 19:09:58.681928 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:58.681770    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/964676bd-8a6d-42a0-96bb-52b90bfa97bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/964676bd-8a6d-42a0-96bb-52b90bfa97bb-maas-api-openshift-ca\") pod \"964676bd-8a6d-42a0-96bb-52b90bfa97bb\" (UID: \"964676bd-8a6d-42a0-96bb-52b90bfa97bb\") "
Nov 03 19:09:58.681928 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:58.681827    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/964676bd-8a6d-42a0-96bb-52b90bfa97bb-kube-api-access-krqjn\" (UniqueName: \"kubernetes.io/projected/964676bd-8a6d-42a0-96bb-52b90bfa97bb-kube-api-access-krqjn\") pod \"964676bd-8a6d-42a0-96bb-52b90bfa97bb\" (UID: \"964676bd-8a6d-42a0-96bb-52b90bfa97bb\") "
Nov 03 19:09:58.682166 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:58.682142    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/964676bd-8a6d-42a0-96bb-52b90bfa97bb-maas-api-openshift-ca" pod "964676bd-8a6d-42a0-96bb-52b90bfa97bb" (UID: "964676bd-8a6d-42a0-96bb-52b90bfa97bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:09:58.683816 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:58.683787    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964676bd-8a6d-42a0-96bb-52b90bfa97bb-kube-api-access-krqjn" pod "964676bd-8a6d-42a0-96bb-52b90bfa97bb" (UID: "964676bd-8a6d-42a0-96bb-52b90bfa97bb"). InnerVolumeSpecName "kube-api-access-krqjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:09:58.683922 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:58.683853    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/964676bd-8a6d-42a0-96bb-52b90bfa97bb-maas-api-xks-ca" pod "964676bd-8a6d-42a0-96bb-52b90bfa97bb" (UID: "964676bd-8a6d-42a0-96bb-52b90bfa97bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:09:58.783203 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:58.783121    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krqjn\" (UniqueName: \"kubernetes.io/projected/964676bd-8a6d-42a0-96bb-52b90bfa97bb-kube-api-access-krqjn\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:58.783203 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:58.783150    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/964676bd-8a6d-42a0-96bb-52b90bfa97bb-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:58.783203 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:58.783160    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/964676bd-8a6d-42a0-96bb-52b90bfa97bb-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:59.473497 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:59.473465    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx"
Nov 03 19:09:59.473648 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:59.473465    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx" event={"ID":"964676bd-8a6d-42a0-96bb-52b90bfa97bb","Type":"ContainerDied","Data":"af9711dae2edc86ce18f749e7c98daabf26a9793f3ccd221537ca0f90ec94a94"}
Nov 03 19:09:59.473648 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:09:59.473594    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af9711dae2edc86ce18f749e7c98daabf26a9793f3ccd221537ca0f90ec94a94"
Nov 03 19:10:06.861376 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:10:06.861321    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:10:09.274824 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:09.274743    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:09.452820 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:09.452782    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:35.875226 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:35.875145    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:36.048901 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:36.048861    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:36.859662 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:10:36.859624    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:10:39.313181 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:39.313150    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk"]
Nov 03 19:10:39.313651 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:39.313635    6059 cpu_manager.go:414] "containerMap: removing container" podUID="964676bd-8a6d-42a0-96bb-52b90bfa97bb" containerName="revoke-keys"
Nov 03 19:10:39.313710 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:39.313652    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="964676bd-8a6d-42a0-96bb-52b90bfa97bb" containerName="revoke-keys"
Nov 03 19:10:39.313761 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:39.313735    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="964676bd-8a6d-42a0-96bb-52b90bfa97bb" containerName="revoke-keys"
Nov 03 19:10:39.318961 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:39.318927    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk"
Nov 03 19:10:39.369478 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:39.369447    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:10:39.417007 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:39.416974    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:10:39.417007 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:39.416993    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:10:39.417243 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:39.417087    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:10:39.419284 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:39.419252    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk"]
Nov 03 19:10:39.443583 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:39.443552    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e4548a-9fa7-4846-9933-02ce4c6901da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-105678f7-hgpwk\" (UID: \"f5e4548a-9fa7-4846-9933-02ce4c6901da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk"
Nov 03 19:10:39.443769 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:39.443592    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e4548a-9fa7-4846-9933-02ce4c6901da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-105678f7-hgpwk\" (UID: \"f5e4548a-9fa7-4846-9933-02ce4c6901da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk"
Nov 03 19:10:39.443769 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:39.443684    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlpgk\" (UniqueName: \"kubernetes.io/projected/f5e4548a-9fa7-4846-9933-02ce4c6901da-kube-api-access-jlpgk\") pod \"maas-api-revoke-sub-105678f7-hgpwk\" (UID: \"f5e4548a-9fa7-4846-9933-02ce4c6901da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk"
Nov 03 19:10:39.544536 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:39.544505    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlpgk\" (UniqueName: \"kubernetes.io/projected/f5e4548a-9fa7-4846-9933-02ce4c6901da-kube-api-access-jlpgk\") pod \"maas-api-revoke-sub-105678f7-hgpwk\" (UID: \"f5e4548a-9fa7-4846-9933-02ce4c6901da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk"
Nov 03 19:10:39.544748 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:39.544615    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e4548a-9fa7-4846-9933-02ce4c6901da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-105678f7-hgpwk\" (UID: \"f5e4548a-9fa7-4846-9933-02ce4c6901da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk"
Nov 03 19:10:39.544748 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:39.544642    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e4548a-9fa7-4846-9933-02ce4c6901da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-105678f7-hgpwk\" (UID: \"f5e4548a-9fa7-4846-9933-02ce4c6901da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk"
Nov 03 19:10:39.545401 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:39.545346    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e4548a-9fa7-4846-9933-02ce4c6901da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-105678f7-hgpwk\" (UID: \"f5e4548a-9fa7-4846-9933-02ce4c6901da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk"
Nov 03 19:10:39.547042 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:39.547020    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e4548a-9fa7-4846-9933-02ce4c6901da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-105678f7-hgpwk\" (UID: \"f5e4548a-9fa7-4846-9933-02ce4c6901da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk"
Nov 03 19:10:39.603747 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:39.603672    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlpgk\" (UniqueName: \"kubernetes.io/projected/f5e4548a-9fa7-4846-9933-02ce4c6901da-kube-api-access-jlpgk\") pod \"maas-api-revoke-sub-105678f7-hgpwk\" (UID: \"f5e4548a-9fa7-4846-9933-02ce4c6901da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk"
Nov 03 19:10:39.634921 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:39.634886    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk"
Nov 03 19:10:40.005673 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:40.005646    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk"]
Nov 03 19:10:40.007570 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:10:40.007540    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e4548a_9fa7_4846_9933_02ce4c6901da.slice/crio-7c1e33351b25ade3a927ed65583194ccda11413e12963ee0caff80ff3506e198 WatchSource:0}: Error finding container 7c1e33351b25ade3a927ed65583194ccda11413e12963ee0caff80ff3506e198: Status 404 returned error can't find the container with id 7c1e33351b25ade3a927ed65583194ccda11413e12963ee0caff80ff3506e198
Nov 03 19:10:40.857406 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:40.857336    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk" event={"ID":"f5e4548a-9fa7-4846-9933-02ce4c6901da","Type":"ContainerStarted","Data":"ae6149371f244d50d1b5f1a226b1b028ed30d79acf913115c8f859222799ae65"}
Nov 03 19:10:40.857406 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:40.857429    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk" event={"ID":"f5e4548a-9fa7-4846-9933-02ce4c6901da","Type":"ContainerStarted","Data":"7c1e33351b25ade3a927ed65583194ccda11413e12963ee0caff80ff3506e198"}
Nov 03 19:10:40.969651 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:40.969597    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk" podStartSLOduration=1.969583391 podStartE2EDuration="1.969583391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:10:40.967581198 +0000 UTC m=+6781.765293981" watchObservedRunningTime="2026-11-03 19:10:40.969583391 +0000 UTC m=+6781.767296171"
Nov 03 19:10:41.864431 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:41.864397    6059 generic.go:356] "Generic (PLEG): container finished" podID="f5e4548a-9fa7-4846-9933-02ce4c6901da" containerID="ae6149371f244d50d1b5f1a226b1b028ed30d79acf913115c8f859222799ae65" exitCode=0
Nov 03 19:10:41.864841 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:41.864491    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk" event={"ID":"f5e4548a-9fa7-4846-9933-02ce4c6901da","Type":"ContainerDied","Data":"ae6149371f244d50d1b5f1a226b1b028ed30d79acf913115c8f859222799ae65"}
Nov 03 19:10:43.009524 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:43.009496    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk"
Nov 03 19:10:43.184523 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:43.184440    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5e4548a-9fa7-4846-9933-02ce4c6901da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e4548a-9fa7-4846-9933-02ce4c6901da-maas-api-xks-ca\") pod \"f5e4548a-9fa7-4846-9933-02ce4c6901da\" (UID: \"f5e4548a-9fa7-4846-9933-02ce4c6901da\") "
Nov 03 19:10:43.184523 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:43.184502    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5e4548a-9fa7-4846-9933-02ce4c6901da-kube-api-access-jlpgk\" (UniqueName: \"kubernetes.io/projected/f5e4548a-9fa7-4846-9933-02ce4c6901da-kube-api-access-jlpgk\") pod \"f5e4548a-9fa7-4846-9933-02ce4c6901da\" (UID: \"f5e4548a-9fa7-4846-9933-02ce4c6901da\") "
Nov 03 19:10:43.184718 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:43.184652    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5e4548a-9fa7-4846-9933-02ce4c6901da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e4548a-9fa7-4846-9933-02ce4c6901da-maas-api-openshift-ca\") pod \"f5e4548a-9fa7-4846-9933-02ce4c6901da\" (UID: \"f5e4548a-9fa7-4846-9933-02ce4c6901da\") "
Nov 03 19:10:43.185036 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:43.185007    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5e4548a-9fa7-4846-9933-02ce4c6901da-maas-api-openshift-ca" pod "f5e4548a-9fa7-4846-9933-02ce4c6901da" (UID: "f5e4548a-9fa7-4846-9933-02ce4c6901da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:10:43.186499 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:43.186470    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5e4548a-9fa7-4846-9933-02ce4c6901da-maas-api-xks-ca" pod "f5e4548a-9fa7-4846-9933-02ce4c6901da" (UID: "f5e4548a-9fa7-4846-9933-02ce4c6901da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:10:43.186589 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:43.186515    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e4548a-9fa7-4846-9933-02ce4c6901da-kube-api-access-jlpgk" pod "f5e4548a-9fa7-4846-9933-02ce4c6901da" (UID: "f5e4548a-9fa7-4846-9933-02ce4c6901da"). InnerVolumeSpecName "kube-api-access-jlpgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:10:43.285916 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:43.285870    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e4548a-9fa7-4846-9933-02ce4c6901da-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:43.285916 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:43.285910    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e4548a-9fa7-4846-9933-02ce4c6901da-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:43.285916 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:43.285921    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlpgk\" (UniqueName: \"kubernetes.io/projected/f5e4548a-9fa7-4846-9933-02ce4c6901da-kube-api-access-jlpgk\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:43.876594 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:43.876562    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk"
Nov 03 19:10:43.876774 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:43.876592    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk" event={"ID":"f5e4548a-9fa7-4846-9933-02ce4c6901da","Type":"ContainerDied","Data":"7c1e33351b25ade3a927ed65583194ccda11413e12963ee0caff80ff3506e198"}
Nov 03 19:10:43.876774 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:43.876639    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c1e33351b25ade3a927ed65583194ccda11413e12963ee0caff80ff3506e198"
Nov 03 19:10:50.197968 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:50.197928    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47cbfe5e-wxm5n"]
Nov 03 19:10:50.198576 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:50.198551    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f5e4548a-9fa7-4846-9933-02ce4c6901da" containerName="revoke-keys"
Nov 03 19:10:50.198649 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:50.198579    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5e4548a-9fa7-4846-9933-02ce4c6901da" containerName="revoke-keys"
Nov 03 19:10:50.198721 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:50.198709    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5e4548a-9fa7-4846-9933-02ce4c6901da" containerName="revoke-keys"
Nov 03 19:10:50.202158 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:50.202138    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbfe5e-wxm5n"
Nov 03 19:10:50.253429 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:50.253287    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:10:50.256059 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:50.256028    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2de70020-6b22-440e-93ca-113796ad4dbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47cbfe5e-wxm5n\" (UID: \"2de70020-6b22-440e-93ca-113796ad4dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbfe5e-wxm5n"
Nov 03 19:10:50.256295 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:50.256277    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2de70020-6b22-440e-93ca-113796ad4dbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47cbfe5e-wxm5n\" (UID: \"2de70020-6b22-440e-93ca-113796ad4dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbfe5e-wxm5n"
Nov 03 19:10:50.256450 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:50.256435    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ptgx\" (UniqueName: \"kubernetes.io/projected/2de70020-6b22-440e-93ca-113796ad4dbf-kube-api-access-2ptgx\") pod \"maas-api-revoke-sub-47cbfe5e-wxm5n\" (UID: \"2de70020-6b22-440e-93ca-113796ad4dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbfe5e-wxm5n"
Nov 03 19:10:50.300667 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:50.300629    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:10:50.300667 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:50.300637    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:10:50.300890 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:50.300647    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:10:50.302893 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:50.302873    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47cbfe5e-wxm5n"]
Nov 03 19:10:50.357593 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:50.357559    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2de70020-6b22-440e-93ca-113796ad4dbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47cbfe5e-wxm5n\" (UID: \"2de70020-6b22-440e-93ca-113796ad4dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbfe5e-wxm5n"
Nov 03 19:10:50.357760 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:50.357635    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2de70020-6b22-440e-93ca-113796ad4dbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47cbfe5e-wxm5n\" (UID: \"2de70020-6b22-440e-93ca-113796ad4dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbfe5e-wxm5n"
Nov 03 19:10:50.357760 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:50.357674    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ptgx\" (UniqueName: \"kubernetes.io/projected/2de70020-6b22-440e-93ca-113796ad4dbf-kube-api-access-2ptgx\") pod \"maas-api-revoke-sub-47cbfe5e-wxm5n\" (UID: \"2de70020-6b22-440e-93ca-113796ad4dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbfe5e-wxm5n"
Nov 03 19:10:50.358274 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:50.358248    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2de70020-6b22-440e-93ca-113796ad4dbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47cbfe5e-wxm5n\" (UID: \"2de70020-6b22-440e-93ca-113796ad4dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbfe5e-wxm5n"
Nov 03 19:10:50.359810 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:50.359786    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2de70020-6b22-440e-93ca-113796ad4dbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47cbfe5e-wxm5n\" (UID: \"2de70020-6b22-440e-93ca-113796ad4dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbfe5e-wxm5n"
Nov 03 19:10:50.416807 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:50.416774    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ptgx\" (UniqueName: \"kubernetes.io/projected/2de70020-6b22-440e-93ca-113796ad4dbf-kube-api-access-2ptgx\") pod \"maas-api-revoke-sub-47cbfe5e-wxm5n\" (UID: \"2de70020-6b22-440e-93ca-113796ad4dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbfe5e-wxm5n"
Nov 03 19:10:50.516021 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:50.515982    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbfe5e-wxm5n"
Nov 03 19:10:50.882528 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:50.882503    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47cbfe5e-wxm5n"]
Nov 03 19:10:50.884676 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:10:50.884632    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de70020_6b22_440e_93ca_113796ad4dbf.slice/crio-4fa68ec0f16def214a11fcbbba214f17325802a565dbfd31a1e8fdd078c96f4a WatchSource:0}: Error finding container 4fa68ec0f16def214a11fcbbba214f17325802a565dbfd31a1e8fdd078c96f4a: Status 404 returned error can't find the container with id 4fa68ec0f16def214a11fcbbba214f17325802a565dbfd31a1e8fdd078c96f4a
Nov 03 19:10:50.914617 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:50.914586    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbfe5e-wxm5n" event={"ID":"2de70020-6b22-440e-93ca-113796ad4dbf","Type":"ContainerStarted","Data":"4fa68ec0f16def214a11fcbbba214f17325802a565dbfd31a1e8fdd078c96f4a"}
Nov 03 19:10:51.921417 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:51.921321    6059 generic.go:356] "Generic (PLEG): container finished" podID="2de70020-6b22-440e-93ca-113796ad4dbf" containerID="484e9fd887165b357fd26517ac1b8deeba7ae8f015291ad4f7e8b43910fb31b0" exitCode=0
Nov 03 19:10:51.921417 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:51.921389    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbfe5e-wxm5n" event={"ID":"2de70020-6b22-440e-93ca-113796ad4dbf","Type":"ContainerDied","Data":"484e9fd887165b357fd26517ac1b8deeba7ae8f015291ad4f7e8b43910fb31b0"}
Nov 03 19:10:53.128983 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:53.128949    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbfe5e-wxm5n"
Nov 03 19:10:53.185217 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:53.185107    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2de70020-6b22-440e-93ca-113796ad4dbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2de70020-6b22-440e-93ca-113796ad4dbf-maas-api-xks-ca\") pod \"2de70020-6b22-440e-93ca-113796ad4dbf\" (UID: \"2de70020-6b22-440e-93ca-113796ad4dbf\") "
Nov 03 19:10:53.185217 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:53.185208    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2de70020-6b22-440e-93ca-113796ad4dbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2de70020-6b22-440e-93ca-113796ad4dbf-maas-api-openshift-ca\") pod \"2de70020-6b22-440e-93ca-113796ad4dbf\" (UID: \"2de70020-6b22-440e-93ca-113796ad4dbf\") "
Nov 03 19:10:53.185510 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:53.185297    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2de70020-6b22-440e-93ca-113796ad4dbf-kube-api-access-2ptgx\" (UniqueName: \"kubernetes.io/projected/2de70020-6b22-440e-93ca-113796ad4dbf-kube-api-access-2ptgx\") pod \"2de70020-6b22-440e-93ca-113796ad4dbf\" (UID: \"2de70020-6b22-440e-93ca-113796ad4dbf\") "
Nov 03 19:10:53.185734 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:53.185701    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2de70020-6b22-440e-93ca-113796ad4dbf-maas-api-openshift-ca" pod "2de70020-6b22-440e-93ca-113796ad4dbf" (UID: "2de70020-6b22-440e-93ca-113796ad4dbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:10:53.188081 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:53.188052    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2de70020-6b22-440e-93ca-113796ad4dbf-maas-api-xks-ca" pod "2de70020-6b22-440e-93ca-113796ad4dbf" (UID: "2de70020-6b22-440e-93ca-113796ad4dbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:10:53.188272 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:53.188250    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de70020-6b22-440e-93ca-113796ad4dbf-kube-api-access-2ptgx" pod "2de70020-6b22-440e-93ca-113796ad4dbf" (UID: "2de70020-6b22-440e-93ca-113796ad4dbf"). InnerVolumeSpecName "kube-api-access-2ptgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:10:53.286757 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:53.286717    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2de70020-6b22-440e-93ca-113796ad4dbf-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:53.286757 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:53.286758    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2de70020-6b22-440e-93ca-113796ad4dbf-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:53.287021 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:53.286775    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ptgx\" (UniqueName: \"kubernetes.io/projected/2de70020-6b22-440e-93ca-113796ad4dbf-kube-api-access-2ptgx\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:54.017184 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:54.017142    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbfe5e-wxm5n" event={"ID":"2de70020-6b22-440e-93ca-113796ad4dbf","Type":"ContainerDied","Data":"4fa68ec0f16def214a11fcbbba214f17325802a565dbfd31a1e8fdd078c96f4a"}
Nov 03 19:10:54.017570 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:54.017546    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fa68ec0f16def214a11fcbbba214f17325802a565dbfd31a1e8fdd078c96f4a"
Nov 03 19:10:54.017790 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:10:54.017775    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47cbfe5e-wxm5n"
Nov 03 19:11:01.625988 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:01.625952    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:06.869965 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:11:06.869915    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:11:09.656930 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:09.656896    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:30.164588 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:30.164534    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-c55ww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:11:30.165425 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:30.165388    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww" podUID="acc06524-297b-4b57-b0a2-4b2a92d4a00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:11:30.236648 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:30.236487    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bwg8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:11:30.236648 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:30.236574    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p" podUID="c17b0b8a-a3a3-4824-acc3-ce83fb644937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:11:30.257620 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:30.256469    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xkkj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:11:30.257620 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:30.256540    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4" podUID="c24c1306-3d52-4b39-af5d-5b769ec03a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:11:37.021667 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:11:37.021626    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:11:40.104630 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:40.104596    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:40.289849 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:40.289815    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:41.459717 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:41.459681    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-111e3b0e-wg5qc"]
Nov 03 19:11:41.460456 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:41.460436    6059 cpu_manager.go:414] "containerMap: removing container" podUID="2de70020-6b22-440e-93ca-113796ad4dbf" containerName="revoke-keys"
Nov 03 19:11:41.460521 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:41.460459    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2de70020-6b22-440e-93ca-113796ad4dbf" containerName="revoke-keys"
Nov 03 19:11:41.460580 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:41.460569    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2de70020-6b22-440e-93ca-113796ad4dbf" containerName="revoke-keys"
Nov 03 19:11:41.464306 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:41.464279    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111e3b0e-wg5qc"
Nov 03 19:11:41.508011 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:41.507970    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4c396ce-f5d5-47d0-91ec-b64f16749ab6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-111e3b0e-wg5qc\" (UID: \"f4c396ce-f5d5-47d0-91ec-b64f16749ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111e3b0e-wg5qc"
Nov 03 19:11:41.508011 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:41.508010    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgtt7\" (UniqueName: \"kubernetes.io/projected/f4c396ce-f5d5-47d0-91ec-b64f16749ab6-kube-api-access-wgtt7\") pod \"maas-api-revoke-sub-111e3b0e-wg5qc\" (UID: \"f4c396ce-f5d5-47d0-91ec-b64f16749ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111e3b0e-wg5qc"
Nov 03 19:11:41.508328 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:41.508160    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4c396ce-f5d5-47d0-91ec-b64f16749ab6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-111e3b0e-wg5qc\" (UID: \"f4c396ce-f5d5-47d0-91ec-b64f16749ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111e3b0e-wg5qc"
Nov 03 19:11:41.517333 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:41.517299    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-111e3b0e-wg5qc"]
Nov 03 19:11:41.560677 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:41.560633    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:11:41.560928 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:41.560910    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:11:41.561173 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:41.561156    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:11:41.561978 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:41.561955    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:11:41.608824 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:41.608784    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4c396ce-f5d5-47d0-91ec-b64f16749ab6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-111e3b0e-wg5qc\" (UID: \"f4c396ce-f5d5-47d0-91ec-b64f16749ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111e3b0e-wg5qc"
Nov 03 19:11:41.609022 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:41.608831    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgtt7\" (UniqueName: \"kubernetes.io/projected/f4c396ce-f5d5-47d0-91ec-b64f16749ab6-kube-api-access-wgtt7\") pod \"maas-api-revoke-sub-111e3b0e-wg5qc\" (UID: \"f4c396ce-f5d5-47d0-91ec-b64f16749ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111e3b0e-wg5qc"
Nov 03 19:11:41.609022 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:41.608992    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4c396ce-f5d5-47d0-91ec-b64f16749ab6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-111e3b0e-wg5qc\" (UID: \"f4c396ce-f5d5-47d0-91ec-b64f16749ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111e3b0e-wg5qc"
Nov 03 19:11:41.609657 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:41.609620    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4c396ce-f5d5-47d0-91ec-b64f16749ab6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-111e3b0e-wg5qc\" (UID: \"f4c396ce-f5d5-47d0-91ec-b64f16749ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111e3b0e-wg5qc"
Nov 03 19:11:41.611146 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:41.611127    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4c396ce-f5d5-47d0-91ec-b64f16749ab6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-111e3b0e-wg5qc\" (UID: \"f4c396ce-f5d5-47d0-91ec-b64f16749ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111e3b0e-wg5qc"
Nov 03 19:11:41.667288 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:41.667266    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgtt7\" (UniqueName: \"kubernetes.io/projected/f4c396ce-f5d5-47d0-91ec-b64f16749ab6-kube-api-access-wgtt7\") pod \"maas-api-revoke-sub-111e3b0e-wg5qc\" (UID: \"f4c396ce-f5d5-47d0-91ec-b64f16749ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111e3b0e-wg5qc"
Nov 03 19:11:41.777616 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:41.777541    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111e3b0e-wg5qc"
Nov 03 19:11:42.146684 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:42.146651    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-111e3b0e-wg5qc"]
Nov 03 19:11:42.508632 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:42.508599    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111e3b0e-wg5qc" event={"ID":"f4c396ce-f5d5-47d0-91ec-b64f16749ab6","Type":"ContainerStarted","Data":"4b4a599e31566d7de4c39179947664e6251f32dcf4a505cfa0f0145b65d55b50"}
Nov 03 19:11:42.509093 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:42.508654    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111e3b0e-wg5qc" event={"ID":"f4c396ce-f5d5-47d0-91ec-b64f16749ab6","Type":"ContainerStarted","Data":"14b5b21cdbb64d36b13e76e87d8c026fc7964a0e4d108cf6d90e0ad9cd51ab5c"}
Nov 03 19:11:42.621111 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:42.621056    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111e3b0e-wg5qc" podStartSLOduration=1.621038268 podStartE2EDuration="1.621038268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:11:42.619294859 +0000 UTC m=+6843.417007640" watchObservedRunningTime="2026-11-03 19:11:42.621038268 +0000 UTC m=+6843.418751050"
Nov 03 19:11:43.516413 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:43.516374    6059 generic.go:356] "Generic (PLEG): container finished" podID="f4c396ce-f5d5-47d0-91ec-b64f16749ab6" containerID="4b4a599e31566d7de4c39179947664e6251f32dcf4a505cfa0f0145b65d55b50" exitCode=0
Nov 03 19:11:43.516832 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:43.516440    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111e3b0e-wg5qc" event={"ID":"f4c396ce-f5d5-47d0-91ec-b64f16749ab6","Type":"ContainerDied","Data":"4b4a599e31566d7de4c39179947664e6251f32dcf4a505cfa0f0145b65d55b50"}
Nov 03 19:11:44.667535 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:44.667504    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111e3b0e-wg5qc"
Nov 03 19:11:44.740600 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:44.740569    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4c396ce-f5d5-47d0-91ec-b64f16749ab6-kube-api-access-wgtt7\" (UniqueName: \"kubernetes.io/projected/f4c396ce-f5d5-47d0-91ec-b64f16749ab6-kube-api-access-wgtt7\") pod \"f4c396ce-f5d5-47d0-91ec-b64f16749ab6\" (UID: \"f4c396ce-f5d5-47d0-91ec-b64f16749ab6\") "
Nov 03 19:11:44.740771 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:44.740718    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4c396ce-f5d5-47d0-91ec-b64f16749ab6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4c396ce-f5d5-47d0-91ec-b64f16749ab6-maas-api-xks-ca\") pod \"f4c396ce-f5d5-47d0-91ec-b64f16749ab6\" (UID: \"f4c396ce-f5d5-47d0-91ec-b64f16749ab6\") "
Nov 03 19:11:44.740771 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:44.740750    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4c396ce-f5d5-47d0-91ec-b64f16749ab6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4c396ce-f5d5-47d0-91ec-b64f16749ab6-maas-api-openshift-ca\") pod \"f4c396ce-f5d5-47d0-91ec-b64f16749ab6\" (UID: \"f4c396ce-f5d5-47d0-91ec-b64f16749ab6\") "
Nov 03 19:11:44.741154 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:44.741130    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c396ce-f5d5-47d0-91ec-b64f16749ab6-maas-api-openshift-ca" pod "f4c396ce-f5d5-47d0-91ec-b64f16749ab6" (UID: "f4c396ce-f5d5-47d0-91ec-b64f16749ab6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:11:44.742670 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:44.742643    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c396ce-f5d5-47d0-91ec-b64f16749ab6-kube-api-access-wgtt7" pod "f4c396ce-f5d5-47d0-91ec-b64f16749ab6" (UID: "f4c396ce-f5d5-47d0-91ec-b64f16749ab6"). InnerVolumeSpecName "kube-api-access-wgtt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:11:44.742774 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:44.742713    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4c396ce-f5d5-47d0-91ec-b64f16749ab6-maas-api-xks-ca" pod "f4c396ce-f5d5-47d0-91ec-b64f16749ab6" (UID: "f4c396ce-f5d5-47d0-91ec-b64f16749ab6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:11:44.842553 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:44.842465    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4c396ce-f5d5-47d0-91ec-b64f16749ab6-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:44.842553 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:44.842498    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4c396ce-f5d5-47d0-91ec-b64f16749ab6-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:44.842553 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:44.842509    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgtt7\" (UniqueName: \"kubernetes.io/projected/f4c396ce-f5d5-47d0-91ec-b64f16749ab6-kube-api-access-wgtt7\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:45.537066 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:45.537038    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111e3b0e-wg5qc"
Nov 03 19:11:45.537066 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:45.537044    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111e3b0e-wg5qc" event={"ID":"f4c396ce-f5d5-47d0-91ec-b64f16749ab6","Type":"ContainerDied","Data":"14b5b21cdbb64d36b13e76e87d8c026fc7964a0e4d108cf6d90e0ad9cd51ab5c"}
Nov 03 19:11:45.537292 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:45.537106    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14b5b21cdbb64d36b13e76e87d8c026fc7964a0e4d108cf6d90e0ad9cd51ab5c"
Nov 03 19:11:52.713917 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:52.713875    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7caf9856-nrjqw"]
Nov 03 19:11:52.714439 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:52.714394    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f4c396ce-f5d5-47d0-91ec-b64f16749ab6" containerName="revoke-keys"
Nov 03 19:11:52.714439 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:52.714411    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4c396ce-f5d5-47d0-91ec-b64f16749ab6" containerName="revoke-keys"
Nov 03 19:11:52.714577 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:52.714563    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4c396ce-f5d5-47d0-91ec-b64f16749ab6" containerName="revoke-keys"
Nov 03 19:11:52.717570 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:52.717542    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7caf9856-nrjqw"
Nov 03 19:11:52.816479 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:52.816433    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:11:52.816479 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:52.816450    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:11:52.816479 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:52.816451    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:11:52.816810 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:52.816506    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:11:52.818913 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:52.818889    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7caf9856-nrjqw"]
Nov 03 19:11:52.837128 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:52.837089    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a867663e-30b8-4fab-ae7a-e5e54b05208b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7caf9856-nrjqw\" (UID: \"a867663e-30b8-4fab-ae7a-e5e54b05208b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7caf9856-nrjqw"
Nov 03 19:11:52.837291 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:52.837200    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a867663e-30b8-4fab-ae7a-e5e54b05208b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7caf9856-nrjqw\" (UID: \"a867663e-30b8-4fab-ae7a-e5e54b05208b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7caf9856-nrjqw"
Nov 03 19:11:52.837291 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:52.837264    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx999\" (UniqueName: \"kubernetes.io/projected/a867663e-30b8-4fab-ae7a-e5e54b05208b-kube-api-access-nx999\") pod \"maas-api-revoke-sub-7caf9856-nrjqw\" (UID: \"a867663e-30b8-4fab-ae7a-e5e54b05208b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7caf9856-nrjqw"
Nov 03 19:11:52.938263 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:52.938224    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a867663e-30b8-4fab-ae7a-e5e54b05208b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7caf9856-nrjqw\" (UID: \"a867663e-30b8-4fab-ae7a-e5e54b05208b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7caf9856-nrjqw"
Nov 03 19:11:52.938470 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:52.938280    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx999\" (UniqueName: \"kubernetes.io/projected/a867663e-30b8-4fab-ae7a-e5e54b05208b-kube-api-access-nx999\") pod \"maas-api-revoke-sub-7caf9856-nrjqw\" (UID: \"a867663e-30b8-4fab-ae7a-e5e54b05208b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7caf9856-nrjqw"
Nov 03 19:11:52.938470 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:52.938343    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a867663e-30b8-4fab-ae7a-e5e54b05208b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7caf9856-nrjqw\" (UID: \"a867663e-30b8-4fab-ae7a-e5e54b05208b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7caf9856-nrjqw"
Nov 03 19:11:52.938893 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:52.938862    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a867663e-30b8-4fab-ae7a-e5e54b05208b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7caf9856-nrjqw\" (UID: \"a867663e-30b8-4fab-ae7a-e5e54b05208b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7caf9856-nrjqw"
Nov 03 19:11:52.940692 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:52.940674    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a867663e-30b8-4fab-ae7a-e5e54b05208b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7caf9856-nrjqw\" (UID: \"a867663e-30b8-4fab-ae7a-e5e54b05208b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7caf9856-nrjqw"
Nov 03 19:11:52.998941 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:52.998867    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx999\" (UniqueName: \"kubernetes.io/projected/a867663e-30b8-4fab-ae7a-e5e54b05208b-kube-api-access-nx999\") pod \"maas-api-revoke-sub-7caf9856-nrjqw\" (UID: \"a867663e-30b8-4fab-ae7a-e5e54b05208b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7caf9856-nrjqw"
Nov 03 19:11:53.031319 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:53.031292    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7caf9856-nrjqw"
Nov 03 19:11:53.400996 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:53.400963    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7caf9856-nrjqw"]
Nov 03 19:11:53.402636 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:11:53.402610    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda867663e_30b8_4fab_ae7a_e5e54b05208b.slice/crio-fc8b5e9f11f67095a8c35b67a996744b50a7bb3c201d9c165bf8797f7d238e50 WatchSource:0}: Error finding container fc8b5e9f11f67095a8c35b67a996744b50a7bb3c201d9c165bf8797f7d238e50: Status 404 returned error can't find the container with id fc8b5e9f11f67095a8c35b67a996744b50a7bb3c201d9c165bf8797f7d238e50
Nov 03 19:11:53.582139 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:53.582100    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7caf9856-nrjqw" event={"ID":"a867663e-30b8-4fab-ae7a-e5e54b05208b","Type":"ContainerStarted","Data":"fc8b5e9f11f67095a8c35b67a996744b50a7bb3c201d9c165bf8797f7d238e50"}
Nov 03 19:11:54.588503 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:54.588421    6059 generic.go:356] "Generic (PLEG): container finished" podID="a867663e-30b8-4fab-ae7a-e5e54b05208b" containerID="15cba139f05ad2d131286163e4493d090c0d91da038bd499d2499572b4cc69c9" exitCode=0
Nov 03 19:11:54.588886 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:54.588504    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7caf9856-nrjqw" event={"ID":"a867663e-30b8-4fab-ae7a-e5e54b05208b","Type":"ContainerDied","Data":"15cba139f05ad2d131286163e4493d090c0d91da038bd499d2499572b4cc69c9"}
Nov 03 19:11:55.724919 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:55.724893    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7caf9856-nrjqw"
Nov 03 19:11:55.866004 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:55.865906    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a867663e-30b8-4fab-ae7a-e5e54b05208b-kube-api-access-nx999\" (UniqueName: \"kubernetes.io/projected/a867663e-30b8-4fab-ae7a-e5e54b05208b-kube-api-access-nx999\") pod \"a867663e-30b8-4fab-ae7a-e5e54b05208b\" (UID: \"a867663e-30b8-4fab-ae7a-e5e54b05208b\") "
Nov 03 19:11:55.866199 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:55.866040    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a867663e-30b8-4fab-ae7a-e5e54b05208b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a867663e-30b8-4fab-ae7a-e5e54b05208b-maas-api-xks-ca\") pod \"a867663e-30b8-4fab-ae7a-e5e54b05208b\" (UID: \"a867663e-30b8-4fab-ae7a-e5e54b05208b\") "
Nov 03 19:11:55.866199 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:55.866089    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a867663e-30b8-4fab-ae7a-e5e54b05208b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a867663e-30b8-4fab-ae7a-e5e54b05208b-maas-api-openshift-ca\") pod \"a867663e-30b8-4fab-ae7a-e5e54b05208b\" (UID: \"a867663e-30b8-4fab-ae7a-e5e54b05208b\") "
Nov 03 19:11:55.866516 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:55.866488    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a867663e-30b8-4fab-ae7a-e5e54b05208b-maas-api-openshift-ca" pod "a867663e-30b8-4fab-ae7a-e5e54b05208b" (UID: "a867663e-30b8-4fab-ae7a-e5e54b05208b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:11:55.868182 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:55.868150    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a867663e-30b8-4fab-ae7a-e5e54b05208b-maas-api-xks-ca" pod "a867663e-30b8-4fab-ae7a-e5e54b05208b" (UID: "a867663e-30b8-4fab-ae7a-e5e54b05208b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:11:55.868318 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:55.868250    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a867663e-30b8-4fab-ae7a-e5e54b05208b-kube-api-access-nx999" pod "a867663e-30b8-4fab-ae7a-e5e54b05208b" (UID: "a867663e-30b8-4fab-ae7a-e5e54b05208b"). InnerVolumeSpecName "kube-api-access-nx999". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:11:55.967471 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:55.967434    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a867663e-30b8-4fab-ae7a-e5e54b05208b-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:55.967471 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:55.967466    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a867663e-30b8-4fab-ae7a-e5e54b05208b-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:55.967471 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:55.967477    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx999\" (UniqueName: \"kubernetes.io/projected/a867663e-30b8-4fab-ae7a-e5e54b05208b-kube-api-access-nx999\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:56.603899 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:56.603867    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7caf9856-nrjqw"
Nov 03 19:11:56.604115 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:56.603982    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7caf9856-nrjqw" event={"ID":"a867663e-30b8-4fab-ae7a-e5e54b05208b","Type":"ContainerDied","Data":"fc8b5e9f11f67095a8c35b67a996744b50a7bb3c201d9c165bf8797f7d238e50"}
Nov 03 19:11:56.604115 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:11:56.604040    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc8b5e9f11f67095a8c35b67a996744b50a7bb3c201d9c165bf8797f7d238e50"
Nov 03 19:12:00.161380 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:00.160466    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-c55ww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:12:00.161380 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:00.160540    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww" podUID="acc06524-297b-4b57-b0a2-4b2a92d4a00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:12:00.243781 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:00.243736    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bwg8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:12:00.244074 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:00.244043    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p" podUID="c17b0b8a-a3a3-4824-acc3-ce83fb644937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:12:00.256579 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:00.256523    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xkkj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:12:00.256768 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:00.256603    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4" podUID="c24c1306-3d52-4b39-af5d-5b769ec03a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:12:03.500502 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:03.500462    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19c0efa0-pxxrc"]
Nov 03 19:12:03.501001 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:03.500931    6059 cpu_manager.go:414] "containerMap: removing container" podUID="a867663e-30b8-4fab-ae7a-e5e54b05208b" containerName="revoke-keys"
Nov 03 19:12:03.501001 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:03.500945    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a867663e-30b8-4fab-ae7a-e5e54b05208b" containerName="revoke-keys"
Nov 03 19:12:03.501132 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:03.501079    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a867663e-30b8-4fab-ae7a-e5e54b05208b" containerName="revoke-keys"
Nov 03 19:12:03.504450 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:03.504431    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c0efa0-pxxrc"
Nov 03 19:12:03.555891 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:03.555861    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:12:03.603556 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:03.603523    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:12:03.603556 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:03.603531    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:12:03.603556 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:03.603536    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:12:03.605796 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:03.605772    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19c0efa0-pxxrc"]
Nov 03 19:12:03.608967 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:03.608946    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx7ps\" (UniqueName: \"kubernetes.io/projected/cb41caea-40c1-41c3-b9d8-2b1843dfcb45-kube-api-access-dx7ps\") pod \"maas-api-revoke-sub-19c0efa0-pxxrc\" (UID: \"cb41caea-40c1-41c3-b9d8-2b1843dfcb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c0efa0-pxxrc"
Nov 03 19:12:03.609120 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:03.609096    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb41caea-40c1-41c3-b9d8-2b1843dfcb45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19c0efa0-pxxrc\" (UID: \"cb41caea-40c1-41c3-b9d8-2b1843dfcb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c0efa0-pxxrc"
Nov 03 19:12:03.609199 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:03.609142    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb41caea-40c1-41c3-b9d8-2b1843dfcb45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19c0efa0-pxxrc\" (UID: \"cb41caea-40c1-41c3-b9d8-2b1843dfcb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c0efa0-pxxrc"
Nov 03 19:12:03.709869 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:03.709828    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb41caea-40c1-41c3-b9d8-2b1843dfcb45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19c0efa0-pxxrc\" (UID: \"cb41caea-40c1-41c3-b9d8-2b1843dfcb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c0efa0-pxxrc"
Nov 03 19:12:03.710073 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:03.709874    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb41caea-40c1-41c3-b9d8-2b1843dfcb45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19c0efa0-pxxrc\" (UID: \"cb41caea-40c1-41c3-b9d8-2b1843dfcb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c0efa0-pxxrc"
Nov 03 19:12:03.710073 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:03.709946    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx7ps\" (UniqueName: \"kubernetes.io/projected/cb41caea-40c1-41c3-b9d8-2b1843dfcb45-kube-api-access-dx7ps\") pod \"maas-api-revoke-sub-19c0efa0-pxxrc\" (UID: \"cb41caea-40c1-41c3-b9d8-2b1843dfcb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c0efa0-pxxrc"
Nov 03 19:12:03.710590 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:03.710567    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb41caea-40c1-41c3-b9d8-2b1843dfcb45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19c0efa0-pxxrc\" (UID: \"cb41caea-40c1-41c3-b9d8-2b1843dfcb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c0efa0-pxxrc"
Nov 03 19:12:03.712322 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:03.712298    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb41caea-40c1-41c3-b9d8-2b1843dfcb45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19c0efa0-pxxrc\" (UID: \"cb41caea-40c1-41c3-b9d8-2b1843dfcb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c0efa0-pxxrc"
Nov 03 19:12:03.768729 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:03.768662    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx7ps\" (UniqueName: \"kubernetes.io/projected/cb41caea-40c1-41c3-b9d8-2b1843dfcb45-kube-api-access-dx7ps\") pod \"maas-api-revoke-sub-19c0efa0-pxxrc\" (UID: \"cb41caea-40c1-41c3-b9d8-2b1843dfcb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c0efa0-pxxrc"
Nov 03 19:12:03.817738 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:03.817709    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c0efa0-pxxrc"
Nov 03 19:12:04.185055 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:04.184919    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19c0efa0-pxxrc"]
Nov 03 19:12:04.185907 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:12:04.185879    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb41caea_40c1_41c3_b9d8_2b1843dfcb45.slice/crio-11d595a7003f6549329cbc275f73d887a51f2d897ae8c185cb365f99eb894dc7 WatchSource:0}: Error finding container 11d595a7003f6549329cbc275f73d887a51f2d897ae8c185cb365f99eb894dc7: Status 404 returned error can't find the container with id 11d595a7003f6549329cbc275f73d887a51f2d897ae8c185cb365f99eb894dc7
Nov 03 19:12:04.664999 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:04.664960    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c0efa0-pxxrc" event={"ID":"cb41caea-40c1-41c3-b9d8-2b1843dfcb45","Type":"ContainerStarted","Data":"f3124a4b0fc5d4d226bcde5914f6ebb24dbbb8e660ee1837ba969be36642f8ab"}
Nov 03 19:12:04.665419 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:04.665015    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c0efa0-pxxrc" event={"ID":"cb41caea-40c1-41c3-b9d8-2b1843dfcb45","Type":"ContainerStarted","Data":"11d595a7003f6549329cbc275f73d887a51f2d897ae8c185cb365f99eb894dc7"}
Nov 03 19:12:04.777462 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:04.777407    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c0efa0-pxxrc" podStartSLOduration=1.777392652 podStartE2EDuration="1.777392652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:12:04.775506844 +0000 UTC m=+6865.573219820" watchObservedRunningTime="2026-11-03 19:12:04.777392652 +0000 UTC m=+6865.575105423"
Nov 03 19:12:05.672202 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:05.672164    6059 generic.go:356] "Generic (PLEG): container finished" podID="cb41caea-40c1-41c3-b9d8-2b1843dfcb45" containerID="f3124a4b0fc5d4d226bcde5914f6ebb24dbbb8e660ee1837ba969be36642f8ab" exitCode=0
Nov 03 19:12:05.672679 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:05.672253    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c0efa0-pxxrc" event={"ID":"cb41caea-40c1-41c3-b9d8-2b1843dfcb45","Type":"ContainerDied","Data":"f3124a4b0fc5d4d226bcde5914f6ebb24dbbb8e660ee1837ba969be36642f8ab"}
Nov 03 19:12:06.812115 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:06.812088    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c0efa0-pxxrc"
Nov 03 19:12:06.860718 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:12:06.860684    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:12:06.945150 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:06.945069    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb41caea-40c1-41c3-b9d8-2b1843dfcb45-kube-api-access-dx7ps\" (UniqueName: \"kubernetes.io/projected/cb41caea-40c1-41c3-b9d8-2b1843dfcb45-kube-api-access-dx7ps\") pod \"cb41caea-40c1-41c3-b9d8-2b1843dfcb45\" (UID: \"cb41caea-40c1-41c3-b9d8-2b1843dfcb45\") "
Nov 03 19:12:06.945302 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:06.945196    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb41caea-40c1-41c3-b9d8-2b1843dfcb45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb41caea-40c1-41c3-b9d8-2b1843dfcb45-maas-api-openshift-ca\") pod \"cb41caea-40c1-41c3-b9d8-2b1843dfcb45\" (UID: \"cb41caea-40c1-41c3-b9d8-2b1843dfcb45\") "
Nov 03 19:12:06.945302 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:06.945249    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb41caea-40c1-41c3-b9d8-2b1843dfcb45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb41caea-40c1-41c3-b9d8-2b1843dfcb45-maas-api-xks-ca\") pod \"cb41caea-40c1-41c3-b9d8-2b1843dfcb45\" (UID: \"cb41caea-40c1-41c3-b9d8-2b1843dfcb45\") "
Nov 03 19:12:06.945610 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:06.945582    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb41caea-40c1-41c3-b9d8-2b1843dfcb45-maas-api-openshift-ca" pod "cb41caea-40c1-41c3-b9d8-2b1843dfcb45" (UID: "cb41caea-40c1-41c3-b9d8-2b1843dfcb45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:12:06.945730 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:06.945708    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb41caea-40c1-41c3-b9d8-2b1843dfcb45-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:06.947212 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:06.947182    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb41caea-40c1-41c3-b9d8-2b1843dfcb45-kube-api-access-dx7ps" pod "cb41caea-40c1-41c3-b9d8-2b1843dfcb45" (UID: "cb41caea-40c1-41c3-b9d8-2b1843dfcb45"). InnerVolumeSpecName "kube-api-access-dx7ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:12:06.947307 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:06.947229    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb41caea-40c1-41c3-b9d8-2b1843dfcb45-maas-api-xks-ca" pod "cb41caea-40c1-41c3-b9d8-2b1843dfcb45" (UID: "cb41caea-40c1-41c3-b9d8-2b1843dfcb45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:12:07.046514 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:07.046477    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb41caea-40c1-41c3-b9d8-2b1843dfcb45-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:07.046514 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:07.046508    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx7ps\" (UniqueName: \"kubernetes.io/projected/cb41caea-40c1-41c3-b9d8-2b1843dfcb45-kube-api-access-dx7ps\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:07.684180 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:07.684091    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c0efa0-pxxrc" event={"ID":"cb41caea-40c1-41c3-b9d8-2b1843dfcb45","Type":"ContainerDied","Data":"11d595a7003f6549329cbc275f73d887a51f2d897ae8c185cb365f99eb894dc7"}
Nov 03 19:12:07.684180 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:07.684112    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c0efa0-pxxrc"
Nov 03 19:12:07.684180 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:07.684143    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11d595a7003f6549329cbc275f73d887a51f2d897ae8c185cb365f99eb894dc7"
Nov 03 19:12:11.714703 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:11.714666    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:14.482693 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:14.482657    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d288ac11-2n9vb"]
Nov 03 19:12:14.483305 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:14.483276    6059 cpu_manager.go:414] "containerMap: removing container" podUID="cb41caea-40c1-41c3-b9d8-2b1843dfcb45" containerName="revoke-keys"
Nov 03 19:12:14.483454 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:14.483314    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb41caea-40c1-41c3-b9d8-2b1843dfcb45" containerName="revoke-keys"
Nov 03 19:12:14.483538 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:14.483475    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb41caea-40c1-41c3-b9d8-2b1843dfcb45" containerName="revoke-keys"
Nov 03 19:12:14.494636 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:14.494608    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d288ac11-2n9vb"
Nov 03 19:12:14.541395 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:14.541347    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d288ac11-2n9vb"]
Nov 03 19:12:14.585471 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:14.585437    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:12:14.585471 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:14.585437    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:12:14.585689 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:14.585504    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:12:14.592242 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:14.592217    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:12:14.645975 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:14.645939    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb20844-7ae2-4d22-a7cf-cd6d74d72e56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d288ac11-2n9vb\" (UID: \"fbb20844-7ae2-4d22-a7cf-cd6d74d72e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d288ac11-2n9vb"
Nov 03 19:12:14.646111 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:14.646088    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4kgv\" (UniqueName: \"kubernetes.io/projected/fbb20844-7ae2-4d22-a7cf-cd6d74d72e56-kube-api-access-x4kgv\") pod \"maas-api-revoke-sub-d288ac11-2n9vb\" (UID: \"fbb20844-7ae2-4d22-a7cf-cd6d74d72e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d288ac11-2n9vb"
Nov 03 19:12:14.646160 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:14.646151    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb20844-7ae2-4d22-a7cf-cd6d74d72e56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d288ac11-2n9vb\" (UID: \"fbb20844-7ae2-4d22-a7cf-cd6d74d72e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d288ac11-2n9vb"
Nov 03 19:12:14.747109 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:14.747031    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb20844-7ae2-4d22-a7cf-cd6d74d72e56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d288ac11-2n9vb\" (UID: \"fbb20844-7ae2-4d22-a7cf-cd6d74d72e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d288ac11-2n9vb"
Nov 03 19:12:14.747109 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:14.747098    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb20844-7ae2-4d22-a7cf-cd6d74d72e56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d288ac11-2n9vb\" (UID: \"fbb20844-7ae2-4d22-a7cf-cd6d74d72e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d288ac11-2n9vb"
Nov 03 19:12:14.747298 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:14.747160    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4kgv\" (UniqueName: \"kubernetes.io/projected/fbb20844-7ae2-4d22-a7cf-cd6d74d72e56-kube-api-access-x4kgv\") pod \"maas-api-revoke-sub-d288ac11-2n9vb\" (UID: \"fbb20844-7ae2-4d22-a7cf-cd6d74d72e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d288ac11-2n9vb"
Nov 03 19:12:14.747910 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:14.747869    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb20844-7ae2-4d22-a7cf-cd6d74d72e56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d288ac11-2n9vb\" (UID: \"fbb20844-7ae2-4d22-a7cf-cd6d74d72e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d288ac11-2n9vb"
Nov 03 19:12:14.749430 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:14.749409    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb20844-7ae2-4d22-a7cf-cd6d74d72e56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d288ac11-2n9vb\" (UID: \"fbb20844-7ae2-4d22-a7cf-cd6d74d72e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d288ac11-2n9vb"
Nov 03 19:12:14.805326 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:14.805298    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4kgv\" (UniqueName: \"kubernetes.io/projected/fbb20844-7ae2-4d22-a7cf-cd6d74d72e56-kube-api-access-x4kgv\") pod \"maas-api-revoke-sub-d288ac11-2n9vb\" (UID: \"fbb20844-7ae2-4d22-a7cf-cd6d74d72e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d288ac11-2n9vb"
Nov 03 19:12:14.808135 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:14.808119    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d288ac11-2n9vb"
Nov 03 19:12:15.175711 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:15.175679    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d288ac11-2n9vb"]
Nov 03 19:12:15.177541 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:12:15.177511    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb20844_7ae2_4d22_a7cf_cd6d74d72e56.slice/crio-d4582514cabdc05ae258bf77354010cb18497eb7ea7e18b873d0fa6c49c3aa8f WatchSource:0}: Error finding container d4582514cabdc05ae258bf77354010cb18497eb7ea7e18b873d0fa6c49c3aa8f: Status 404 returned error can't find the container with id d4582514cabdc05ae258bf77354010cb18497eb7ea7e18b873d0fa6c49c3aa8f
Nov 03 19:12:15.741388 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:15.741333    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d288ac11-2n9vb" event={"ID":"fbb20844-7ae2-4d22-a7cf-cd6d74d72e56","Type":"ContainerStarted","Data":"4647378c31977f0d63253c14f1a9775bb4f612000b260d51dedbb4ffb475fa86"}
Nov 03 19:12:15.741800 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:15.741399    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d288ac11-2n9vb" event={"ID":"fbb20844-7ae2-4d22-a7cf-cd6d74d72e56","Type":"ContainerStarted","Data":"d4582514cabdc05ae258bf77354010cb18497eb7ea7e18b873d0fa6c49c3aa8f"}
Nov 03 19:12:15.853227 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:15.853177    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d288ac11-2n9vb" podStartSLOduration=1.8531637600000002 podStartE2EDuration="1.85316376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:12:15.852434743 +0000 UTC m=+6876.650147525" watchObservedRunningTime="2026-11-03 19:12:15.85316376 +0000 UTC m=+6876.650876540"
Nov 03 19:12:16.748973 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:16.748938    6059 generic.go:356] "Generic (PLEG): container finished" podID="fbb20844-7ae2-4d22-a7cf-cd6d74d72e56" containerID="4647378c31977f0d63253c14f1a9775bb4f612000b260d51dedbb4ffb475fa86" exitCode=0
Nov 03 19:12:16.749369 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:16.749036    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d288ac11-2n9vb" event={"ID":"fbb20844-7ae2-4d22-a7cf-cd6d74d72e56","Type":"ContainerDied","Data":"4647378c31977f0d63253c14f1a9775bb4f612000b260d51dedbb4ffb475fa86"}
Nov 03 19:12:17.890012 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:17.889985    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d288ac11-2n9vb"
Nov 03 19:12:18.083845 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:18.083763    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb20844-7ae2-4d22-a7cf-cd6d74d72e56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb20844-7ae2-4d22-a7cf-cd6d74d72e56-maas-api-openshift-ca\") pod \"fbb20844-7ae2-4d22-a7cf-cd6d74d72e56\" (UID: \"fbb20844-7ae2-4d22-a7cf-cd6d74d72e56\") "
Nov 03 19:12:18.083845 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:18.083817    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb20844-7ae2-4d22-a7cf-cd6d74d72e56-kube-api-access-x4kgv\" (UniqueName: \"kubernetes.io/projected/fbb20844-7ae2-4d22-a7cf-cd6d74d72e56-kube-api-access-x4kgv\") pod \"fbb20844-7ae2-4d22-a7cf-cd6d74d72e56\" (UID: \"fbb20844-7ae2-4d22-a7cf-cd6d74d72e56\") "
Nov 03 19:12:18.084090 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:18.083850    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbb20844-7ae2-4d22-a7cf-cd6d74d72e56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb20844-7ae2-4d22-a7cf-cd6d74d72e56-maas-api-xks-ca\") pod \"fbb20844-7ae2-4d22-a7cf-cd6d74d72e56\" (UID: \"fbb20844-7ae2-4d22-a7cf-cd6d74d72e56\") "
Nov 03 19:12:18.084157 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:18.084095    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb20844-7ae2-4d22-a7cf-cd6d74d72e56-maas-api-openshift-ca" pod "fbb20844-7ae2-4d22-a7cf-cd6d74d72e56" (UID: "fbb20844-7ae2-4d22-a7cf-cd6d74d72e56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:12:18.085940 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:18.085915    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb20844-7ae2-4d22-a7cf-cd6d74d72e56-maas-api-xks-ca" pod "fbb20844-7ae2-4d22-a7cf-cd6d74d72e56" (UID: "fbb20844-7ae2-4d22-a7cf-cd6d74d72e56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:12:18.086078 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:18.086056    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb20844-7ae2-4d22-a7cf-cd6d74d72e56-kube-api-access-x4kgv" pod "fbb20844-7ae2-4d22-a7cf-cd6d74d72e56" (UID: "fbb20844-7ae2-4d22-a7cf-cd6d74d72e56"). InnerVolumeSpecName "kube-api-access-x4kgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:12:18.184938 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:18.184897    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb20844-7ae2-4d22-a7cf-cd6d74d72e56-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:18.184938 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:18.184930    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4kgv\" (UniqueName: \"kubernetes.io/projected/fbb20844-7ae2-4d22-a7cf-cd6d74d72e56-kube-api-access-x4kgv\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:18.184938 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:18.184944    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb20844-7ae2-4d22-a7cf-cd6d74d72e56-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:18.765756 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:18.765715    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d288ac11-2n9vb"
Nov 03 19:12:18.766021 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:18.765707    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d288ac11-2n9vb" event={"ID":"fbb20844-7ae2-4d22-a7cf-cd6d74d72e56","Type":"ContainerDied","Data":"d4582514cabdc05ae258bf77354010cb18497eb7ea7e18b873d0fa6c49c3aa8f"}
Nov 03 19:12:18.766021 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:18.765969    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4582514cabdc05ae258bf77354010cb18497eb7ea7e18b873d0fa6c49c3aa8f"
Nov 03 19:12:25.247924 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:25.247890    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7c483d2-6dtt7"]
Nov 03 19:12:25.248416 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:25.248378    6059 cpu_manager.go:414] "containerMap: removing container" podUID="fbb20844-7ae2-4d22-a7cf-cd6d74d72e56" containerName="revoke-keys"
Nov 03 19:12:25.248416 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:25.248404    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb20844-7ae2-4d22-a7cf-cd6d74d72e56" containerName="revoke-keys"
Nov 03 19:12:25.248528 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:25.248480    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb20844-7ae2-4d22-a7cf-cd6d74d72e56" containerName="revoke-keys"
Nov 03 19:12:25.251930 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:25.251913    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c483d2-6dtt7"
Nov 03 19:12:25.303290 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:25.303256    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:12:25.303489 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:25.303319    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:12:25.309025 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:25.308990    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c39758e-e9fc-4cbd-92c0-c3550cf8510b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7c483d2-6dtt7\" (UID: \"4c39758e-e9fc-4cbd-92c0-c3550cf8510b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c483d2-6dtt7"
Nov 03 19:12:25.309200 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:25.309149    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c39758e-e9fc-4cbd-92c0-c3550cf8510b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7c483d2-6dtt7\" (UID: \"4c39758e-e9fc-4cbd-92c0-c3550cf8510b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c483d2-6dtt7"
Nov 03 19:12:25.309273 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:25.309223    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nfn2\" (UniqueName: \"kubernetes.io/projected/4c39758e-e9fc-4cbd-92c0-c3550cf8510b-kube-api-access-6nfn2\") pod \"maas-api-revoke-sub-e7c483d2-6dtt7\" (UID: \"4c39758e-e9fc-4cbd-92c0-c3550cf8510b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c483d2-6dtt7"
Nov 03 19:12:25.350627 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:25.350596    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:12:25.350627 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:25.350596    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:12:25.353290 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:25.353268    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7c483d2-6dtt7"]
Nov 03 19:12:25.410755 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:25.410719    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c39758e-e9fc-4cbd-92c0-c3550cf8510b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7c483d2-6dtt7\" (UID: \"4c39758e-e9fc-4cbd-92c0-c3550cf8510b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c483d2-6dtt7"
Nov 03 19:12:25.410934 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:25.410803    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c39758e-e9fc-4cbd-92c0-c3550cf8510b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7c483d2-6dtt7\" (UID: \"4c39758e-e9fc-4cbd-92c0-c3550cf8510b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c483d2-6dtt7"
Nov 03 19:12:25.410934 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:25.410849    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nfn2\" (UniqueName: \"kubernetes.io/projected/4c39758e-e9fc-4cbd-92c0-c3550cf8510b-kube-api-access-6nfn2\") pod \"maas-api-revoke-sub-e7c483d2-6dtt7\" (UID: \"4c39758e-e9fc-4cbd-92c0-c3550cf8510b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c483d2-6dtt7"
Nov 03 19:12:25.411519 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:25.411494    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c39758e-e9fc-4cbd-92c0-c3550cf8510b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7c483d2-6dtt7\" (UID: \"4c39758e-e9fc-4cbd-92c0-c3550cf8510b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c483d2-6dtt7"
Nov 03 19:12:25.413096 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:25.413081    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c39758e-e9fc-4cbd-92c0-c3550cf8510b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7c483d2-6dtt7\" (UID: \"4c39758e-e9fc-4cbd-92c0-c3550cf8510b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c483d2-6dtt7"
Nov 03 19:12:25.469838 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:25.469808    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nfn2\" (UniqueName: \"kubernetes.io/projected/4c39758e-e9fc-4cbd-92c0-c3550cf8510b-kube-api-access-6nfn2\") pod \"maas-api-revoke-sub-e7c483d2-6dtt7\" (UID: \"4c39758e-e9fc-4cbd-92c0-c3550cf8510b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c483d2-6dtt7"
Nov 03 19:12:25.566139 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:25.566049    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c483d2-6dtt7"
Nov 03 19:12:25.935456 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:25.935431    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7c483d2-6dtt7"]
Nov 03 19:12:25.937416 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:12:25.937376    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c39758e_e9fc_4cbd_92c0_c3550cf8510b.slice/crio-6e521e44ac84270e37063d6885bbc6a3355f484d0f29e6c371bc059246ef5a17 WatchSource:0}: Error finding container 6e521e44ac84270e37063d6885bbc6a3355f484d0f29e6c371bc059246ef5a17: Status 404 returned error can't find the container with id 6e521e44ac84270e37063d6885bbc6a3355f484d0f29e6c371bc059246ef5a17
Nov 03 19:12:26.833833 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:26.833798    6059 generic.go:356] "Generic (PLEG): container finished" podID="4c39758e-e9fc-4cbd-92c0-c3550cf8510b" containerID="3b63e6ff0b402e8687634139fec8eb684f430989acb802a5d27ff79f270c1330" exitCode=0
Nov 03 19:12:26.834260 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:26.833889    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c483d2-6dtt7" event={"ID":"4c39758e-e9fc-4cbd-92c0-c3550cf8510b","Type":"ContainerDied","Data":"3b63e6ff0b402e8687634139fec8eb684f430989acb802a5d27ff79f270c1330"}
Nov 03 19:12:26.834260 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:26.833945    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c483d2-6dtt7" event={"ID":"4c39758e-e9fc-4cbd-92c0-c3550cf8510b","Type":"ContainerStarted","Data":"6e521e44ac84270e37063d6885bbc6a3355f484d0f29e6c371bc059246ef5a17"}
Nov 03 19:12:27.984006 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:27.983978    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c483d2-6dtt7"
Nov 03 19:12:28.036431 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:28.036398    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c39758e-e9fc-4cbd-92c0-c3550cf8510b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c39758e-e9fc-4cbd-92c0-c3550cf8510b-maas-api-openshift-ca\") pod \"4c39758e-e9fc-4cbd-92c0-c3550cf8510b\" (UID: \"4c39758e-e9fc-4cbd-92c0-c3550cf8510b\") "
Nov 03 19:12:28.036613 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:28.036500    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c39758e-e9fc-4cbd-92c0-c3550cf8510b-kube-api-access-6nfn2\" (UniqueName: \"kubernetes.io/projected/4c39758e-e9fc-4cbd-92c0-c3550cf8510b-kube-api-access-6nfn2\") pod \"4c39758e-e9fc-4cbd-92c0-c3550cf8510b\" (UID: \"4c39758e-e9fc-4cbd-92c0-c3550cf8510b\") "
Nov 03 19:12:28.036613 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:28.036561    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c39758e-e9fc-4cbd-92c0-c3550cf8510b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c39758e-e9fc-4cbd-92c0-c3550cf8510b-maas-api-xks-ca\") pod \"4c39758e-e9fc-4cbd-92c0-c3550cf8510b\" (UID: \"4c39758e-e9fc-4cbd-92c0-c3550cf8510b\") "
Nov 03 19:12:28.036817 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:28.036793    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c39758e-e9fc-4cbd-92c0-c3550cf8510b-maas-api-openshift-ca" pod "4c39758e-e9fc-4cbd-92c0-c3550cf8510b" (UID: "4c39758e-e9fc-4cbd-92c0-c3550cf8510b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:12:28.036904 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:28.036890    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c39758e-e9fc-4cbd-92c0-c3550cf8510b-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:28.038684 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:28.038658    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c39758e-e9fc-4cbd-92c0-c3550cf8510b-maas-api-xks-ca" pod "4c39758e-e9fc-4cbd-92c0-c3550cf8510b" (UID: "4c39758e-e9fc-4cbd-92c0-c3550cf8510b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:12:28.038806 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:28.038751    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c39758e-e9fc-4cbd-92c0-c3550cf8510b-kube-api-access-6nfn2" pod "4c39758e-e9fc-4cbd-92c0-c3550cf8510b" (UID: "4c39758e-e9fc-4cbd-92c0-c3550cf8510b"). InnerVolumeSpecName "kube-api-access-6nfn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:12:28.138675 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:28.138594    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nfn2\" (UniqueName: \"kubernetes.io/projected/4c39758e-e9fc-4cbd-92c0-c3550cf8510b-kube-api-access-6nfn2\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:28.138675 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:28.138623    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c39758e-e9fc-4cbd-92c0-c3550cf8510b-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:28.846253 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:28.846219    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c483d2-6dtt7"
Nov 03 19:12:28.846449 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:28.846262    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c483d2-6dtt7" event={"ID":"4c39758e-e9fc-4cbd-92c0-c3550cf8510b","Type":"ContainerDied","Data":"6e521e44ac84270e37063d6885bbc6a3355f484d0f29e6c371bc059246ef5a17"}
Nov 03 19:12:28.846449 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:28.846308    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e521e44ac84270e37063d6885bbc6a3355f484d0f29e6c371bc059246ef5a17"
Nov 03 19:12:30.125790 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:30.125757    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:30.308510 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:30.308479    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:32.561941 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:32.561903    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:36.180213 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:36.180181    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42918c8e-mttv9"]
Nov 03 19:12:36.180702 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:36.180685    6059 cpu_manager.go:414] "containerMap: removing container" podUID="4c39758e-e9fc-4cbd-92c0-c3550cf8510b" containerName="revoke-keys"
Nov 03 19:12:36.180784 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:36.180705    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c39758e-e9fc-4cbd-92c0-c3550cf8510b" containerName="revoke-keys"
Nov 03 19:12:36.180878 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:36.180860    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c39758e-e9fc-4cbd-92c0-c3550cf8510b" containerName="revoke-keys"
Nov 03 19:12:36.183951 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:36.183930    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42918c8e-mttv9"
Nov 03 19:12:36.235001 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:36.234956    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:12:36.282231 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:36.282196    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:12:36.282231 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:36.282204    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:12:36.282490 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:36.282328    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:12:36.284757 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:36.284735    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42918c8e-mttv9"]
Nov 03 19:12:36.324969 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:36.324937    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a5a1dea-0df1-4268-b9d0-360fdcd887de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42918c8e-mttv9\" (UID: \"2a5a1dea-0df1-4268-b9d0-360fdcd887de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42918c8e-mttv9"
Nov 03 19:12:36.325117 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:36.325070    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a5a1dea-0df1-4268-b9d0-360fdcd887de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42918c8e-mttv9\" (UID: \"2a5a1dea-0df1-4268-b9d0-360fdcd887de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42918c8e-mttv9"
Nov 03 19:12:36.325160 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:36.325137    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mtpl\" (UniqueName: \"kubernetes.io/projected/2a5a1dea-0df1-4268-b9d0-360fdcd887de-kube-api-access-9mtpl\") pod \"maas-api-revoke-sub-42918c8e-mttv9\" (UID: \"2a5a1dea-0df1-4268-b9d0-360fdcd887de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42918c8e-mttv9"
Nov 03 19:12:36.425678 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:36.425641    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mtpl\" (UniqueName: \"kubernetes.io/projected/2a5a1dea-0df1-4268-b9d0-360fdcd887de-kube-api-access-9mtpl\") pod \"maas-api-revoke-sub-42918c8e-mttv9\" (UID: \"2a5a1dea-0df1-4268-b9d0-360fdcd887de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42918c8e-mttv9"
Nov 03 19:12:36.425869 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:36.425691    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a5a1dea-0df1-4268-b9d0-360fdcd887de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42918c8e-mttv9\" (UID: \"2a5a1dea-0df1-4268-b9d0-360fdcd887de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42918c8e-mttv9"
Nov 03 19:12:36.425869 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:36.425784    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a5a1dea-0df1-4268-b9d0-360fdcd887de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42918c8e-mttv9\" (UID: \"2a5a1dea-0df1-4268-b9d0-360fdcd887de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42918c8e-mttv9"
Nov 03 19:12:36.426452 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:36.426426    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a5a1dea-0df1-4268-b9d0-360fdcd887de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42918c8e-mttv9\" (UID: \"2a5a1dea-0df1-4268-b9d0-360fdcd887de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42918c8e-mttv9"
Nov 03 19:12:36.428032 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:36.428012    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a5a1dea-0df1-4268-b9d0-360fdcd887de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42918c8e-mttv9\" (UID: \"2a5a1dea-0df1-4268-b9d0-360fdcd887de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42918c8e-mttv9"
Nov 03 19:12:36.483886 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:36.483858    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mtpl\" (UniqueName: \"kubernetes.io/projected/2a5a1dea-0df1-4268-b9d0-360fdcd887de-kube-api-access-9mtpl\") pod \"maas-api-revoke-sub-42918c8e-mttv9\" (UID: \"2a5a1dea-0df1-4268-b9d0-360fdcd887de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42918c8e-mttv9"
Nov 03 19:12:36.503803 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:36.503780    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42918c8e-mttv9"
Nov 03 19:12:36.860541 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:12:36.860461    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:12:36.872968 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:36.872930    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42918c8e-mttv9"]
Nov 03 19:12:36.875030 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:12:36.875001    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5a1dea_0df1_4268_b9d0_360fdcd887de.slice/crio-0242c769968a98542cb341bc18b24dc2f786081ad4df6a4b8dacec8be1442fe7 WatchSource:0}: Error finding container 0242c769968a98542cb341bc18b24dc2f786081ad4df6a4b8dacec8be1442fe7: Status 404 returned error can't find the container with id 0242c769968a98542cb341bc18b24dc2f786081ad4df6a4b8dacec8be1442fe7
Nov 03 19:12:36.896412 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:36.896385    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42918c8e-mttv9" event={"ID":"2a5a1dea-0df1-4268-b9d0-360fdcd887de","Type":"ContainerStarted","Data":"0242c769968a98542cb341bc18b24dc2f786081ad4df6a4b8dacec8be1442fe7"}
Nov 03 19:12:37.903542 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:37.903505    6059 generic.go:356] "Generic (PLEG): container finished" podID="2a5a1dea-0df1-4268-b9d0-360fdcd887de" containerID="fa94fef8646a8a7f1af3c28236e02adc6c62a601b5bec4fb2906a27970f311bb" exitCode=0
Nov 03 19:12:37.903936 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:37.903570    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42918c8e-mttv9" event={"ID":"2a5a1dea-0df1-4268-b9d0-360fdcd887de","Type":"ContainerDied","Data":"fa94fef8646a8a7f1af3c28236e02adc6c62a601b5bec4fb2906a27970f311bb"}
Nov 03 19:12:39.138030 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:39.137995    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42918c8e-mttv9"
Nov 03 19:12:39.259831 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:39.259792    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a5a1dea-0df1-4268-b9d0-360fdcd887de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a5a1dea-0df1-4268-b9d0-360fdcd887de-maas-api-openshift-ca\") pod \"2a5a1dea-0df1-4268-b9d0-360fdcd887de\" (UID: \"2a5a1dea-0df1-4268-b9d0-360fdcd887de\") "
Nov 03 19:12:39.260029 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:39.259851    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a5a1dea-0df1-4268-b9d0-360fdcd887de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a5a1dea-0df1-4268-b9d0-360fdcd887de-maas-api-xks-ca\") pod \"2a5a1dea-0df1-4268-b9d0-360fdcd887de\" (UID: \"2a5a1dea-0df1-4268-b9d0-360fdcd887de\") "
Nov 03 19:12:39.260029 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:39.259966    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a5a1dea-0df1-4268-b9d0-360fdcd887de-kube-api-access-9mtpl\" (UniqueName: \"kubernetes.io/projected/2a5a1dea-0df1-4268-b9d0-360fdcd887de-kube-api-access-9mtpl\") pod \"2a5a1dea-0df1-4268-b9d0-360fdcd887de\" (UID: \"2a5a1dea-0df1-4268-b9d0-360fdcd887de\") "
Nov 03 19:12:39.262377 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:39.261469    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5a1dea-0df1-4268-b9d0-360fdcd887de-maas-api-openshift-ca" pod "2a5a1dea-0df1-4268-b9d0-360fdcd887de" (UID: "2a5a1dea-0df1-4268-b9d0-360fdcd887de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:12:39.271901 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:39.271867    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a5a1dea-0df1-4268-b9d0-360fdcd887de-maas-api-xks-ca" pod "2a5a1dea-0df1-4268-b9d0-360fdcd887de" (UID: "2a5a1dea-0df1-4268-b9d0-360fdcd887de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:12:39.281997 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:39.281965    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5a1dea-0df1-4268-b9d0-360fdcd887de-kube-api-access-9mtpl" pod "2a5a1dea-0df1-4268-b9d0-360fdcd887de" (UID: "2a5a1dea-0df1-4268-b9d0-360fdcd887de"). InnerVolumeSpecName "kube-api-access-9mtpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:12:39.362094 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:39.362015    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a5a1dea-0df1-4268-b9d0-360fdcd887de-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:39.362094 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:39.362052    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a5a1dea-0df1-4268-b9d0-360fdcd887de-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:39.362094 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:39.362071    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mtpl\" (UniqueName: \"kubernetes.io/projected/2a5a1dea-0df1-4268-b9d0-360fdcd887de-kube-api-access-9mtpl\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:39.926248 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:39.926192    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42918c8e-mttv9" event={"ID":"2a5a1dea-0df1-4268-b9d0-360fdcd887de","Type":"ContainerDied","Data":"0242c769968a98542cb341bc18b24dc2f786081ad4df6a4b8dacec8be1442fe7"}
Nov 03 19:12:39.926492 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:39.926310    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0242c769968a98542cb341bc18b24dc2f786081ad4df6a4b8dacec8be1442fe7"
Nov 03 19:12:39.926492 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:39.926446    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42918c8e-mttv9"
Nov 03 19:12:40.580971 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:40.580939    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 19:12:40.620788 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:40.620760    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a62h5t_510707fc-6270-4156-a83f-8d4ce5fbcfa0/extract/0.log"
Nov 03 19:12:40.764379 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:40.749832    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 19:12:40.846383 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:40.845742    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c5n7n_8906c21b-3233-4214-8ce1-f9189b79ccd3/ovn-acl-logging/0.log"
Nov 03 19:12:44.010797 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:44.010762    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:47.292171 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:47.292130    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5d3f5d6-pxblx"]
Nov 03 19:12:47.292718 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:47.292696    6059 cpu_manager.go:414] "containerMap: removing container" podUID="2a5a1dea-0df1-4268-b9d0-360fdcd887de" containerName="revoke-keys"
Nov 03 19:12:47.292822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:47.292719    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a5a1dea-0df1-4268-b9d0-360fdcd887de" containerName="revoke-keys"
Nov 03 19:12:47.292893 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:47.292840    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a5a1dea-0df1-4268-b9d0-360fdcd887de" containerName="revoke-keys"
Nov 03 19:12:47.296393 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:47.296370    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d3f5d6-pxblx"
Nov 03 19:12:47.351415 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:47.351379    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5d3f5d6-pxblx"]
Nov 03 19:12:47.394762 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:47.394719    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:12:47.394762 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:47.394736    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:12:47.394986 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:47.394777    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:12:47.394986 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:47.394902    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:12:47.463913 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:47.463877    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5d3f5d6-pxblx\" (UID: \"0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d3f5d6-pxblx"
Nov 03 19:12:47.464076 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:47.463949    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5d3f5d6-pxblx\" (UID: \"0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d3f5d6-pxblx"
Nov 03 19:12:47.464076 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:47.464030    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkrhn\" (UniqueName: \"kubernetes.io/projected/0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50-kube-api-access-hkrhn\") pod \"maas-api-revoke-sub-c5d3f5d6-pxblx\" (UID: \"0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d3f5d6-pxblx"
Nov 03 19:12:47.564956 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:47.564862    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5d3f5d6-pxblx\" (UID: \"0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d3f5d6-pxblx"
Nov 03 19:12:47.565113 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:47.564955    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5d3f5d6-pxblx\" (UID: \"0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d3f5d6-pxblx"
Nov 03 19:12:47.565113 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:47.564983    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkrhn\" (UniqueName: \"kubernetes.io/projected/0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50-kube-api-access-hkrhn\") pod \"maas-api-revoke-sub-c5d3f5d6-pxblx\" (UID: \"0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d3f5d6-pxblx"
Nov 03 19:12:47.565605 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:47.565573    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5d3f5d6-pxblx\" (UID: \"0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d3f5d6-pxblx"
Nov 03 19:12:47.567322 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:47.567294    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5d3f5d6-pxblx\" (UID: \"0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d3f5d6-pxblx"
Nov 03 19:12:47.623744 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:47.623707    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkrhn\" (UniqueName: \"kubernetes.io/projected/0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50-kube-api-access-hkrhn\") pod \"maas-api-revoke-sub-c5d3f5d6-pxblx\" (UID: \"0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d3f5d6-pxblx"
Nov 03 19:12:47.912932 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:47.912846    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d3f5d6-pxblx"
Nov 03 19:12:48.282394 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:48.282366    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5d3f5d6-pxblx"]
Nov 03 19:12:48.284864 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:12:48.284832    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea8ccde_e7a7_4d3c_acf1_d35e42abaa50.slice/crio-03d624a40a6ce52a43ec01100abd42a014a040a4f9e5b2b4270ce55c27ffa1dc WatchSource:0}: Error finding container 03d624a40a6ce52a43ec01100abd42a014a040a4f9e5b2b4270ce55c27ffa1dc: Status 404 returned error can't find the container with id 03d624a40a6ce52a43ec01100abd42a014a040a4f9e5b2b4270ce55c27ffa1dc
Nov 03 19:12:49.004783 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:49.004742    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d3f5d6-pxblx" event={"ID":"0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50","Type":"ContainerStarted","Data":"dc3b2e2ff63ea051747c3a562d850acb64d5fd9d9905e6f6525dbfbfd3507a94"}
Nov 03 19:12:49.005192 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:49.004806    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d3f5d6-pxblx" event={"ID":"0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50","Type":"ContainerStarted","Data":"03d624a40a6ce52a43ec01100abd42a014a040a4f9e5b2b4270ce55c27ffa1dc"}
Nov 03 19:12:49.116027 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:49.115976    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d3f5d6-pxblx" podStartSLOduration=2.115962212 podStartE2EDuration="2.115962212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:12:49.114427565 +0000 UTC m=+6909.912140350" watchObservedRunningTime="2026-11-03 19:12:49.115962212 +0000 UTC m=+6909.913674993"
Nov 03 19:12:50.011876 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:50.011835    6059 generic.go:356] "Generic (PLEG): container finished" podID="0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50" containerID="dc3b2e2ff63ea051747c3a562d850acb64d5fd9d9905e6f6525dbfbfd3507a94" exitCode=0
Nov 03 19:12:50.012300 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:50.011919    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d3f5d6-pxblx" event={"ID":"0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50","Type":"ContainerDied","Data":"dc3b2e2ff63ea051747c3a562d850acb64d5fd9d9905e6f6525dbfbfd3507a94"}
Nov 03 19:12:51.155227 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:51.155201    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d3f5d6-pxblx"
Nov 03 19:12:51.306468 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:51.305532    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50-maas-api-xks-ca\") pod \"0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50\" (UID: \"0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50\") "
Nov 03 19:12:51.306468 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:51.305631    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50-kube-api-access-hkrhn\" (UniqueName: \"kubernetes.io/projected/0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50-kube-api-access-hkrhn\") pod \"0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50\" (UID: \"0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50\") "
Nov 03 19:12:51.306468 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:51.305790    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50-maas-api-openshift-ca\") pod \"0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50\" (UID: \"0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50\") "
Nov 03 19:12:51.307071 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:51.306987    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50-maas-api-openshift-ca" pod "0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50" (UID: "0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:12:51.311000 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:51.310974    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50-kube-api-access-hkrhn" pod "0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50" (UID: "0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50"). InnerVolumeSpecName "kube-api-access-hkrhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:12:51.311636 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:51.311601    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50-maas-api-xks-ca" pod "0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50" (UID: "0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:12:51.407593 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:51.407549    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:51.407593 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:51.407592    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:51.407828 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:51.407609    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkrhn\" (UniqueName: \"kubernetes.io/projected/0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50-kube-api-access-hkrhn\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:52.028440 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:52.028408    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d3f5d6-pxblx"
Nov 03 19:12:52.029130 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:52.028397    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d3f5d6-pxblx" event={"ID":"0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50","Type":"ContainerDied","Data":"03d624a40a6ce52a43ec01100abd42a014a040a4f9e5b2b4270ce55c27ffa1dc"}
Nov 03 19:12:52.029130 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:52.029021    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03d624a40a6ce52a43ec01100abd42a014a040a4f9e5b2b4270ce55c27ffa1dc"
Nov 03 19:12:58.064833 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:58.064798    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31e680a3-m4r6j"]
Nov 03 19:12:58.065295 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:58.065253    6059 cpu_manager.go:414] "containerMap: removing container" podUID="0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50" containerName="revoke-keys"
Nov 03 19:12:58.065295 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:58.065263    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50" containerName="revoke-keys"
Nov 03 19:12:58.065396 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:58.065377    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50" containerName="revoke-keys"
Nov 03 19:12:58.068871 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:58.068854    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e680a3-m4r6j"
Nov 03 19:12:58.120072 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:58.120039    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:12:58.120302 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:58.120127    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:12:58.122015 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:58.121991    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31e680a3-m4r6j"]
Nov 03 19:12:58.129624 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:58.129598    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6b058ce-0bd8-48fa-baaa-7541c74a63ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31e680a3-m4r6j\" (UID: \"f6b058ce-0bd8-48fa-baaa-7541c74a63ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e680a3-m4r6j"
Nov 03 19:12:58.129761 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:58.129635    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6b058ce-0bd8-48fa-baaa-7541c74a63ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31e680a3-m4r6j\" (UID: \"f6b058ce-0bd8-48fa-baaa-7541c74a63ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e680a3-m4r6j"
Nov 03 19:12:58.129761 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:58.129669    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvxp5\" (UniqueName: \"kubernetes.io/projected/f6b058ce-0bd8-48fa-baaa-7541c74a63ea-kube-api-access-kvxp5\") pod \"maas-api-revoke-sub-31e680a3-m4r6j\" (UID: \"f6b058ce-0bd8-48fa-baaa-7541c74a63ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e680a3-m4r6j"
Nov 03 19:12:58.166327 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:58.166287    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:12:58.166327 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:58.166287    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:12:58.231341 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:58.231309    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvxp5\" (UniqueName: \"kubernetes.io/projected/f6b058ce-0bd8-48fa-baaa-7541c74a63ea-kube-api-access-kvxp5\") pod \"maas-api-revoke-sub-31e680a3-m4r6j\" (UID: \"f6b058ce-0bd8-48fa-baaa-7541c74a63ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e680a3-m4r6j"
Nov 03 19:12:58.231545 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:58.231464    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6b058ce-0bd8-48fa-baaa-7541c74a63ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31e680a3-m4r6j\" (UID: \"f6b058ce-0bd8-48fa-baaa-7541c74a63ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e680a3-m4r6j"
Nov 03 19:12:58.231545 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:58.231485    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6b058ce-0bd8-48fa-baaa-7541c74a63ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31e680a3-m4r6j\" (UID: \"f6b058ce-0bd8-48fa-baaa-7541c74a63ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e680a3-m4r6j"
Nov 03 19:12:58.232192 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:58.232162    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6b058ce-0bd8-48fa-baaa-7541c74a63ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31e680a3-m4r6j\" (UID: \"f6b058ce-0bd8-48fa-baaa-7541c74a63ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e680a3-m4r6j"
Nov 03 19:12:58.233741 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:58.233718    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6b058ce-0bd8-48fa-baaa-7541c74a63ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31e680a3-m4r6j\" (UID: \"f6b058ce-0bd8-48fa-baaa-7541c74a63ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e680a3-m4r6j"
Nov 03 19:12:58.289688 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:58.289658    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvxp5\" (UniqueName: \"kubernetes.io/projected/f6b058ce-0bd8-48fa-baaa-7541c74a63ea-kube-api-access-kvxp5\") pod \"maas-api-revoke-sub-31e680a3-m4r6j\" (UID: \"f6b058ce-0bd8-48fa-baaa-7541c74a63ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e680a3-m4r6j"
Nov 03 19:12:58.382044 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:58.381969    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e680a3-m4r6j"
Nov 03 19:12:58.753063 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:58.753021    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31e680a3-m4r6j"]
Nov 03 19:12:58.754671 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:12:58.754638    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b058ce_0bd8_48fa_baaa_7541c74a63ea.slice/crio-767b93030dfd0f1ceee649453935feece6657c334d8b62cebcba97c6f0665392 WatchSource:0}: Error finding container 767b93030dfd0f1ceee649453935feece6657c334d8b62cebcba97c6f0665392: Status 404 returned error can't find the container with id 767b93030dfd0f1ceee649453935feece6657c334d8b62cebcba97c6f0665392
Nov 03 19:12:59.157694 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:59.157657    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e680a3-m4r6j" event={"ID":"f6b058ce-0bd8-48fa-baaa-7541c74a63ea","Type":"ContainerStarted","Data":"4d5607b22836c1eaa4648635684949d51ed52f29195c0f3b95c7034dffd0b0fa"}
Nov 03 19:12:59.158079 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:59.157717    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e680a3-m4r6j" event={"ID":"f6b058ce-0bd8-48fa-baaa-7541c74a63ea","Type":"ContainerStarted","Data":"767b93030dfd0f1ceee649453935feece6657c334d8b62cebcba97c6f0665392"}
Nov 03 19:12:59.269605 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:12:59.269555    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e680a3-m4r6j" podStartSLOduration=1.269541883 podStartE2EDuration="1.269541883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:12:59.268909949 +0000 UTC m=+6920.066622730" watchObservedRunningTime="2026-11-03 19:12:59.269541883 +0000 UTC m=+6920.067254664"
Nov 03 19:13:00.165943 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:00.165905    6059 generic.go:356] "Generic (PLEG): container finished" podID="f6b058ce-0bd8-48fa-baaa-7541c74a63ea" containerID="4d5607b22836c1eaa4648635684949d51ed52f29195c0f3b95c7034dffd0b0fa" exitCode=0
Nov 03 19:13:00.166371 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:00.165982    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e680a3-m4r6j" event={"ID":"f6b058ce-0bd8-48fa-baaa-7541c74a63ea","Type":"ContainerDied","Data":"4d5607b22836c1eaa4648635684949d51ed52f29195c0f3b95c7034dffd0b0fa"}
Nov 03 19:13:01.303848 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:01.303820    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e680a3-m4r6j"
Nov 03 19:13:01.463488 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:01.463450    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b058ce-0bd8-48fa-baaa-7541c74a63ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6b058ce-0bd8-48fa-baaa-7541c74a63ea-maas-api-openshift-ca\") pod \"f6b058ce-0bd8-48fa-baaa-7541c74a63ea\" (UID: \"f6b058ce-0bd8-48fa-baaa-7541c74a63ea\") "
Nov 03 19:13:01.463691 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:01.463579    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6b058ce-0bd8-48fa-baaa-7541c74a63ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6b058ce-0bd8-48fa-baaa-7541c74a63ea-maas-api-xks-ca\") pod \"f6b058ce-0bd8-48fa-baaa-7541c74a63ea\" (UID: \"f6b058ce-0bd8-48fa-baaa-7541c74a63ea\") "
Nov 03 19:13:01.463691 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:01.463624    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b058ce-0bd8-48fa-baaa-7541c74a63ea-kube-api-access-kvxp5\" (UniqueName: \"kubernetes.io/projected/f6b058ce-0bd8-48fa-baaa-7541c74a63ea-kube-api-access-kvxp5\") pod \"f6b058ce-0bd8-48fa-baaa-7541c74a63ea\" (UID: \"f6b058ce-0bd8-48fa-baaa-7541c74a63ea\") "
Nov 03 19:13:01.463875 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:01.463845    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b058ce-0bd8-48fa-baaa-7541c74a63ea-maas-api-openshift-ca" pod "f6b058ce-0bd8-48fa-baaa-7541c74a63ea" (UID: "f6b058ce-0bd8-48fa-baaa-7541c74a63ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:13:01.464381 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:01.464343    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6b058ce-0bd8-48fa-baaa-7541c74a63ea-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:01.466203 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:01.466183    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6b058ce-0bd8-48fa-baaa-7541c74a63ea-maas-api-xks-ca" pod "f6b058ce-0bd8-48fa-baaa-7541c74a63ea" (UID: "f6b058ce-0bd8-48fa-baaa-7541c74a63ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:13:01.466292 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:01.466206    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b058ce-0bd8-48fa-baaa-7541c74a63ea-kube-api-access-kvxp5" pod "f6b058ce-0bd8-48fa-baaa-7541c74a63ea" (UID: "f6b058ce-0bd8-48fa-baaa-7541c74a63ea"). InnerVolumeSpecName "kube-api-access-kvxp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:13:01.565671 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:01.565636    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6b058ce-0bd8-48fa-baaa-7541c74a63ea-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:01.565671 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:01.565665    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvxp5\" (UniqueName: \"kubernetes.io/projected/f6b058ce-0bd8-48fa-baaa-7541c74a63ea-kube-api-access-kvxp5\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:02.178763 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:02.178726    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e680a3-m4r6j" event={"ID":"f6b058ce-0bd8-48fa-baaa-7541c74a63ea","Type":"ContainerDied","Data":"767b93030dfd0f1ceee649453935feece6657c334d8b62cebcba97c6f0665392"}
Nov 03 19:13:02.178940 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:02.178783    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="767b93030dfd0f1ceee649453935feece6657c334d8b62cebcba97c6f0665392"
Nov 03 19:13:02.178940 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:02.178738    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e680a3-m4r6j"
Nov 03 19:13:06.860193 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:13:06.860154    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:13:08.896155 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:08.896105    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4a139f5-d99b2"]
Nov 03 19:13:08.896782 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:08.896761    6059 cpu_manager.go:414] "containerMap: removing container" podUID="f6b058ce-0bd8-48fa-baaa-7541c74a63ea" containerName="revoke-keys"
Nov 03 19:13:08.896879 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:08.896784    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b058ce-0bd8-48fa-baaa-7541c74a63ea" containerName="revoke-keys"
Nov 03 19:13:08.896939 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:08.896931    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b058ce-0bd8-48fa-baaa-7541c74a63ea" containerName="revoke-keys"
Nov 03 19:13:08.900200 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:08.900179    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4a139f5-d99b2"
Nov 03 19:13:08.955726 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:08.955693    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4a139f5-d99b2"]
Nov 03 19:13:08.999210 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:08.999178    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:13:08.999210 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:08.999200    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:13:08.999463 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:08.999258    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:13:08.999463 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:08.999266    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:13:09.046813 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:09.046775    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b86f325-e858-499c-8ecb-0fa987fa5a88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4a139f5-d99b2\" (UID: \"2b86f325-e858-499c-8ecb-0fa987fa5a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4a139f5-d99b2"
Nov 03 19:13:09.046991 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:09.046827    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b86f325-e858-499c-8ecb-0fa987fa5a88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4a139f5-d99b2\" (UID: \"2b86f325-e858-499c-8ecb-0fa987fa5a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4a139f5-d99b2"
Nov 03 19:13:09.046991 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:09.046955    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfhq5\" (UniqueName: \"kubernetes.io/projected/2b86f325-e858-499c-8ecb-0fa987fa5a88-kube-api-access-bfhq5\") pod \"maas-api-revoke-sub-f4a139f5-d99b2\" (UID: \"2b86f325-e858-499c-8ecb-0fa987fa5a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4a139f5-d99b2"
Nov 03 19:13:09.148093 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:09.148014    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b86f325-e858-499c-8ecb-0fa987fa5a88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4a139f5-d99b2\" (UID: \"2b86f325-e858-499c-8ecb-0fa987fa5a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4a139f5-d99b2"
Nov 03 19:13:09.148093 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:09.148059    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b86f325-e858-499c-8ecb-0fa987fa5a88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4a139f5-d99b2\" (UID: \"2b86f325-e858-499c-8ecb-0fa987fa5a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4a139f5-d99b2"
Nov 03 19:13:09.148282 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:09.148107    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfhq5\" (UniqueName: \"kubernetes.io/projected/2b86f325-e858-499c-8ecb-0fa987fa5a88-kube-api-access-bfhq5\") pod \"maas-api-revoke-sub-f4a139f5-d99b2\" (UID: \"2b86f325-e858-499c-8ecb-0fa987fa5a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4a139f5-d99b2"
Nov 03 19:13:09.148721 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:09.148694    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b86f325-e858-499c-8ecb-0fa987fa5a88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4a139f5-d99b2\" (UID: \"2b86f325-e858-499c-8ecb-0fa987fa5a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4a139f5-d99b2"
Nov 03 19:13:09.150317 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:09.150297    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b86f325-e858-499c-8ecb-0fa987fa5a88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4a139f5-d99b2\" (UID: \"2b86f325-e858-499c-8ecb-0fa987fa5a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4a139f5-d99b2"
Nov 03 19:13:09.206684 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:09.206658    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfhq5\" (UniqueName: \"kubernetes.io/projected/2b86f325-e858-499c-8ecb-0fa987fa5a88-kube-api-access-bfhq5\") pod \"maas-api-revoke-sub-f4a139f5-d99b2\" (UID: \"2b86f325-e858-499c-8ecb-0fa987fa5a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4a139f5-d99b2"
Nov 03 19:13:09.215959 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:09.215930    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4a139f5-d99b2"
Nov 03 19:13:09.587539 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:09.587516    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4a139f5-d99b2"]
Nov 03 19:13:09.590076 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:13:09.590044    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b86f325_e858_499c_8ecb_0fa987fa5a88.slice/crio-88013b549778a6efacb6f2508c731fda214d3979b2559f624f90ad45f0cc02df WatchSource:0}: Error finding container 88013b549778a6efacb6f2508c731fda214d3979b2559f624f90ad45f0cc02df: Status 404 returned error can't find the container with id 88013b549778a6efacb6f2508c731fda214d3979b2559f624f90ad45f0cc02df
Nov 03 19:13:10.227703 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:10.227653    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4a139f5-d99b2" event={"ID":"2b86f325-e858-499c-8ecb-0fa987fa5a88","Type":"ContainerStarted","Data":"4609550462d1f50566c412ee3d2e67cedc246eb8a96a5a9c4a9dd63e709daba6"}
Nov 03 19:13:10.228330 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:10.227739    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4a139f5-d99b2" event={"ID":"2b86f325-e858-499c-8ecb-0fa987fa5a88","Type":"ContainerStarted","Data":"88013b549778a6efacb6f2508c731fda214d3979b2559f624f90ad45f0cc02df"}
Nov 03 19:13:10.341308 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:10.341252    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4a139f5-d99b2" podStartSLOduration=2.341238445 podStartE2EDuration="2.341238445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:13:10.338955161 +0000 UTC m=+6931.136667943" watchObservedRunningTime="2026-11-03 19:13:10.341238445 +0000 UTC m=+6931.138951226"
Nov 03 19:13:10.373755 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:10.373726    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:11.235133 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:11.235097    6059 generic.go:356] "Generic (PLEG): container finished" podID="2b86f325-e858-499c-8ecb-0fa987fa5a88" containerID="4609550462d1f50566c412ee3d2e67cedc246eb8a96a5a9c4a9dd63e709daba6" exitCode=0
Nov 03 19:13:11.235548 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:11.235157    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4a139f5-d99b2" event={"ID":"2b86f325-e858-499c-8ecb-0fa987fa5a88","Type":"ContainerDied","Data":"4609550462d1f50566c412ee3d2e67cedc246eb8a96a5a9c4a9dd63e709daba6"}
Nov 03 19:13:12.446686 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:12.446650    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4a139f5-d99b2"
Nov 03 19:13:12.490767 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:12.490724    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b86f325-e858-499c-8ecb-0fa987fa5a88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b86f325-e858-499c-8ecb-0fa987fa5a88-maas-api-xks-ca\") pod \"2b86f325-e858-499c-8ecb-0fa987fa5a88\" (UID: \"2b86f325-e858-499c-8ecb-0fa987fa5a88\") "
Nov 03 19:13:12.490985 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:12.490848    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b86f325-e858-499c-8ecb-0fa987fa5a88-kube-api-access-bfhq5\" (UniqueName: \"kubernetes.io/projected/2b86f325-e858-499c-8ecb-0fa987fa5a88-kube-api-access-bfhq5\") pod \"2b86f325-e858-499c-8ecb-0fa987fa5a88\" (UID: \"2b86f325-e858-499c-8ecb-0fa987fa5a88\") "
Nov 03 19:13:12.491071 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:12.491003    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b86f325-e858-499c-8ecb-0fa987fa5a88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b86f325-e858-499c-8ecb-0fa987fa5a88-maas-api-openshift-ca\") pod \"2b86f325-e858-499c-8ecb-0fa987fa5a88\" (UID: \"2b86f325-e858-499c-8ecb-0fa987fa5a88\") "
Nov 03 19:13:12.491465 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:12.491422    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b86f325-e858-499c-8ecb-0fa987fa5a88-maas-api-openshift-ca" pod "2b86f325-e858-499c-8ecb-0fa987fa5a88" (UID: "2b86f325-e858-499c-8ecb-0fa987fa5a88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:13:12.493586 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:12.493548    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b86f325-e858-499c-8ecb-0fa987fa5a88-maas-api-xks-ca" pod "2b86f325-e858-499c-8ecb-0fa987fa5a88" (UID: "2b86f325-e858-499c-8ecb-0fa987fa5a88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:13:12.493881 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:12.493853    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b86f325-e858-499c-8ecb-0fa987fa5a88-kube-api-access-bfhq5" pod "2b86f325-e858-499c-8ecb-0fa987fa5a88" (UID: "2b86f325-e858-499c-8ecb-0fa987fa5a88"). InnerVolumeSpecName "kube-api-access-bfhq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:13:12.593055 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:12.592999    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b86f325-e858-499c-8ecb-0fa987fa5a88-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:12.593055 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:12.593042    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfhq5\" (UniqueName: \"kubernetes.io/projected/2b86f325-e858-499c-8ecb-0fa987fa5a88-kube-api-access-bfhq5\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:12.593055 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:12.593063    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b86f325-e858-499c-8ecb-0fa987fa5a88-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:13.269390 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:13.265398    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4a139f5-d99b2" event={"ID":"2b86f325-e858-499c-8ecb-0fa987fa5a88","Type":"ContainerDied","Data":"88013b549778a6efacb6f2508c731fda214d3979b2559f624f90ad45f0cc02df"}
Nov 03 19:13:13.269390 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:13.265500    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88013b549778a6efacb6f2508c731fda214d3979b2559f624f90ad45f0cc02df"
Nov 03 19:13:13.269390 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:13.265576    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4a139f5-d99b2"
Nov 03 19:13:14.074379 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:14.073563    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3390c81f-gwzqj"]
Nov 03 19:13:14.157381 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:14.146990    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3390c81f-gwzqj"]
Nov 03 19:13:15.213496 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:15.213452    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-c55ww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:13:15.214089 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:15.214053    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww" podUID="acc06524-297b-4b57-b0a2-4b2a92d4a00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:13:15.247502 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:15.247460    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bwg8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:13:15.247795 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:15.247763    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p" podUID="c17b0b8a-a3a3-4824-acc3-ce83fb644937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:13:15.339932 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:15.339894    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b64a378-7575-4d35-acec-4bf6a3b41959" path="/var/lib/kubelet/pods/8b64a378-7575-4d35-acec-4bf6a3b41959/volumes"
Nov 03 19:13:19.713526 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:19.713448    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1317113d-cddl4"]
Nov 03 19:13:19.714057 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:19.714038    6059 cpu_manager.go:414] "containerMap: removing container" podUID="2b86f325-e858-499c-8ecb-0fa987fa5a88" containerName="revoke-keys"
Nov 03 19:13:19.714118 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:19.714059    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b86f325-e858-499c-8ecb-0fa987fa5a88" containerName="revoke-keys"
Nov 03 19:13:19.714189 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:19.714179    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b86f325-e858-499c-8ecb-0fa987fa5a88" containerName="revoke-keys"
Nov 03 19:13:19.718488 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:19.718469    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1317113d-cddl4"
Nov 03 19:13:19.760254 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:19.760225    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd4f6b4-1539-423f-914a-5eac9068fd48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1317113d-cddl4\" (UID: \"ddd4f6b4-1539-423f-914a-5eac9068fd48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1317113d-cddl4"
Nov 03 19:13:19.760388 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:19.760294    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwb2t\" (UniqueName: \"kubernetes.io/projected/ddd4f6b4-1539-423f-914a-5eac9068fd48-kube-api-access-lwb2t\") pod \"maas-api-revoke-sub-1317113d-cddl4\" (UID: \"ddd4f6b4-1539-423f-914a-5eac9068fd48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1317113d-cddl4"
Nov 03 19:13:19.760438 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:19.760398    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd4f6b4-1539-423f-914a-5eac9068fd48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1317113d-cddl4\" (UID: \"ddd4f6b4-1539-423f-914a-5eac9068fd48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1317113d-cddl4"
Nov 03 19:13:19.771691 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:19.771664    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1317113d-cddl4"]
Nov 03 19:13:19.815791 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:19.815758    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:13:19.815791 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:19.815781    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:13:19.815979 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:19.815781    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:13:19.815979 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:19.815859    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:13:19.861288 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:19.861262    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwb2t\" (UniqueName: \"kubernetes.io/projected/ddd4f6b4-1539-423f-914a-5eac9068fd48-kube-api-access-lwb2t\") pod \"maas-api-revoke-sub-1317113d-cddl4\" (UID: \"ddd4f6b4-1539-423f-914a-5eac9068fd48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1317113d-cddl4"
Nov 03 19:13:19.861455 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:19.861290    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd4f6b4-1539-423f-914a-5eac9068fd48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1317113d-cddl4\" (UID: \"ddd4f6b4-1539-423f-914a-5eac9068fd48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1317113d-cddl4"
Nov 03 19:13:19.861455 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:19.861346    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd4f6b4-1539-423f-914a-5eac9068fd48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1317113d-cddl4\" (UID: \"ddd4f6b4-1539-423f-914a-5eac9068fd48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1317113d-cddl4"
Nov 03 19:13:19.862007 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:19.861985    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd4f6b4-1539-423f-914a-5eac9068fd48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1317113d-cddl4\" (UID: \"ddd4f6b4-1539-423f-914a-5eac9068fd48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1317113d-cddl4"
Nov 03 19:13:19.863669 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:19.863652    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd4f6b4-1539-423f-914a-5eac9068fd48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1317113d-cddl4\" (UID: \"ddd4f6b4-1539-423f-914a-5eac9068fd48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1317113d-cddl4"
Nov 03 19:13:19.920417 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:19.920344    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwb2t\" (UniqueName: \"kubernetes.io/projected/ddd4f6b4-1539-423f-914a-5eac9068fd48-kube-api-access-lwb2t\") pod \"maas-api-revoke-sub-1317113d-cddl4\" (UID: \"ddd4f6b4-1539-423f-914a-5eac9068fd48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1317113d-cddl4"
Nov 03 19:13:20.030859 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:20.030780    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1317113d-cddl4"
Nov 03 19:13:20.402551 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:20.402523    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1317113d-cddl4"]
Nov 03 19:13:20.404428 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:13:20.404397    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd4f6b4_1539_423f_914a_5eac9068fd48.slice/crio-62343b932369ab2c3d0ac0636ee6348035ca84facbb52124222745657a65f8fa WatchSource:0}: Error finding container 62343b932369ab2c3d0ac0636ee6348035ca84facbb52124222745657a65f8fa: Status 404 returned error can't find the container with id 62343b932369ab2c3d0ac0636ee6348035ca84facbb52124222745657a65f8fa
Nov 03 19:13:21.409296 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:21.409255    6059 generic.go:356] "Generic (PLEG): container finished" podID="ddd4f6b4-1539-423f-914a-5eac9068fd48" containerID="c1136008c3d666439f82c633ea2e22004bc62bba4d27825eb7960199e6498c3e" exitCode=0
Nov 03 19:13:21.409688 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:21.409332    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1317113d-cddl4" event={"ID":"ddd4f6b4-1539-423f-914a-5eac9068fd48","Type":"ContainerDied","Data":"c1136008c3d666439f82c633ea2e22004bc62bba4d27825eb7960199e6498c3e"}
Nov 03 19:13:21.409688 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:21.409396    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1317113d-cddl4" event={"ID":"ddd4f6b4-1539-423f-914a-5eac9068fd48","Type":"ContainerStarted","Data":"62343b932369ab2c3d0ac0636ee6348035ca84facbb52124222745657a65f8fa"}
Nov 03 19:13:22.616798 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:22.616768    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1317113d-cddl4"
Nov 03 19:13:22.691136 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:22.691029    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddd4f6b4-1539-423f-914a-5eac9068fd48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd4f6b4-1539-423f-914a-5eac9068fd48-maas-api-xks-ca\") pod \"ddd4f6b4-1539-423f-914a-5eac9068fd48\" (UID: \"ddd4f6b4-1539-423f-914a-5eac9068fd48\") "
Nov 03 19:13:22.691136 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:22.691121    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd4f6b4-1539-423f-914a-5eac9068fd48-kube-api-access-lwb2t\" (UniqueName: \"kubernetes.io/projected/ddd4f6b4-1539-423f-914a-5eac9068fd48-kube-api-access-lwb2t\") pod \"ddd4f6b4-1539-423f-914a-5eac9068fd48\" (UID: \"ddd4f6b4-1539-423f-914a-5eac9068fd48\") "
Nov 03 19:13:22.691406 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:22.691170    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddd4f6b4-1539-423f-914a-5eac9068fd48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd4f6b4-1539-423f-914a-5eac9068fd48-maas-api-openshift-ca\") pod \"ddd4f6b4-1539-423f-914a-5eac9068fd48\" (UID: \"ddd4f6b4-1539-423f-914a-5eac9068fd48\") "
Nov 03 19:13:22.691918 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:22.691829    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd4f6b4-1539-423f-914a-5eac9068fd48-maas-api-openshift-ca" pod "ddd4f6b4-1539-423f-914a-5eac9068fd48" (UID: "ddd4f6b4-1539-423f-914a-5eac9068fd48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:13:22.693843 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:22.693803    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddd4f6b4-1539-423f-914a-5eac9068fd48-maas-api-xks-ca" pod "ddd4f6b4-1539-423f-914a-5eac9068fd48" (UID: "ddd4f6b4-1539-423f-914a-5eac9068fd48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:13:22.693991 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:22.693914    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd4f6b4-1539-423f-914a-5eac9068fd48-kube-api-access-lwb2t" pod "ddd4f6b4-1539-423f-914a-5eac9068fd48" (UID: "ddd4f6b4-1539-423f-914a-5eac9068fd48"). InnerVolumeSpecName "kube-api-access-lwb2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:13:22.793326 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:22.793281    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd4f6b4-1539-423f-914a-5eac9068fd48-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:22.793326 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:22.793325    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwb2t\" (UniqueName: \"kubernetes.io/projected/ddd4f6b4-1539-423f-914a-5eac9068fd48-kube-api-access-lwb2t\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:22.793657 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:22.793343    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd4f6b4-1539-423f-914a-5eac9068fd48-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:23.429816 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:23.429773    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1317113d-cddl4"
Nov 03 19:13:23.429816 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:23.429791    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1317113d-cddl4" event={"ID":"ddd4f6b4-1539-423f-914a-5eac9068fd48","Type":"ContainerDied","Data":"62343b932369ab2c3d0ac0636ee6348035ca84facbb52124222745657a65f8fa"}
Nov 03 19:13:23.430085 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:23.429854    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62343b932369ab2c3d0ac0636ee6348035ca84facbb52124222745657a65f8fa"
Nov 03 19:13:25.057844 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:25.057776    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn"]
Nov 03 19:13:25.060655 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:25.060624    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccdf6d1d-95nqn"]
Nov 03 19:13:25.413431 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:25.392233    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ae8d86b-4382-417f-9eb5-e2709576686f" path="/var/lib/kubelet/pods/9ae8d86b-4382-417f-9eb5-e2709576686f/volumes"
Nov 03 19:13:30.490609 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:30.490572    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9a946ce-46s5c"]
Nov 03 19:13:30.491090 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:30.491071    6059 cpu_manager.go:414] "containerMap: removing container" podUID="ddd4f6b4-1539-423f-914a-5eac9068fd48" containerName="revoke-keys"
Nov 03 19:13:30.491150 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:30.491121    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd4f6b4-1539-423f-914a-5eac9068fd48" containerName="revoke-keys"
Nov 03 19:13:30.491235 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:30.491222    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd4f6b4-1539-423f-914a-5eac9068fd48" containerName="revoke-keys"
Nov 03 19:13:30.495819 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:30.495798    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a946ce-46s5c"
Nov 03 19:13:30.546071 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:30.546037    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:13:30.548949 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:30.548918    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9a946ce-46s5c"]
Nov 03 19:13:30.592917 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:30.592886    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:13:30.592917 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:30.592910    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:13:30.593147 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:30.593084    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:13:30.645791 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:30.645718    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82ed33de-befa-453e-a0c2-1672f37ef6c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9a946ce-46s5c\" (UID: \"82ed33de-befa-453e-a0c2-1672f37ef6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a946ce-46s5c"
Nov 03 19:13:30.646024 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:30.645893    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkbz2\" (UniqueName: \"kubernetes.io/projected/82ed33de-befa-453e-a0c2-1672f37ef6c8-kube-api-access-hkbz2\") pod \"maas-api-revoke-sub-d9a946ce-46s5c\" (UID: \"82ed33de-befa-453e-a0c2-1672f37ef6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a946ce-46s5c"
Nov 03 19:13:30.646024 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:30.645960    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82ed33de-befa-453e-a0c2-1672f37ef6c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9a946ce-46s5c\" (UID: \"82ed33de-befa-453e-a0c2-1672f37ef6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a946ce-46s5c"
Nov 03 19:13:30.746980 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:30.746875    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82ed33de-befa-453e-a0c2-1672f37ef6c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9a946ce-46s5c\" (UID: \"82ed33de-befa-453e-a0c2-1672f37ef6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a946ce-46s5c"
Nov 03 19:13:30.746980 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:30.746958    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkbz2\" (UniqueName: \"kubernetes.io/projected/82ed33de-befa-453e-a0c2-1672f37ef6c8-kube-api-access-hkbz2\") pod \"maas-api-revoke-sub-d9a946ce-46s5c\" (UID: \"82ed33de-befa-453e-a0c2-1672f37ef6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a946ce-46s5c"
Nov 03 19:13:30.746980 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:30.746980    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82ed33de-befa-453e-a0c2-1672f37ef6c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9a946ce-46s5c\" (UID: \"82ed33de-befa-453e-a0c2-1672f37ef6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a946ce-46s5c"
Nov 03 19:13:30.747709 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:30.747686    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82ed33de-befa-453e-a0c2-1672f37ef6c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9a946ce-46s5c\" (UID: \"82ed33de-befa-453e-a0c2-1672f37ef6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a946ce-46s5c"
Nov 03 19:13:30.749240 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:30.749219    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82ed33de-befa-453e-a0c2-1672f37ef6c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9a946ce-46s5c\" (UID: \"82ed33de-befa-453e-a0c2-1672f37ef6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a946ce-46s5c"
Nov 03 19:13:30.805389 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:30.805325    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkbz2\" (UniqueName: \"kubernetes.io/projected/82ed33de-befa-453e-a0c2-1672f37ef6c8-kube-api-access-hkbz2\") pod \"maas-api-revoke-sub-d9a946ce-46s5c\" (UID: \"82ed33de-befa-453e-a0c2-1672f37ef6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a946ce-46s5c"
Nov 03 19:13:30.809186 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:30.809162    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a946ce-46s5c"
Nov 03 19:13:31.182946 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:31.182908    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9a946ce-46s5c"]
Nov 03 19:13:31.184762 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:13:31.184732    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ed33de_befa_453e_a0c2_1672f37ef6c8.slice/crio-6a51ecff679fb7fcbd047af02753cadd987b9ae50f99c6fc619615af44251f1d WatchSource:0}: Error finding container 6a51ecff679fb7fcbd047af02753cadd987b9ae50f99c6fc619615af44251f1d: Status 404 returned error can't find the container with id 6a51ecff679fb7fcbd047af02753cadd987b9ae50f99c6fc619615af44251f1d
Nov 03 19:13:31.602226 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:31.602190    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a946ce-46s5c" event={"ID":"82ed33de-befa-453e-a0c2-1672f37ef6c8","Type":"ContainerStarted","Data":"10c7e8fd76afce683a69bdf838629363b1d0bef237cd696f6841db8651d4509e"}
Nov 03 19:13:31.602645 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:31.602250    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a946ce-46s5c" event={"ID":"82ed33de-befa-453e-a0c2-1672f37ef6c8","Type":"ContainerStarted","Data":"6a51ecff679fb7fcbd047af02753cadd987b9ae50f99c6fc619615af44251f1d"}
Nov 03 19:13:31.714155 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:31.714104    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a946ce-46s5c" podStartSLOduration=1.714089696 podStartE2EDuration="1.714089696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:13:31.713281322 +0000 UTC m=+6952.510994103" watchObservedRunningTime="2026-11-03 19:13:31.714089696 +0000 UTC m=+6952.511802477"
Nov 03 19:13:32.609612 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:32.609572    6059 generic.go:356] "Generic (PLEG): container finished" podID="82ed33de-befa-453e-a0c2-1672f37ef6c8" containerID="10c7e8fd76afce683a69bdf838629363b1d0bef237cd696f6841db8651d4509e" exitCode=0
Nov 03 19:13:32.610039 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:32.609621    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a946ce-46s5c" event={"ID":"82ed33de-befa-453e-a0c2-1672f37ef6c8","Type":"ContainerDied","Data":"10c7e8fd76afce683a69bdf838629363b1d0bef237cd696f6841db8651d4509e"}
Nov 03 19:13:33.814670 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:33.814635    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a946ce-46s5c"
Nov 03 19:13:33.881252 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:33.881110    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ed33de-befa-453e-a0c2-1672f37ef6c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82ed33de-befa-453e-a0c2-1672f37ef6c8-maas-api-openshift-ca\") pod \"82ed33de-befa-453e-a0c2-1672f37ef6c8\" (UID: \"82ed33de-befa-453e-a0c2-1672f37ef6c8\") "
Nov 03 19:13:33.881252 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:33.881206    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ed33de-befa-453e-a0c2-1672f37ef6c8-kube-api-access-hkbz2\" (UniqueName: \"kubernetes.io/projected/82ed33de-befa-453e-a0c2-1672f37ef6c8-kube-api-access-hkbz2\") pod \"82ed33de-befa-453e-a0c2-1672f37ef6c8\" (UID: \"82ed33de-befa-453e-a0c2-1672f37ef6c8\") "
Nov 03 19:13:33.881537 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:33.881312    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82ed33de-befa-453e-a0c2-1672f37ef6c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82ed33de-befa-453e-a0c2-1672f37ef6c8-maas-api-xks-ca\") pod \"82ed33de-befa-453e-a0c2-1672f37ef6c8\" (UID: \"82ed33de-befa-453e-a0c2-1672f37ef6c8\") "
Nov 03 19:13:33.881958 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:33.881931    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ed33de-befa-453e-a0c2-1672f37ef6c8-maas-api-openshift-ca" pod "82ed33de-befa-453e-a0c2-1672f37ef6c8" (UID: "82ed33de-befa-453e-a0c2-1672f37ef6c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:13:33.884091 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:33.884065    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ed33de-befa-453e-a0c2-1672f37ef6c8-kube-api-access-hkbz2" pod "82ed33de-befa-453e-a0c2-1672f37ef6c8" (UID: "82ed33de-befa-453e-a0c2-1672f37ef6c8"). InnerVolumeSpecName "kube-api-access-hkbz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:13:33.884219 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:33.884124    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82ed33de-befa-453e-a0c2-1672f37ef6c8-maas-api-xks-ca" pod "82ed33de-befa-453e-a0c2-1672f37ef6c8" (UID: "82ed33de-befa-453e-a0c2-1672f37ef6c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:13:33.982788 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:33.982738    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82ed33de-befa-453e-a0c2-1672f37ef6c8-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:33.982788 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:33.982781    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkbz2\" (UniqueName: \"kubernetes.io/projected/82ed33de-befa-453e-a0c2-1672f37ef6c8-kube-api-access-hkbz2\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:33.983079 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:33.982798    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82ed33de-befa-453e-a0c2-1672f37ef6c8-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:34.645298 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:34.645254    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a946ce-46s5c" event={"ID":"82ed33de-befa-453e-a0c2-1672f37ef6c8","Type":"ContainerDied","Data":"6a51ecff679fb7fcbd047af02753cadd987b9ae50f99c6fc619615af44251f1d"}
Nov 03 19:13:34.645650 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:34.645610    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a51ecff679fb7fcbd047af02753cadd987b9ae50f99c6fc619615af44251f1d"
Nov 03 19:13:34.645884 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:34.645868    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a946ce-46s5c"
Nov 03 19:13:36.908260 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:13:36.908223    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:13:41.365821 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:41.365782    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf70cd4-d24n5"]
Nov 03 19:13:41.366253 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:41.366227    6059 cpu_manager.go:414] "containerMap: removing container" podUID="82ed33de-befa-453e-a0c2-1672f37ef6c8" containerName="revoke-keys"
Nov 03 19:13:41.366253 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:41.366246    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ed33de-befa-453e-a0c2-1672f37ef6c8" containerName="revoke-keys"
Nov 03 19:13:41.366431 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:41.366349    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ed33de-befa-453e-a0c2-1672f37ef6c8" containerName="revoke-keys"
Nov 03 19:13:41.370251 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:41.370226    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf70cd4-d24n5"
Nov 03 19:13:41.467574 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:41.467542    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:13:41.467574 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:41.467541    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:13:41.467574 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:41.467541    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:13:41.467833 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:41.467588    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:13:41.470413 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:41.470394    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf70cd4-d24n5"]
Nov 03 19:13:41.548527 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:41.548482    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e92cb2ae-5f3e-4a57-95fd-c006406b0590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cf70cd4-d24n5\" (UID: \"e92cb2ae-5f3e-4a57-95fd-c006406b0590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf70cd4-d24n5"
Nov 03 19:13:41.548527 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:41.548535    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e92cb2ae-5f3e-4a57-95fd-c006406b0590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cf70cd4-d24n5\" (UID: \"e92cb2ae-5f3e-4a57-95fd-c006406b0590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf70cd4-d24n5"
Nov 03 19:13:41.548814 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:41.548704    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx5kv\" (UniqueName: \"kubernetes.io/projected/e92cb2ae-5f3e-4a57-95fd-c006406b0590-kube-api-access-qx5kv\") pod \"maas-api-revoke-sub-1cf70cd4-d24n5\" (UID: \"e92cb2ae-5f3e-4a57-95fd-c006406b0590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf70cd4-d24n5"
Nov 03 19:13:41.649924 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:41.649828    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e92cb2ae-5f3e-4a57-95fd-c006406b0590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cf70cd4-d24n5\" (UID: \"e92cb2ae-5f3e-4a57-95fd-c006406b0590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf70cd4-d24n5"
Nov 03 19:13:41.649924 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:41.649871    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e92cb2ae-5f3e-4a57-95fd-c006406b0590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cf70cd4-d24n5\" (UID: \"e92cb2ae-5f3e-4a57-95fd-c006406b0590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf70cd4-d24n5"
Nov 03 19:13:41.650142 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:41.649935    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx5kv\" (UniqueName: \"kubernetes.io/projected/e92cb2ae-5f3e-4a57-95fd-c006406b0590-kube-api-access-qx5kv\") pod \"maas-api-revoke-sub-1cf70cd4-d24n5\" (UID: \"e92cb2ae-5f3e-4a57-95fd-c006406b0590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf70cd4-d24n5"
Nov 03 19:13:41.650738 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:41.650715    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e92cb2ae-5f3e-4a57-95fd-c006406b0590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cf70cd4-d24n5\" (UID: \"e92cb2ae-5f3e-4a57-95fd-c006406b0590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf70cd4-d24n5"
Nov 03 19:13:41.652387 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:41.652345    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e92cb2ae-5f3e-4a57-95fd-c006406b0590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cf70cd4-d24n5\" (UID: \"e92cb2ae-5f3e-4a57-95fd-c006406b0590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf70cd4-d24n5"
Nov 03 19:13:41.708794 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:41.708761    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx5kv\" (UniqueName: \"kubernetes.io/projected/e92cb2ae-5f3e-4a57-95fd-c006406b0590-kube-api-access-qx5kv\") pod \"maas-api-revoke-sub-1cf70cd4-d24n5\" (UID: \"e92cb2ae-5f3e-4a57-95fd-c006406b0590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf70cd4-d24n5"
Nov 03 19:13:41.983328 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:41.983284    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf70cd4-d24n5"
Nov 03 19:13:42.354205 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:42.354176    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf70cd4-d24n5"]
Nov 03 19:13:42.356013 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:13:42.355979    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92cb2ae_5f3e_4a57_95fd_c006406b0590.slice/crio-3760a8ad4e4eb960d5e1400ce60b825dda98983df5bbf3ea1760697453b2aa0c WatchSource:0}: Error finding container 3760a8ad4e4eb960d5e1400ce60b825dda98983df5bbf3ea1760697453b2aa0c: Status 404 returned error can't find the container with id 3760a8ad4e4eb960d5e1400ce60b825dda98983df5bbf3ea1760697453b2aa0c
Nov 03 19:13:42.731632 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:42.731596    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf70cd4-d24n5" event={"ID":"e92cb2ae-5f3e-4a57-95fd-c006406b0590","Type":"ContainerStarted","Data":"226514355d5bae2ff6718c4f082e585d3184f40fa10a921f47ebb291d39b7047"}
Nov 03 19:13:42.732051 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:42.731659    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf70cd4-d24n5" event={"ID":"e92cb2ae-5f3e-4a57-95fd-c006406b0590","Type":"ContainerStarted","Data":"3760a8ad4e4eb960d5e1400ce60b825dda98983df5bbf3ea1760697453b2aa0c"}
Nov 03 19:13:42.844868 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:42.844810    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf70cd4-d24n5" podStartSLOduration=1.844795186 podStartE2EDuration="1.844795186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:13:42.842490532 +0000 UTC m=+6963.640203312" watchObservedRunningTime="2026-11-03 19:13:42.844795186 +0000 UTC m=+6963.642507966"
Nov 03 19:13:43.754543 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:43.754504    6059 generic.go:356] "Generic (PLEG): container finished" podID="e92cb2ae-5f3e-4a57-95fd-c006406b0590" containerID="226514355d5bae2ff6718c4f082e585d3184f40fa10a921f47ebb291d39b7047" exitCode=0
Nov 03 19:13:43.755109 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:43.754591    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf70cd4-d24n5" event={"ID":"e92cb2ae-5f3e-4a57-95fd-c006406b0590","Type":"ContainerDied","Data":"226514355d5bae2ff6718c4f082e585d3184f40fa10a921f47ebb291d39b7047"}
Nov 03 19:13:45.252381 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:45.246590    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-bwg8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:13:45.273836 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:45.273576    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-bwg8p" podUID="c17b0b8a-a3a3-4824-acc3-ce83fb644937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:13:45.321026 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:45.320570    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xkkj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:13:45.321026 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:45.320637    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xkkj4" podUID="c24c1306-3d52-4b39-af5d-5b769ec03a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:13:45.407641 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:45.397112    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-c55ww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:13:45.407641 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:45.397184    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c55ww" podUID="acc06524-297b-4b57-b0a2-4b2a92d4a00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:13:45.607943 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:45.607854    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf70cd4-d24n5"
Nov 03 19:13:45.692385 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:45.690534    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e92cb2ae-5f3e-4a57-95fd-c006406b0590-kube-api-access-qx5kv\" (UniqueName: \"kubernetes.io/projected/e92cb2ae-5f3e-4a57-95fd-c006406b0590-kube-api-access-qx5kv\") pod \"e92cb2ae-5f3e-4a57-95fd-c006406b0590\" (UID: \"e92cb2ae-5f3e-4a57-95fd-c006406b0590\") "
Nov 03 19:13:45.692385 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:45.690674    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e92cb2ae-5f3e-4a57-95fd-c006406b0590-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e92cb2ae-5f3e-4a57-95fd-c006406b0590-maas-api-openshift-ca\") pod \"e92cb2ae-5f3e-4a57-95fd-c006406b0590\" (UID: \"e92cb2ae-5f3e-4a57-95fd-c006406b0590\") "
Nov 03 19:13:45.692385 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:45.690769    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e92cb2ae-5f3e-4a57-95fd-c006406b0590-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e92cb2ae-5f3e-4a57-95fd-c006406b0590-maas-api-xks-ca\") pod \"e92cb2ae-5f3e-4a57-95fd-c006406b0590\" (UID: \"e92cb2ae-5f3e-4a57-95fd-c006406b0590\") "
Nov 03 19:13:45.692385 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:45.692067    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92cb2ae-5f3e-4a57-95fd-c006406b0590-maas-api-openshift-ca" pod "e92cb2ae-5f3e-4a57-95fd-c006406b0590" (UID: "e92cb2ae-5f3e-4a57-95fd-c006406b0590"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:13:45.724075 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:45.724038    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92cb2ae-5f3e-4a57-95fd-c006406b0590-maas-api-xks-ca" pod "e92cb2ae-5f3e-4a57-95fd-c006406b0590" (UID: "e92cb2ae-5f3e-4a57-95fd-c006406b0590"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:13:45.734588 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:45.734549    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92cb2ae-5f3e-4a57-95fd-c006406b0590-kube-api-access-qx5kv" pod "e92cb2ae-5f3e-4a57-95fd-c006406b0590" (UID: "e92cb2ae-5f3e-4a57-95fd-c006406b0590"). InnerVolumeSpecName "kube-api-access-qx5kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:13:45.809384 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:45.801641    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx5kv\" (UniqueName: \"kubernetes.io/projected/e92cb2ae-5f3e-4a57-95fd-c006406b0590-kube-api-access-qx5kv\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:45.809651 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:45.809627    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e92cb2ae-5f3e-4a57-95fd-c006406b0590-maas-api-openshift-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:45.809793 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:45.809777    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e92cb2ae-5f3e-4a57-95fd-c006406b0590-maas-api-xks-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:46.028668 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:46.028626    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf70cd4-d24n5" event={"ID":"e92cb2ae-5f3e-4a57-95fd-c006406b0590","Type":"ContainerDied","Data":"3760a8ad4e4eb960d5e1400ce60b825dda98983df5bbf3ea1760697453b2aa0c"}
Nov 03 19:13:46.028971 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:46.028946    6059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3760a8ad4e4eb960d5e1400ce60b825dda98983df5bbf3ea1760697453b2aa0c"
Nov 03 19:13:46.029161 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:46.029147    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf70cd4-d24n5"
Nov 03 19:13:46.260380 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:46.259471    6059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-csr7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:13:46.260380 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:13:46.259563    6059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csr7q" podUID="3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:14:02.045669 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:02.045629    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9925ccc3-htbcg"]
Nov 03 19:14:02.093048 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:02.093010    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9925ccc3-htbcg"]
Nov 03 19:14:03.334596 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:03.334562    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9249e174-6085-4a7d-84ae-fd8ef208ced0" path="/var/lib/kubelet/pods/9249e174-6085-4a7d-84ae-fd8ef208ced0/volumes"
Nov 03 19:14:06.860108 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:14:06.860066    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:14:10.699767 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:10.699737    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:10.887575 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:10.887541    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:25.596108 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:25.596073    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5fdd44f589-9c6qf_e9f4c7ae-adf2-4c9a-a9be-a8b5829b69d2/manager/0.log"
Nov 03 19:14:27.946615 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:27.946578    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75c4dd589d-nztw8_6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a/manager/0.log"
Nov 03 19:14:31.472230 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:31.472201    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-105678f7-hgpwk_f5e4548a-9fa7-4846-9933-02ce4c6901da/revoke-keys/0.log"
Nov 03 19:14:31.717335 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:31.717297    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-111e3b0e-wg5qc_f4c396ce-f5d5-47d0-91ec-b64f16749ab6/revoke-keys/0.log"
Nov 03 19:14:31.932967 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:31.932895    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1317113d-cddl4_ddd4f6b4-1539-423f-914a-5eac9068fd48/revoke-keys/0.log"
Nov 03 19:14:32.154101 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:32.154059    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-19c0efa0-pxxrc_cb41caea-40c1-41c3-b9d8-2b1843dfcb45/revoke-keys/0.log"
Nov 03 19:14:32.384870 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:32.384838    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1cf70cd4-d24n5_e92cb2ae-5f3e-4a57-95fd-c006406b0590/revoke-keys/0.log"
Nov 03 19:14:32.624246 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:32.624216    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-31e680a3-m4r6j_f6b058ce-0bd8-48fa-baaa-7541c74a63ea/revoke-keys/0.log"
Nov 03 19:14:32.867087 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:32.867056    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-388b812d-llqp9_4f2c5146-978a-4165-b722-6438daa73f55/revoke-keys/0.log"
Nov 03 19:14:33.111560 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:33.111531    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3d67d5fb-n8kgx_964676bd-8a6d-42a0-96bb-52b90bfa97bb/revoke-keys/0.log"
Nov 03 19:14:33.356255 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:33.356225    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-42918c8e-mttv9_2a5a1dea-0df1-4268-b9d0-360fdcd887de/revoke-keys/0.log"
Nov 03 19:14:33.607924 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:33.607845    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-47cbfe5e-wxm5n_2de70020-6b22-440e-93ca-113796ad4dbf/revoke-keys/0.log"
Nov 03 19:14:33.849595 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:33.849566    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7caf9856-nrjqw_a867663e-30b8-4fab-ae7a-e5e54b05208b/revoke-keys/0.log"
Nov 03 19:14:34.108475 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:34.108443    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c5d3f5d6-pxblx_0ea8ccde-e7a7-4d3c-acf1-d35e42abaa50/revoke-keys/0.log"
Nov 03 19:14:34.356370 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:34.356339    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d288ac11-2n9vb_fbb20844-7ae2-4d22-a7cf-cd6d74d72e56/revoke-keys/0.log"
Nov 03 19:14:34.584437 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:34.584407    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d9a946ce-46s5c_82ed33de-befa-453e-a0c2-1672f37ef6c8/revoke-keys/0.log"
Nov 03 19:14:34.816620 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:34.816589    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e7c483d2-6dtt7_4c39758e-e9fc-4cbd-92c0-c3550cf8510b/revoke-keys/0.log"
Nov 03 19:14:35.044443 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:35.044410    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f4a139f5-d99b2_2b86f325-e858-499c-8ecb-0fa987fa5a88/revoke-keys/0.log"
Nov 03 19:14:36.861765 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:14:36.861725    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:14:38.277259 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:38.277229    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-n4k4d_d8bf1633-07f3-4660-9268-cd96bfee740a/kuadrant-console-plugin/0.log"
Nov 03 19:14:41.996890 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:41.996860    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp_96f2949d-40c0-46ec-9cb6-235e005b0736/istio-proxy/0.log"
Nov 03 19:14:42.847027 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:42.846994    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fc78d544-x9ljt_079406c1-7060-4a19-b62f-0f8c25e4aea7/kube-auth-proxy/0.log"
Nov 03 19:14:43.663796 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:43.663756    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-bwg8p_c17b0b8a-a3a3-4824-acc3-ce83fb644937/istio-proxy/0.log"
Nov 03 19:14:43.889500 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:43.889475    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-c55ww_acc06524-297b-4b57-b0a2-4b2a92d4a00c/istio-proxy/0.log"
Nov 03 19:14:44.125520 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:44.125490    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-csr7q_3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3/istio-proxy/0.log"
Nov 03 19:14:44.739003 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:44.738971    6059 scope.go:122] "RemoveContainer" containerID="eb554b31a669bad9dde860368f3da87f2c48576e3b176b956a28d19d906276e0"
Nov 03 19:14:44.748742 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:44.748720    6059 scope.go:122] "RemoveContainer" containerID="df7c48786840ada42191193b6e1d166a61485f6455a8de1ecd23899d11d9916c"
Nov 03 19:14:45.476200 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:45.476170    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-xkkj4_c24c1306-3d52-4b39-af5d-5b769ec03a1a/istio-proxy/0.log"
Nov 03 19:14:45.714187 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:45.714158    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w_58741805-9507-4add-ab80-651305453ee6/istio-proxy/0.log"
Nov 03 19:14:45.936978 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:45.936901    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7987dbcbcc-d2pl9_e7bdff75-bc2d-4591-993b-97eba56293ac/payload-pre-processing/0.log"
Nov 03 19:14:46.152454 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:46.152429    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7b88dbb578-2tqps_42c95c41-1e40-4f67-8f76-6291a182f48c/payload-processing/0.log"
Nov 03 19:14:48.049808 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:48.049771    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9"]
Nov 03 19:14:48.097188 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:48.097154    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388b812d-llqp9"]
Nov 03 19:14:48.342746 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:48.342657    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-zkrg5_e687910a-3808-430e-a834-08c248fe5855/main/0.log"
Nov 03 19:14:48.567981 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:48.567950    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8psl6k_3e9c7d34-2338-4cc0-81c6-7e70e16c8cf2/main/0.log"
Nov 03 19:14:48.798173 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:48.798145    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-4t6c8_078a0cf0-420d-4a24-a609-55e00cc4bc09/main/0.log"
Nov 03 19:14:49.249623 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:49.249592    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-qfxzl_c194a4be-0499-419f-a4a1-910dbaafd303/main/0.log"
Nov 03 19:14:49.335100 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:49.335066    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f2c5146-978a-4165-b722-6438daa73f55" path="/var/lib/kubelet/pods/4f2c5146-978a-4165-b722-6438daa73f55/volumes"
Nov 03 19:14:59.046037 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:59.046004    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx"]
Nov 03 19:14:59.093216 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:59.093185    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d67d5fb-n8kgx"]
Nov 03 19:14:59.335529 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:14:59.335453    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="964676bd-8a6d-42a0-96bb-52b90bfa97bb" path="/var/lib/kubelet/pods/964676bd-8a6d-42a0-96bb-52b90bfa97bb/volumes"
Nov 03 19:15:00.168448 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.168413    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78"]
Nov 03 19:15:00.168854 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.168841    6059 cpu_manager.go:414] "containerMap: removing container" podUID="e92cb2ae-5f3e-4a57-95fd-c006406b0590" containerName="revoke-keys"
Nov 03 19:15:00.168908 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.168854    6059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e92cb2ae-5f3e-4a57-95fd-c006406b0590" containerName="revoke-keys"
Nov 03 19:15:00.168946 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.168936    6059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e92cb2ae-5f3e-4a57-95fd-c006406b0590" containerName="revoke-keys"
Nov 03 19:15:00.173858 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.173835    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78"
Nov 03 19:15:00.184563 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.184538    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvbcp\" (UniqueName: \"kubernetes.io/projected/4a1f4f10-1e39-4fcd-8197-6cca964b9e1e-kube-api-access-jvbcp\") pod \"maas-api-key-cleanup-29895555-5xz78\" (UID: \"4a1f4f10-1e39-4fcd-8197-6cca964b9e1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78"
Nov 03 19:15:00.184706 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.184684    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1f4f10-1e39-4fcd-8197-6cca964b9e1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895555-5xz78\" (UID: \"4a1f4f10-1e39-4fcd-8197-6cca964b9e1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78"
Nov 03 19:15:00.227350 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.227322    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78"]
Nov 03 19:15:00.271241 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.271204    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:15:00.271241 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.271235    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:15:00.271481 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.271309    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\""
Nov 03 19:15:00.273912 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.273892    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rdwfb/must-gather-m9q67"]
Nov 03 19:15:00.277961 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.277943    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rdwfb/must-gather-m9q67"
Nov 03 19:15:00.280107 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.280082    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:00.285399 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.285331    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvbcp\" (UniqueName: \"kubernetes.io/projected/4a1f4f10-1e39-4fcd-8197-6cca964b9e1e-kube-api-access-jvbcp\") pod \"maas-api-key-cleanup-29895555-5xz78\" (UID: \"4a1f4f10-1e39-4fcd-8197-6cca964b9e1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78"
Nov 03 19:15:00.285800 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.285510    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e07bdc73-ae6c-4903-9d36-507590a9fd79-must-gather-output\") pod \"must-gather-m9q67\" (UID: \"e07bdc73-ae6c-4903-9d36-507590a9fd79\") " pod="openshift-must-gather-rdwfb/must-gather-m9q67"
Nov 03 19:15:00.285800 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.285563    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1f4f10-1e39-4fcd-8197-6cca964b9e1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895555-5xz78\" (UID: \"4a1f4f10-1e39-4fcd-8197-6cca964b9e1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78"
Nov 03 19:15:00.285800 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.285591    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gk8z\" (UniqueName: \"kubernetes.io/projected/e07bdc73-ae6c-4903-9d36-507590a9fd79-kube-api-access-8gk8z\") pod \"must-gather-m9q67\" (UID: \"e07bdc73-ae6c-4903-9d36-507590a9fd79\") " pod="openshift-must-gather-rdwfb/must-gather-m9q67"
Nov 03 19:15:00.286382 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.286345    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1f4f10-1e39-4fcd-8197-6cca964b9e1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895555-5xz78\" (UID: \"4a1f4f10-1e39-4fcd-8197-6cca964b9e1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78"
Nov 03 19:15:00.324906 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.324878    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rdwfb/must-gather-m9q67"]
Nov 03 19:15:00.327500 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.327485    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rdwfb\"/\"default-dockercfg-7zd6s\""
Nov 03 19:15:00.327822 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.327802    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rdwfb\"/\"openshift-service-ca.crt\""
Nov 03 19:15:00.327950 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.327910    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rdwfb\"/\"kube-root-ca.crt\""
Nov 03 19:15:00.347871 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.347846    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvbcp\" (UniqueName: \"kubernetes.io/projected/4a1f4f10-1e39-4fcd-8197-6cca964b9e1e-kube-api-access-jvbcp\") pod \"maas-api-key-cleanup-29895555-5xz78\" (UID: \"4a1f4f10-1e39-4fcd-8197-6cca964b9e1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78"
Nov 03 19:15:00.386750 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.386717    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gk8z\" (UniqueName: \"kubernetes.io/projected/e07bdc73-ae6c-4903-9d36-507590a9fd79-kube-api-access-8gk8z\") pod \"must-gather-m9q67\" (UID: \"e07bdc73-ae6c-4903-9d36-507590a9fd79\") " pod="openshift-must-gather-rdwfb/must-gather-m9q67"
Nov 03 19:15:00.386918 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.386842    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e07bdc73-ae6c-4903-9d36-507590a9fd79-must-gather-output\") pod \"must-gather-m9q67\" (UID: \"e07bdc73-ae6c-4903-9d36-507590a9fd79\") " pod="openshift-must-gather-rdwfb/must-gather-m9q67"
Nov 03 19:15:00.387160 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.387143    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e07bdc73-ae6c-4903-9d36-507590a9fd79-must-gather-output\") pod \"must-gather-m9q67\" (UID: \"e07bdc73-ae6c-4903-9d36-507590a9fd79\") " pod="openshift-must-gather-rdwfb/must-gather-m9q67"
Nov 03 19:15:00.445049 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.444983    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gk8z\" (UniqueName: \"kubernetes.io/projected/e07bdc73-ae6c-4903-9d36-507590a9fd79-kube-api-access-8gk8z\") pod \"must-gather-m9q67\" (UID: \"e07bdc73-ae6c-4903-9d36-507590a9fd79\") " pod="openshift-must-gather-rdwfb/must-gather-m9q67"
Nov 03 19:15:00.467847 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.467816    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:00.487428 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.487402    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78"
Nov 03 19:15:00.593850 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.593817    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rdwfb/must-gather-m9q67"
Nov 03 19:15:00.855618 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.855591    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78"]
Nov 03 19:15:00.856891 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:15:00.856857    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1f4f10_1e39_4fcd_8197_6cca964b9e1e.slice/crio-c5bcc2b70d6f5c08981fa64bac2f0d623f5205e57a95e79aef67f8eed91e8d11 WatchSource:0}: Error finding container c5bcc2b70d6f5c08981fa64bac2f0d623f5205e57a95e79aef67f8eed91e8d11: Status 404 returned error can't find the container with id c5bcc2b70d6f5c08981fa64bac2f0d623f5205e57a95e79aef67f8eed91e8d11
Nov 03 19:15:00.996278 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.996250    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rdwfb/must-gather-m9q67"]
Nov 03 19:15:00.997350 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:15:00.997317    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode07bdc73_ae6c_4903_9d36_507590a9fd79.slice/crio-03f4cca3c1303c3536e9d5a7906a4ac5da2c54f3378fa9e8ab2e7b799bfcbb2f WatchSource:0}: Error finding container 03f4cca3c1303c3536e9d5a7906a4ac5da2c54f3378fa9e8ab2e7b799bfcbb2f: Status 404 returned error can't find the container with id 03f4cca3c1303c3536e9d5a7906a4ac5da2c54f3378fa9e8ab2e7b799bfcbb2f
Nov 03 19:15:00.999557 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:00.999538    6059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:15:01.534429 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:01.534390    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78" event={"ID":"4a1f4f10-1e39-4fcd-8197-6cca964b9e1e","Type":"ContainerStarted","Data":"0601d5bccac8e5351515516db97f69440fffdb7c44b684d43bfd261fce307a0d"}
Nov 03 19:15:01.535006 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:01.534454    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78" event={"ID":"4a1f4f10-1e39-4fcd-8197-6cca964b9e1e","Type":"ContainerStarted","Data":"c5bcc2b70d6f5c08981fa64bac2f0d623f5205e57a95e79aef67f8eed91e8d11"}
Nov 03 19:15:01.535731 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:01.535695    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rdwfb/must-gather-m9q67" event={"ID":"e07bdc73-ae6c-4903-9d36-507590a9fd79","Type":"ContainerStarted","Data":"03f4cca3c1303c3536e9d5a7906a4ac5da2c54f3378fa9e8ab2e7b799bfcbb2f"}
Nov 03 19:15:01.647444 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:01.647390    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78" podStartSLOduration=1.6473745690000001 podStartE2EDuration="1.647374569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:01.64406128 +0000 UTC m=+7042.441774063" watchObservedRunningTime="2026-11-03 19:15:01.647374569 +0000 UTC m=+7042.445087347"
Nov 03 19:15:02.543926 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:02.543893    6059 generic.go:356] "Generic (PLEG): container finished" podID="4a1f4f10-1e39-4fcd-8197-6cca964b9e1e" containerID="0601d5bccac8e5351515516db97f69440fffdb7c44b684d43bfd261fce307a0d" exitCode=60
Nov 03 19:15:02.544345 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:02.543968    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78" event={"ID":"4a1f4f10-1e39-4fcd-8197-6cca964b9e1e","Type":"ContainerDied","Data":"0601d5bccac8e5351515516db97f69440fffdb7c44b684d43bfd261fce307a0d"}
Nov 03 19:15:02.544345 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:02.544212    6059 scope.go:122] "RemoveContainer" containerID="0601d5bccac8e5351515516db97f69440fffdb7c44b684d43bfd261fce307a0d"
Nov 03 19:15:03.551344 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:03.551311    6059 generic.go:356] "Generic (PLEG): container finished" podID="4a1f4f10-1e39-4fcd-8197-6cca964b9e1e" containerID="66f76156d5d05874216e70991dc5a260a7295e6c5a9828f11d4c15db540f730f" exitCode=60
Nov 03 19:15:03.551841 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:03.551405    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78" event={"ID":"4a1f4f10-1e39-4fcd-8197-6cca964b9e1e","Type":"ContainerDied","Data":"66f76156d5d05874216e70991dc5a260a7295e6c5a9828f11d4c15db540f730f"}
Nov 03 19:15:03.551841 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:03.551518    6059 scope.go:122] "RemoveContainer" containerID="0601d5bccac8e5351515516db97f69440fffdb7c44b684d43bfd261fce307a0d"
Nov 03 19:15:03.551841 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:03.551797    6059 scope.go:122] "RemoveContainer" containerID="66f76156d5d05874216e70991dc5a260a7295e6c5a9828f11d4c15db540f730f"
Nov 03 19:15:03.552065 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:15:03.552045    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895555-5xz78_odh-ai-gateway-infra(4a1f4f10-1e39-4fcd-8197-6cca964b9e1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78" podUID="4a1f4f10-1e39-4fcd-8197-6cca964b9e1e"
Nov 03 19:15:04.559932 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:04.559907    6059 scope.go:122] "RemoveContainer" containerID="66f76156d5d05874216e70991dc5a260a7295e6c5a9828f11d4c15db540f730f"
Nov 03 19:15:04.560428 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:15:04.560146    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895555-5xz78_odh-ai-gateway-infra(4a1f4f10-1e39-4fcd-8197-6cca964b9e1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78" podUID="4a1f4f10-1e39-4fcd-8197-6cca964b9e1e"
Nov 03 19:15:06.860715 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:15:06.860669    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:15:10.400314 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:10.400279    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rt4lh_902e0389-e70e-4d23-a607-d8f4b9da868d/global-pull-secret-syncer/0.log"
Nov 03 19:15:11.122839 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:11.122798    6059 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:13.002299 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:13.002263    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dzlp4_5923113e-7941-4bf8-bfde-28622a520e86/konnectivity-agent/0.log"
Nov 03 19:15:14.625866 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:14.625765    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rdwfb/must-gather-m9q67" event={"ID":"e07bdc73-ae6c-4903-9d36-507590a9fd79","Type":"ContainerStarted","Data":"477f9c23b58fe44664dfd284b0a69f93ab4a4f8d4f7e1a8e51ebed256b94490b"}
Nov 03 19:15:14.625866 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:14.625814    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rdwfb/must-gather-m9q67" event={"ID":"e07bdc73-ae6c-4903-9d36-507590a9fd79","Type":"ContainerStarted","Data":"90a2908b8b5e5355562a9e0cf324e6c562fd9f39df709b5674142186f48aa93b"}
Nov 03 19:15:14.737073 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:14.737019    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rdwfb/must-gather-m9q67" podStartSLOduration=1.473494334 podStartE2EDuration="14.737003701s" totalImagesPullingTime="13.263509367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:14.735644594 +0000 UTC m=+7055.533357375" watchObservedRunningTime="2026-11-03 19:15:14.737003701 +0000 UTC m=+7055.534716525"
Nov 03 19:15:15.528304 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:15.528263    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-117.us-west-2.compute.internal_01359929955a1f3cff637d7cc239a2bc/haproxy/0.log"
Nov 03 19:15:18.328778 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:18.328744    6059 scope.go:122] "RemoveContainer" containerID="66f76156d5d05874216e70991dc5a260a7295e6c5a9828f11d4c15db540f730f"
Nov 03 19:15:18.494469 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:18.494440    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5fdd44f589-9c6qf_e9f4c7ae-adf2-4c9a-a9be-a8b5829b69d2/manager/0.log"
Nov 03 19:15:18.653021 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:18.652991    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78" event={"ID":"4a1f4f10-1e39-4fcd-8197-6cca964b9e1e","Type":"ContainerStarted","Data":"d20eb35b72dc93d0e84bfc2dba2953a028014d658167f2da9c722b8e1041cc79"}
Nov 03 19:15:19.661991 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:19.661947    6059 generic.go:356] "Generic (PLEG): container finished" podID="4a1f4f10-1e39-4fcd-8197-6cca964b9e1e" containerID="d20eb35b72dc93d0e84bfc2dba2953a028014d658167f2da9c722b8e1041cc79" exitCode=60
Nov 03 19:15:19.662612 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:19.662027    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78" event={"ID":"4a1f4f10-1e39-4fcd-8197-6cca964b9e1e","Type":"ContainerDied","Data":"d20eb35b72dc93d0e84bfc2dba2953a028014d658167f2da9c722b8e1041cc79"}
Nov 03 19:15:19.662612 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:19.662102    6059 scope.go:122] "RemoveContainer" containerID="66f76156d5d05874216e70991dc5a260a7295e6c5a9828f11d4c15db540f730f"
Nov 03 19:15:19.662612 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:19.662480    6059 scope.go:122] "RemoveContainer" containerID="d20eb35b72dc93d0e84bfc2dba2953a028014d658167f2da9c722b8e1041cc79"
Nov 03 19:15:19.662825 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:15:19.662802    6059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895555-5xz78_odh-ai-gateway-infra(4a1f4f10-1e39-4fcd-8197-6cca964b9e1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78" podUID="4a1f4f10-1e39-4fcd-8197-6cca964b9e1e"
Nov 03 19:15:19.775988 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:19.775951    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78"]
Nov 03 19:15:20.238957 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.238922    6059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph"]
Nov 03 19:15:20.244043 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.244022    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph"
Nov 03 19:15:20.294338 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.294308    6059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-45cfb\"/\"default-dockercfg-chcxq\""
Nov 03 19:15:20.294518 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.294411    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-45cfb\"/\"openshift-service-ca.crt\""
Nov 03 19:15:20.294518 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.294423    6059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-45cfb\"/\"kube-root-ca.crt\""
Nov 03 19:15:20.297578 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.297551    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph"]
Nov 03 19:15:20.406027 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.405988    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e61307c6-78a3-4a4c-be8b-dfc2506b1ad5-proc\") pod \"perf-node-gather-daemonset-rxcph\" (UID: \"e61307c6-78a3-4a4c-be8b-dfc2506b1ad5\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph"
Nov 03 19:15:20.406210 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.406046    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e61307c6-78a3-4a4c-be8b-dfc2506b1ad5-lib-modules\") pod \"perf-node-gather-daemonset-rxcph\" (UID: \"e61307c6-78a3-4a4c-be8b-dfc2506b1ad5\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph"
Nov 03 19:15:20.406210 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.406147    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e61307c6-78a3-4a4c-be8b-dfc2506b1ad5-podres\") pod \"perf-node-gather-daemonset-rxcph\" (UID: \"e61307c6-78a3-4a4c-be8b-dfc2506b1ad5\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph"
Nov 03 19:15:20.406210 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.406199    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e61307c6-78a3-4a4c-be8b-dfc2506b1ad5-sys\") pod \"perf-node-gather-daemonset-rxcph\" (UID: \"e61307c6-78a3-4a4c-be8b-dfc2506b1ad5\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph"
Nov 03 19:15:20.406330 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.406286    6059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t25vr\" (UniqueName: \"kubernetes.io/projected/e61307c6-78a3-4a4c-be8b-dfc2506b1ad5-kube-api-access-t25vr\") pod \"perf-node-gather-daemonset-rxcph\" (UID: \"e61307c6-78a3-4a4c-be8b-dfc2506b1ad5\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph"
Nov 03 19:15:20.507260 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.507167    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e61307c6-78a3-4a4c-be8b-dfc2506b1ad5-proc\") pod \"perf-node-gather-daemonset-rxcph\" (UID: \"e61307c6-78a3-4a4c-be8b-dfc2506b1ad5\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph"
Nov 03 19:15:20.507260 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.507241    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e61307c6-78a3-4a4c-be8b-dfc2506b1ad5-lib-modules\") pod \"perf-node-gather-daemonset-rxcph\" (UID: \"e61307c6-78a3-4a4c-be8b-dfc2506b1ad5\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph"
Nov 03 19:15:20.507260 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.507253    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e61307c6-78a3-4a4c-be8b-dfc2506b1ad5-proc\") pod \"perf-node-gather-daemonset-rxcph\" (UID: \"e61307c6-78a3-4a4c-be8b-dfc2506b1ad5\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph"
Nov 03 19:15:20.507614 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.507313    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e61307c6-78a3-4a4c-be8b-dfc2506b1ad5-podres\") pod \"perf-node-gather-daemonset-rxcph\" (UID: \"e61307c6-78a3-4a4c-be8b-dfc2506b1ad5\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph"
Nov 03 19:15:20.507614 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.507341    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e61307c6-78a3-4a4c-be8b-dfc2506b1ad5-lib-modules\") pod \"perf-node-gather-daemonset-rxcph\" (UID: \"e61307c6-78a3-4a4c-be8b-dfc2506b1ad5\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph"
Nov 03 19:15:20.507614 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.507417    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e61307c6-78a3-4a4c-be8b-dfc2506b1ad5-podres\") pod \"perf-node-gather-daemonset-rxcph\" (UID: \"e61307c6-78a3-4a4c-be8b-dfc2506b1ad5\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph"
Nov 03 19:15:20.507614 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.507418    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e61307c6-78a3-4a4c-be8b-dfc2506b1ad5-sys\") pod \"perf-node-gather-daemonset-rxcph\" (UID: \"e61307c6-78a3-4a4c-be8b-dfc2506b1ad5\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph"
Nov 03 19:15:20.507614 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.507459    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e61307c6-78a3-4a4c-be8b-dfc2506b1ad5-sys\") pod \"perf-node-gather-daemonset-rxcph\" (UID: \"e61307c6-78a3-4a4c-be8b-dfc2506b1ad5\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph"
Nov 03 19:15:20.507614 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.507471    6059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t25vr\" (UniqueName: \"kubernetes.io/projected/e61307c6-78a3-4a4c-be8b-dfc2506b1ad5-kube-api-access-t25vr\") pod \"perf-node-gather-daemonset-rxcph\" (UID: \"e61307c6-78a3-4a4c-be8b-dfc2506b1ad5\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph"
Nov 03 19:15:20.567735 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.567694    6059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t25vr\" (UniqueName: \"kubernetes.io/projected/e61307c6-78a3-4a4c-be8b-dfc2506b1ad5-kube-api-access-t25vr\") pod \"perf-node-gather-daemonset-rxcph\" (UID: \"e61307c6-78a3-4a4c-be8b-dfc2506b1ad5\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph"
Nov 03 19:15:20.596438 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.596402    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75c4dd589d-nztw8_6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a/manager/0.log"
Nov 03 19:15:20.806405 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.806380    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78"
Nov 03 19:15:20.856804 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.856768    6059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph"
Nov 03 19:15:20.912445 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.912418    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a1f4f10-1e39-4fcd-8197-6cca964b9e1e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1f4f10-1e39-4fcd-8197-6cca964b9e1e-maas-api-service-ca\") pod \"4a1f4f10-1e39-4fcd-8197-6cca964b9e1e\" (UID: \"4a1f4f10-1e39-4fcd-8197-6cca964b9e1e\") "
Nov 03 19:15:20.912582 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.912483    6059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a1f4f10-1e39-4fcd-8197-6cca964b9e1e-kube-api-access-jvbcp\" (UniqueName: \"kubernetes.io/projected/4a1f4f10-1e39-4fcd-8197-6cca964b9e1e-kube-api-access-jvbcp\") pod \"4a1f4f10-1e39-4fcd-8197-6cca964b9e1e\" (UID: \"4a1f4f10-1e39-4fcd-8197-6cca964b9e1e\") "
Nov 03 19:15:20.913596 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.913572    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a1f4f10-1e39-4fcd-8197-6cca964b9e1e-maas-api-service-ca" pod "4a1f4f10-1e39-4fcd-8197-6cca964b9e1e" (UID: "4a1f4f10-1e39-4fcd-8197-6cca964b9e1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:15:20.915445 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:20.915402    6059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a1f4f10-1e39-4fcd-8197-6cca964b9e1e-kube-api-access-jvbcp" pod "4a1f4f10-1e39-4fcd-8197-6cca964b9e1e" (UID: "4a1f4f10-1e39-4fcd-8197-6cca964b9e1e"). InnerVolumeSpecName "kube-api-access-jvbcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:21.013587 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:21.013550    6059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvbcp\" (UniqueName: \"kubernetes.io/projected/4a1f4f10-1e39-4fcd-8197-6cca964b9e1e-kube-api-access-jvbcp\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:21.013587 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:21.013584    6059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1f4f10-1e39-4fcd-8197-6cca964b9e1e-maas-api-service-ca\") on node \"ip-10-0-143-117.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:21.163057 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:21.162974    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-n4k4d_d8bf1633-07f3-4660-9268-cd96bfee740a/kuadrant-console-plugin/0.log"
Nov 03 19:15:21.227558 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:21.227535    6059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph"]
Nov 03 19:15:21.229210 ip-10-0-143-117 kubenswrapper[6059]: W1103 19:15:21.229178    6059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode61307c6_78a3_4a4c_be8b_dfc2506b1ad5.slice/crio-b23a05fc751a5aa46bc8237189576738e1c395ac78bd46128cdaff3140577f3f WatchSource:0}: Error finding container b23a05fc751a5aa46bc8237189576738e1c395ac78bd46128cdaff3140577f3f: Status 404 returned error can't find the container with id b23a05fc751a5aa46bc8237189576738e1c395ac78bd46128cdaff3140577f3f
Nov 03 19:15:21.681758 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:21.681724    6059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78"
Nov 03 19:15:21.681967 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:21.681765    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78" event={"ID":"4a1f4f10-1e39-4fcd-8197-6cca964b9e1e","Type":"ContainerDied","Data":"c5bcc2b70d6f5c08981fa64bac2f0d623f5205e57a95e79aef67f8eed91e8d11"}
Nov 03 19:15:21.681967 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:21.681833    6059 scope.go:122] "RemoveContainer" containerID="d20eb35b72dc93d0e84bfc2dba2953a028014d658167f2da9c722b8e1041cc79"
Nov 03 19:15:21.684716 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:21.684166    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph" event={"ID":"e61307c6-78a3-4a4c-be8b-dfc2506b1ad5","Type":"ContainerStarted","Data":"1ae994da478809c94fdf72462e9e8748aad933a34b3da1ed6fdf1d9f657b1313"}
Nov 03 19:15:21.684716 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:21.684219    6059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph" event={"ID":"e61307c6-78a3-4a4c-be8b-dfc2506b1ad5","Type":"ContainerStarted","Data":"b23a05fc751a5aa46bc8237189576738e1c395ac78bd46128cdaff3140577f3f"}
Nov 03 19:15:21.684971 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:21.684949    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph"
Nov 03 19:15:22.000275 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:22.000214    6059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph" podStartSLOduration=2.000195103 podStartE2EDuration="2.000195103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:21.893716937 +0000 UTC m=+7062.691429714" watchObservedRunningTime="2026-11-03 19:15:22.000195103 +0000 UTC m=+7062.797907944"
Nov 03 19:15:22.002863 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:22.002835    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78"]
Nov 03 19:15:22.005599 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:22.005575    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895555-5xz78"]
Nov 03 19:15:23.335519 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:23.335485    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a1f4f10-1e39-4fcd-8197-6cca964b9e1e" path="/var/lib/kubelet/pods/4a1f4f10-1e39-4fcd-8197-6cca964b9e1e/volumes"
Nov 03 19:15:25.049941 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:25.049912    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5fdd44f589-9c6qf_e9f4c7ae-adf2-4c9a-a9be-a8b5829b69d2/manager/0.log"
Nov 03 19:15:26.524709 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:26.524675    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75c4dd589d-nztw8_6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a/manager/0.log"
Nov 03 19:15:28.710713 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:28.710684    6059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-rxcph"
Nov 03 19:15:28.943108 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:28.943075    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-c9nbc_08770b22-f9ff-4919-af54-c7916eba9101/kube-state-metrics/0.log"
Nov 03 19:15:29.348022 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:29.347995    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-c9nbc_08770b22-f9ff-4919-af54-c7916eba9101/kube-rbac-proxy-main/0.log"
Nov 03 19:15:29.766754 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:29.766710    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-c9nbc_08770b22-f9ff-4919-af54-c7916eba9101/kube-rbac-proxy-self/0.log"
Nov 03 19:15:30.184095 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:30.184015    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-865c7f6b8c-gq7x9_4f2d426d-7ee0-48d9-b960-04f7fca5dc29/metrics-server/0.log"
Nov 03 19:15:30.595334 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:30.595306    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-67sr8_504075e3-50b2-4616-afea-70d49c4d7293/monitoring-plugin/0.log"
Nov 03 19:15:31.043536 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:31.043506    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5fdd44f589-9c6qf_e9f4c7ae-adf2-4c9a-a9be-a8b5829b69d2/manager/0.log"
Nov 03 19:15:32.510160 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:32.510124    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9gj97_4af20ec7-7ea1-470f-b238-62e0de9ca527/node-exporter/0.log"
Nov 03 19:15:32.538465 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:32.538437    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75c4dd589d-nztw8_6bc700c5-cf97-4b4a-a43e-0dc69cf5b21a/manager/0.log"
Nov 03 19:15:32.918441 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:32.918369    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9gj97_4af20ec7-7ea1-470f-b238-62e0de9ca527/kube-rbac-proxy/0.log"
Nov 03 19:15:33.329524 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:33.329496    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9gj97_4af20ec7-7ea1-470f-b238-62e0de9ca527/init-textfile/0.log"
Nov 03 19:15:34.993244 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:34.993209    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-5srqg_117a0c38-48f5-430d-9efe-1a308f725309/kube-rbac-proxy-main/0.log"
Nov 03 19:15:35.403404 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:35.403242    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-5srqg_117a0c38-48f5-430d-9efe-1a308f725309/kube-rbac-proxy-self/0.log"
Nov 03 19:15:35.813808 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:35.813776    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-5srqg_117a0c38-48f5-430d-9efe-1a308f725309/openshift-state-metrics/0.log"
Nov 03 19:15:36.239061 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:36.239033    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e2f223b3-bcb1-4d64-a561-96c3b1f10127/prometheus/0.log"
Nov 03 19:15:36.648218 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:36.648138    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e2f223b3-bcb1-4d64-a561-96c3b1f10127/config-reloader/0.log"
Nov 03 19:15:36.864226 ip-10-0-143-117 kubenswrapper[6059]: E1103 19:15:36.864186    6059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5gv26" containerName="extractor"
Nov 03 19:15:37.061670 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:37.061628    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e2f223b3-bcb1-4d64-a561-96c3b1f10127/thanos-sidecar/0.log"
Nov 03 19:15:37.470649 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:37.470611    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e2f223b3-bcb1-4d64-a561-96c3b1f10127/kube-rbac-proxy-web/0.log"
Nov 03 19:15:37.545328 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:37.545289    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp_96f2949d-40c0-46ec-9cb6-235e005b0736/istio-proxy/0.log"
Nov 03 19:15:37.884137 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:37.884055    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e2f223b3-bcb1-4d64-a561-96c3b1f10127/kube-rbac-proxy/0.log"
Nov 03 19:15:38.296049 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:38.296009    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e2f223b3-bcb1-4d64-a561-96c3b1f10127/kube-rbac-proxy-thanos/0.log"
Nov 03 19:15:38.706677 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:38.706644    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e2f223b3-bcb1-4d64-a561-96c3b1f10127/init-config-reloader/0.log"
Nov 03 19:15:40.354390 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:40.354339    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-548c4c64fb-pgglg_76f649c8-7dad-410e-a2e8-960e639099d5/telemeter-client/0.log"
Nov 03 19:15:40.617257 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:40.617170    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fc78d544-x9ljt_079406c1-7060-4a19-b62f-0f8c25e4aea7/kube-auth-proxy/0.log"
Nov 03 19:15:40.764792 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:40.764766    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-548c4c64fb-pgglg_76f649c8-7dad-410e-a2e8-960e639099d5/reload/0.log"
Nov 03 19:15:41.175057 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:41.175026    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-548c4c64fb-pgglg_76f649c8-7dad-410e-a2e8-960e639099d5/kube-rbac-proxy/0.log"
Nov 03 19:15:41.301486 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:41.301452    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-bwg8p_c17b0b8a-a3a3-4824-acc3-ce83fb644937/istio-proxy/0.log"
Nov 03 19:15:41.517406 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:41.517375    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-c55ww_acc06524-297b-4b57-b0a2-4b2a92d4a00c/istio-proxy/0.log"
Nov 03 19:15:41.592122 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:41.592097    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-576dfb567c-pbb4z_146e21dc-5041-463e-b0e7-9a79f969fa33/thanos-query/0.log"
Nov 03 19:15:41.733176 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:41.733148    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-csr7q_3b5d6ca4-de05-401f-a8ce-5e1c02a6dcc3/istio-proxy/0.log"
Nov 03 19:15:42.001547 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:42.001521    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-576dfb567c-pbb4z_146e21dc-5041-463e-b0e7-9a79f969fa33/kube-rbac-proxy-web/0.log"
Nov 03 19:15:42.412054 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:42.411977    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-576dfb567c-pbb4z_146e21dc-5041-463e-b0e7-9a79f969fa33/kube-rbac-proxy/0.log"
Nov 03 19:15:42.794609 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:42.794578    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-xkkj4_c24c1306-3d52-4b39-af5d-5b769ec03a1a/istio-proxy/0.log"
Nov 03 19:15:42.822883 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:42.822855    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-576dfb567c-pbb4z_146e21dc-5041-463e-b0e7-9a79f969fa33/prom-label-proxy/0.log"
Nov 03 19:15:43.011208 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:43.011175    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-bc8769f9-95v8w_58741805-9507-4add-ab80-651305453ee6/istio-proxy/0.log"
Nov 03 19:15:43.047413 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:43.047298    6059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk"]
Nov 03 19:15:43.094600 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:43.094566    6059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-105678f7-hgpwk"]
Nov 03 19:15:43.224629 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:43.224598    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7987dbcbcc-d2pl9_e7bdff75-bc2d-4591-993b-97eba56293ac/payload-pre-processing/0.log"
Nov 03 19:15:43.233051 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:43.233028    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-576dfb567c-pbb4z_146e21dc-5041-463e-b0e7-9a79f969fa33/kube-rbac-proxy-rules/0.log"
Nov 03 19:15:43.334815 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:43.334739    6059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5e4548a-9fa7-4846-9933-02ce4c6901da" path="/var/lib/kubelet/pods/f5e4548a-9fa7-4846-9933-02ce4c6901da/volumes"
Nov 03 19:15:43.438784 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:43.438753    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7b88dbb578-2tqps_42c95c41-1e40-4f67-8f76-6291a182f48c/payload-processing/0.log"
Nov 03 19:15:43.643967 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:43.643875    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-576dfb567c-pbb4z_146e21dc-5041-463e-b0e7-9a79f969fa33/kube-rbac-proxy-metrics/0.log"
Nov 03 19:15:44.057685 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:44.057635    6059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c574b7b6f-nbzmp_96f2949d-40c0-46ec-9cb6-235e005b0736/istio-proxy/0.log"
Nov 03 19:15:44.776143 ip-10-0-143-117 kubenswrapper[6059]: I1103 19:15:44.776103    6059 scope.go:122] "RemoveContainer" containerID="7af97f6993dfbdc05485c5553a80fcb17fb40c0a857216e2f22e52604bed4ab3"